Software Engineering Senior JAVA/APACHE
Technologies & Skills:
Java, Apache Spark, Delta Lake, Spark SQL, MySQL, Kafka, drools, Spring Boot, REST APIs, Maven, Jenkins, Docker, Kubernetes
Job Description
Software Engineer (Apache Spark Applications developed in Java)
Must haves
- 5+ years of Java and Apache Spark development experience
- 5+ years of experience developing Data pipelines (with Apache Spark)
- 5+ years working on Big Data projects focused on real time processing and Big Data pipeline optimization (Delta lake, Apache Spark
Responsibilities
- Design and implement highly efficient Spark applications in Java, focusing on performance optimization, including tuning Spark jobs, optimizing transformation, and leveraging advanced Spark features for batch and real-time processing.
- Responsible for developing data pipelines using Apache Spark (Big Data Processing )
- Responsible for analyzing performance concerns and issues. Identifying the bottlenecks that require changes to improve performance.
- Utilize in-depth knowledge of functional and technical expertise in Java/J2EE Micro-services (Spring Boot) and Big Data in delivering solutions.
- Responsible for configuring the applications for DevOps using Jenkins and GIT and build pipeline using Jenkins.
- Deploy the applications to the cloud environment using various cloud services and maintaining the application in Open Shift and AWS
- Responsible for developing Staging Layer using Delta Lake for the Ingestion Framework.
- Implement and manage data publishing and consumption using Kafka.
- Implement solutions to transform and serialize datasets into JSON messages using Spark SQL
- Understanding of RESTful web services and API development.
Technologies & Skills:
Java, Apache Spark, Delta Lake, Spark SQL, MySQL, Kafka, drools, Spring Boot, REST APIs, Maven, Jenkins, Docker, Kubernetes