Data Engineer

REQUIRED SKILLS

Top 3-5 skills:

• Azure – heavy Azure experience
• Spark
• Scala
• Kafka
• Docker/Kubernetes

Preferred Skills/Experience

•        10+ years of overall experience in software development with 5-6 years of relevant experience in designing, developing, deploying and operating data streaming pipelines at scale.

•        3-4 years’ experience with Apache Kafka and Apache Spark Streaming

•        Hands on experience with Spark Structured Streaming

•        Experience in tuning Spark Data pipeline to achieve high throughput.

•        Drive efforts to improve the data quality across data pipelines and implement system controls for managing data quality

•        Programming proficiency in Scala or Java.

•        Open Source Committer (Apache Spark or related Big Data open source technologies)

•        Experience with Containers, Kubernetes and scaling elastically

•        Strong background in algorithms and data structures and continuously develop and acquire new technical skills

•        Lead and mentor junior engineers to ensure systems are built with highest quality and leveraging best practices.

•        Experience in automating Spark pipeline deployment/testing (DevOps, CI/CD)

•        Passion for data engineering and for enabling others by making the product easier to use.

•        Excellent communication in sharing context to effectively collaborate with analytical partners, domain experts and other consumers of your work, preferably in supporting an engineering or product function

Required Skills Summary: Apache Spark, Apache Kafka, Scala/Java, NoSQL Databases, Elasticsearch & Kibana, Kubernetes, Docker Containers