Data Engineer (Spark)

Preferred Skills/Experience

• 10+ years of overall experience in software development with 5-6 years of relevant experience in designing, developing, deploying and operating data streaming pipelines at scale.
• 3-4 years’ experience with Apache Kafka and Apache Spark Streaming
• Hands on experience with Spark Structured Streaming
• Experience in tuning Spark Data pipeline to achieve high throughput.
• Drive efforts to improve the data quality across data pipelines and implement system controls for managing data quality
• Programming proficiency in Scala or Java.
• Open Source Committer (Apache Spark or related Big Data open source technologies)
• Experience with Containers, Kubernetes and scaling elastically
• Strong background in algorithms and data structures and continuously develop and acquire new technical skills
• Lead and mentor junior engineers to ensure systems are built with highest quality and leveraging best practices.
• Experience in automating Spark pipeline deployment/testing (DevOps, CI/CD)
• Passion for data engineering and for enabling others by making the product easier to use.
• Excellent communication in sharing context to effectively collaborate with analytical partners, domain experts and other consumers of your work, preferably in supporting an engineering or product function