20 Jan
Data Engineer
Vacancy expired!
- Strong in Azure Data Factory with Databricks, file movement and general transformation
- Good with Azure Databricks with Database and ADLS (Knowledge on PySpark to Spark context, using Parquet files)
- Good Azure DevOps and its interaction with Databricks and Data Factory
- Good understanding of on-premise big data technology.
- Good knowledge on Hive and pig tables in Hadoop.
- Good/Strong understanding of ETL & Data Warehousing concepts
- Strong in connecting to and ingesting from multiple source types to Azure ADLS
- Good understanding of metadata driven development
- Excellent problem solving, Critical and Analytical thinking skills
- Provide subject matter expertise and hands on delivery of data capture, curation and consumption pipelines on Azure and Hadoop
- Conduct full technical discovery, identifying pain points, business and technical requirements, “as is” and “to be” scenarios.
- Adapt to existing methods and procedures to create possible alternative solutions to moderately complex problems.
- Ensures a consistent and coherent technical delivery of solutions o Participates in testing design and execution, peer code reviews, etc.
- Identifies opportunities to drive down service costs though technology or process enhancements.
- Should be able to profile and Optimize the ADF pipeline performance and be willing to pick up Scala if need arises.
- Participating in architectural reviews and discussions
- Good communication Skills and customer Interacting skills
- This role will focus on Data engineering and ETL development in Azure Databricks.
- A majority of the transformation logic will be in Databricks.
- A strong understanding of Data Factory is Required.
- This role will assist in ETL design leveraging standard Data Warehousing techniques and functional programming techniques
Vacancy expired!