Senior Azure Data Engineer

your knowledge of Azure-based ETL. You will work with cloud architects to establish pipelines and develop spark code using Databricks platform. You should have sufficient knowledge and maturity to effectively deal with technical issues and help to support the broader team. Work can be performed from Lafayette, LA, Bloomfield, CT or Raleigh, NC.

• The senior data engineer will be responsible for development, deployment, maintenance, diagnostics and support of spark ETL jobs on the MS Databricks platform.
• Will streamline code, rationalize datasets and tables to arrive at single source of truth
• Will build detailed architecture diagrams for all workflows and processes
• Migrate existing clients from the existing On-Prem solution to the Azure Solution

• 5 years of hands-on experience with cloud automation and scripting in an Azure environment.
• Experience in writing and running Unit Tests on Databricks using Azure Pipeline is preferred
• Knowledge of hands-on agile development – experience in visualizing and breaking development work into smaller units such as user stories, tasks, etc.
• Experience in peer-reviewing code – provide recommendations for improving code quality and deliverables to the client
• Experience in stream processing tools (Spark structured streaming)
• Experience in serverless and cloud technologies like Azure functions
• Experience in data modeling and in developing data warehouse and data lake solutions.
• Experience in monitoring and logging services in Azure
• Experience in the development of Apache Spark code using PySpark/Scala
• Experience in using Azure Databricks Platform
• Experience in using Azure Data Factory to call Azure Databricks notebook activities
• Experience in using Azure Storage (Blob/Data Lake Store), Databricks Delta
• Experience in using Git-based repositories
• Good fundamental knowledge about distributed computing, RDBMS, and Dimensional modeling concepts
• Exposure to Azure DevOps