Python - Spark with Azure Developer

Job TItle : Python - Spark with Azure Developer Location: Miami, FL Duration : 12+ Months Detailed JD: Background:

• Azure Databricks is unified data analytics' platform for large volume data processing and engineering.

• Data Services (Data Analytics organization) is looking for a senior Data Engineer with experience on ingestion of structured and unstructured data from multitude of sources into the Enterprise Data Lake infrastructure.
• The Engineer performs cleansing, transformation, and standardization of inbound data into Databricks' Delta tables preferably using Spark and/or Python.
• The Engineer performs 'feature engineering' of variables in preparation for data exploration and input to Machine Learning's models built by Data Scientists, using Spark-SQL and DataFrames.
• Experience in Microsoft Azure's Databricks, Data Factory, Data Lake Store (ADLS).
• The requirement is for a hands-on lead developer with enough seniority to perform his/her coding and soft skills to coach other Data Engineers about development patterns, best practices, and design methodologies.
• Data services' expectation is for someone who can participate and contribute to overall Data Engineering solution design based on his/her in-depth technical expertise.

• This developer is not expected to build Machine Learning's models or participating in Statistical programming work.

• Databricks experience on Azure or any virtualized environment.
• Excellent in writing code using PySpark, Spark-SQL or Python.
• Scripting language, such as Linux/Unix Shell, Windows PowerShell, and Azure CLI.
• Knowledge of SQL Server, ETL tools like SSIS, Azure Data Factory is a plus.
• Other preferred skillsets:
• Agile methodology to product delivery.
• Programming skills on .NET or Java.
• Any NoSQL database like CosmosDB, MongoDB, Cassandra, Redis etc.
• Experience building Microservices and Web-APIs