Databricks Platform Engineer

Databricks Platform Engineer Day to day responsibilities: Databricks Platform Administration and Support activities Setup CI/CD DevOps pipeline for various patterns Cosmos DB Graph database data ingestion and Linkurious Support Performance improvement of various scheduled jobs in Databricks Setup realtime streaming data pipelines in Databricks and provide support to Data Scientist Kubernetes container deployment along with AKS cluster setup and identify helping anomaly detection Perform POC's for various upcoming technologies as per direction from senior management Cost optimization based on the cluster usage. Suggesting users to use right type of clusters, advising user on how to perform computations etc. Ensure that data lakes are secured and we also working on automation of the same. Keep the company's WIKI up to date for any Databricks design / architecture related questions. Compute: Databricks, HDI Insight Visualization: Linkurious, Azure Data Explorer, PowerBI/Spotfire Storage: Cosmos DB, Data Lake Store, SQL DB/DW, ADLS Gen 2 Integration: Azure Service Principal, SQL DW, Databricks Software Engineering: Azure DevOps (CI/CD), Log Analytics/Cluster Overrides (monitoring), and Engineering Support - Defining CI/CD of patterns - Databricks is used in a lot of injestions (SAP, Oracle)find patterns and standards on how to implement CI/CD, Testing, etc -Monitoring: clusters being optimally used, etc Experience: - Databricks - Scala (unit testing frameworks), Python, pycharmprimarily use Scala in Data Engineering activities, but also use Python with Azure DevOps - Azure DevOps - Streaming: stream from edge devices, spark streaming, AI models that work on streaming data to run the models. In depth knowledge of structured streaming - How to connect to the storage, upload data, etc. not administrators of Gen 2, but complete understanding of working with it. - Indexes on Azure searches, how to set up cosmos graphs, partitioning on Cosmos?, - Enforce through policies so they don't do things they shouldn't be doing through clusteringHigh level architecture - BI (DBs, SAP, Files, API, Graph, Images, IOT, etc from Data sources) - Azure Data Factory / Azure Databricks / Mulesoft API : Data Ingestion (full loads, CDC, orchestration, monitoring) - Azure Data Lake Gen 2: Storage (ADLS Gen 2 - raw data from systems of record, transformed data from processing layer) - Azure Databricks, ML, Cognitive Services (Processing: clean and transform data, info models, ML and advanced Analytics) - Query & Analytics (DW, Access Mhmt, SQL DW/Azure Analysis Services, Synapse/Cosmo/Lakehouse) - Visualization and Egression (Mulesoft API, Azure DF, PowerBI Pro/Premium.Linkurious) - End: Consumer (Upstream Data Domain end user, api consumer, external vendor, internal projects)EEO Employer Apex Systems is an equal opportunity employer. We do not discriminate or allow discrimination on the basis of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at or