Databricks Platform Engineer
Location: PAN India
Experience: 7+ Years
We are looking for an experienced Databricks Platform Engineer to design, build, and manage enterprise-scale Data & AI platforms on Azure Databricks. The ideal candidate will have strong expertise in Databricks administration, Delta Lake, Unity Catalog, Azure cloud services, and modern data engineering practices, along with experience building AI/RAG solutions.
Key Responsibilities:
- Manage Azure Databricks workspaces, clusters, jobs, and governance using Unity Catalog.
- Design and implement Delta Lake Medallion Architecture (Bronze, Silver, Gold).
- Build and optimize data pipelines using Python, PySpark, and SQL.
- Configure and manage ADLS Gen2, Key Vault, Service Principals, and IAM.
- Develop CI/CD pipelines using Azure DevOps or GitHub Actions.
- Deploy and manage Databricks workflows, DLT pipelines, and model-serving endpoints.
- Build and support RAG (Retrieval-Augmented Generation) solutions using Vector Search, embeddings, and Azure OpenAI.
- Monitor platform performance, troubleshoot issues, and optimize cost and scalability.
Required Skills:
- Azure Databricks (Workspace Administration, Clusters, Jobs, Unity Catalog)
- Delta Lake (MERGE, CDF, Schema Evolution, Z-Ordering)
- Python, PySpark, SQL
- ADLS Gen2, Key Vault, Entra ID, IAM
- CI/CD (Azure DevOps/GitHub Actions)
- Unity Catalog & Data Governance
- RAG Architecture, Vector Search, Embeddings, LLM Integration
- Azure AI Services, Event Hubs, REST APIs
Good to Have:
- MLflow
- Databricks Asset Bundles (DAB)
- Azure OpenAI, Azure AI Search
- Delta Live Tables (DLT)
Notice Period: Immediate to 30 Days Preferred
