LEAD DATA ENGINEER - Databricks

Senior Data Engineer (Databricks + CosmosDB)

Location: Bangalore (Hybrid - 3 days from client office)

Experience: 8-10 Years

Job Summary: We are seeking a highly skilled Senior Data Engineer with deep expertise in Azure Databricks and Cosmos DB. The ideal candidate will be responsible for designing, building, and optimizing scalable, cloud-native data platforms and data pipelines. This role requires a hands-on approach to developing high-performance data solutions that support analytics, AI/ML initiatives, real-time processing, and enterprise reporting requirements while ensuring data quality, security, and operational excellence.

Responsibilities:1. Data Engineering & Pipeline Development

• Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and Apache Spark.
• Build robust ETL/ELT processes to ingest, transform, and curate large volumes of structured and unstructured data.
• Develop reusable data engineering frameworks and accelerators.
• Optimize data processing jobs for performance, scalability, and cost efficiency.

2. Databricks Platform Development

• Build and maintain data solutions using Azure Databricks.
• Develop notebooks and workflows using Apache Spark RDD, Apache Spark SQL, and Python.
• Implement Delta Lake architectures for reliable and efficient data management.
• Configure and optimize Databricks clusters and workloads.
• Support CI/CD implementation for Databricks deployments.

3. Cosmos DB Development & Optimization

• Design and implement scalable solutions using Azure Cosmos DB (NoSQL).
• Model data effectively for NoSQL workloads.
• Optimize partition strategies, indexing policies, and query performance.
• Manage data replication, consistency models, and throughput provisioning.
• Integrate Cosmos DB with analytics and downstream systems.

4. Azure Data Platform Integration

• Develop and integrate end-to-end solutions using Azure services such as Azure Data Factory, Azure Functions, and Azure Logic Apps.

5. Real-Time & Streaming Data Processing

• Build streaming data pipelines using Spark Structured Streaming.
• Process event-driven workloads from Azure Event Hubs, Kafka Stream, or IoT sources.
• Implement near real-time analytics solutions integrated with Cosmos DB and Databricks.

6. DevOps & CI/CD

• Implement Devsecops practices for Data Engineering solutions.
• Automate deployment pipelines using Azure DevOps or GitHub Actions.
• Manage source control and release processes.

Mandatory Skills:

• Azure Databricks
• Apache Spark RDD
• Apache Spark SQL
• Delta Lake
• Azure Cosmos DB (NoSQL)

Preferred Skills:

• Delta Live Tables (DLT)
• Azure Databricks
• Azure Databricks

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 8-10 years of experience in data engineering or related roles.
• Strong understanding of cloud engineering principles and modern data architectures.
• Excellent problem-solving skills and ability to work in a fast-paced environment.