Lead Data Engineer

Senior Databricks Developer

• Location: Gurgaon / Bangalore (5 Days Mandatory Work from Office)
• Employment Type: Full-Time
• Experience: 9+ Years

About the Role

We are seeking an experienced Senior Databricks Developer to design, develop, and optimize scalable data pipelines and modern Lakehouse solutions on the Databricks platform. The ideal candidate will have strong expertise in Data Engineering, distributed data processing, ETL/ELT development, and cloud-based data platforms, along with the ability to lead and mentor a small team.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Delta Lake.
• Build and optimize data ingestion frameworks, transformation workflows, and end-to-end data processing solutions for both batch and streaming workloads.
• Implement and manage Delta Lake architectures, including schema evolution, time travel/versioning, and ACID-compliant data pipelines.
• Collaborate with business stakeholders, architects, and cross-functional teams to gather requirements and deliver scalable data solutions.
• Manage, monitor, and optimize Databricks clusters, jobs, workflows, and notebooks to improve performance and cost efficiency.
• Ensure data quality, reliability, and observability through robust validation, monitoring, and alerting frameworks.
• Contribute to data modeling, metadata management, governance standards, and platform best practices.
• Lead and mentor a team of 2-3 engineers while driving technical excellence and delivery.

Required Skills & Experience

• 9+ years of overall IT experience with 7+ years in Data Engineering.
• Minimum 5+ years of hands-on Data Engineering experience, including 3+ years of strong Databricks expertise.
• Strong understanding of Data Warehouse (DWH) concepts, ETL methodologies, and data integration standards.
• Hands-on experience with PySpark, Spark SQL, and large-scale distributed data processing.
• Excellent SQL skills with experience handling high-volume and complex datasets.
• Strong knowledge of Delta Lake, Medallion Architecture, and Lakehouse design principles.
• Experience with CI/CD pipelines, Git, version control, and modern DevOps practices for data engineering.
• Good understanding of structured and unstructured data processing, data quality frameworks, and performance optimization.
• Proven ability to lead, guide, and mentor a team of 2-3 members.

Preferred Qualifications

• Experience working in cloud-based data platforms and modern data ecosystems.
• Exposure to real-time data processing and streaming frameworks.
• Strong problem-solving, communication, and stakeholder management skills.
• Ability to work in a fast-paced environment and drive end-to-end delivery of data engineering solutions.

Note: This is a Work-from-Office role, requiring candidates to be based out of or willing to relocate to Gurgaon or Bangalore.