Lead Data Engineer (Azure Databricks & Spark)
We are currently hiring for a
Lead Data Engineer for one of our clients (a leading Software Services/Product Company) based out of
Pune.
We are looking for experienced Data Engineering professionals who are passionate about building modern, scalable, and cloud-native data platforms. The ideal candidate should have strong expertise in Azure Databricks, Apache Spark, Azure Data Services, and enterprise-scale data engineering. This role requires both hands-on technical leadership and the ability to mentor engineering teams while delivering high-quality data solutions. This role offers above-market compensation along with standard benefits for the right candidate.
Key Responsibilities
- Design, build, and maintain scalable end-to-end data pipelines for data ingestion, transformation, and serving.
- Develop and optimize large-scale data processing solutions using Azure Databricks and Apache Spark.
- Design and implement Delta Lake architectures following Bronze, Silver, and Gold data layers.
- Build cloud-native data solutions using Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory, Azure Key Vault, Event Grid, and API Management.
- Lead and execute enterprise data migration initiatives, including on-premises migrations, cross-workspace migrations, Unity Catalog migrations, and historical data loads.
- Implement CI/CD pipelines and automation for data engineering workflows and infrastructure.
- Develop Infrastructure as Code (IaC) solutions using Terraform and Databricks Asset Bundles.
- Implement data governance, security, and access management using Unity Catalog, RBAC, and Service Principals.
- Monitor production environments, troubleshoot incidents, and ensure platform reliability and performance.
- Lead technical discussions, mentor engineering teams, conduct code reviews, and drive engineering best practices across projects.
Required Skills / Primary Skills
- 10+ years of experience in Data Engineering and Big Data technologies.
- Minimum 3+ years of hands-on experience with Azure Databricks and Apache Spark.
- Strong experience building end-to-end enterprise data pipelines.
- Deep understanding of Delta Lake Architecture including Bronze, Silver, and Gold data layers.
- Strong hands-on experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Experience with Azure Data Factory (ADF) for orchestration and pipeline development.
- Hands-on experience with Azure Key Vault, Event Grid, and Azure API Management.
- Experience delivering large-scale cloud-based data platforms on Microsoft Azure.
- Strong experience with enterprise data migration projects, including historical data migration and platform modernization.
- Experience implementing CI/CD pipelines for data engineering workloads.
- Hands-on expertise with Terraform and Databricks Asset Bundles.
- Strong understanding of data governance using Unity Catalog, RBAC, and Service Principals.
- Experience supporting production environments, monitoring data platforms, and resolving production incidents.
Additional Skills
- Experience leading and mentoring Data Engineering teams.
- Strong understanding of software engineering best practices, code reviews, and technical governance.
- Experience designing highly available, scalable, and secure cloud data platforms.
- Strong problem-solving and troubleshooting skills in distributed data processing environments.
- Experience working in Agile development environments.
- Excellent communication and stakeholder management skills.
- Ability to drive architecture decisions and establish engineering best practices across teams.
Notice Period
Immediate Joiners Preferred
If you're passionate about building enterprise-scale cloud data platforms and have strong expertise in Azure Databricks, Apache Spark, Delta Lake, and Azure Data Engineering services, we'd love to hear from you. Apply now and become part of an exciting technology team.
