Senior Databricks Data Engineer

We are seeking a Senior Databricks Data Engineer to provide technical leadership for a large-scale enterprise BI modernization initiative in Pune, India. CES is open to considering qualified candidates for either a direct-hire, full-time opportunity or a contract-to-hire arrangement, depending on the candidate's experience, availability, and preferred employment structure.

This is not a lift-and-shift migration effort. The role will lead the systematic transformation of a complex Microsoft BI ecosystem—including SSIS, SSAS, and SSRS—into a scalable, governed Databricks Lakehouse architecture.

The Senior Engineer will establish architectural standards, define repeatable migration frameworks, and guide the modernization of thousands of legacy data assets into reusable, high-performance Spark-based solutions. This individual will combine deep hands-on data engineering expertise with architectural thinking, technical leadership, mentorship, and strong cross-functional collaboration skills.

Key Responsibilities

• Lead the architecture and implementation of enterprise-scale data pipelines on Databricks.
• Serve as technical lead for the migration of an approximately 20TB Microsoft SQL Server data warehouse to the Databricks Lakehouse Platform.
• Design and implement reusable Spark-based pipeline frameworks to replace thousands of legacy SSIS workflows.
• Deconstruct SSAS cube logic and redesign it into Lakehouse-native semantic models.
• Modernize legacy SSRS reporting ecosystems into governed analytics and reporting layers.
• Rebuild dimensional models within Delta Lake using scalable Lakehouse design patterns.
• Define data modeling standards, ingestion frameworks, CI/CD processes, testing approaches, and operational best practices.
• Implement and enforce Medallion architecture standards across Bronze, Silver, and Gold layers, including performance optimization and governance controls.
• Optimize Spark workloads for scalability, reliability, performance, and cost efficiency.
• Implement governance, lineage, security, monitoring, and access controls using Unity Catalog and related Databricks platform capabilities.
• Mentor engineers, conduct architectural and code reviews, and establish and maintain engineering standards.
• Partner with BI, analytics, and AI/ML teams to enable advanced analytics and downstream use cases.
• Contribute to platform strategy, technical roadmap, and architectural decisions.

Qualifications

• Bachelor's or master's degree in Computer Science, Engineering, or a related field.
• 8–12 years of overall data engineering experience.
• 5+ years of hands-on experience building and operating production workloads on Databricks.
• Deep expertise in SQL, PySpark, and distributed data processing.
• Demonstrated technical leadership in structured enterprise BI modernization or large-scale data platform migration initiatives.
• Proven experience replacing SSIS-based ETL processes with scalable Spark-based frameworks.
• Strong understanding of SSAS cube architecture and semantic layer modernization.
• Experience designing and operating high-volume, enterprise-grade analytical data platforms.
• Strong knowledge of Delta Lake, Lakehouse and Medallion architectures, data versioning, and Databricks Workflows/Jobs orchestration.
• Experience implementing CI/CD, automated testing frameworks, monitoring, and operational reliability for data platforms.
• AWS-based Databricks experience is strongly preferred.

Preferred Skills

• Experience designing modern semantic layers and BI integrations, with Power BI experience preferred.
• Exposure to Databricks AI/ML capabilities, including MLflow, feature engineering, and feature stores.
• Experience establishing repeatable enterprise migration patterns across distributed or global teams.
• Prior technical leadership or mentorship experience within multi-team environments.
• Strong stakeholder engagement and communication skills.
• Databricks certification is a plus.

Position Details

• Location: Pune, Maharashtra
• Employment Type: Direct Hire or Contract-to-Hire
• Work Arrangement: Hybrid, with a minimum of three days per week in the Pune office, preferably Tuesday, Wednesday, and Thursday

Additional Information

Customized Energy Solutions India offers competitive compensation along with performance-based incentives tied to individual and company performance. Our benefits are designed to support both professional and personal well-being and include comprehensive medical insurance coverage for associates and eligible dependents, generous paid time off, and opportunities for professional development through training and skill-building programs.

We are committed to creating an environment where associates are supported, challenged, and provided opportunities to grow and develop their careers over the long term.