Databricks Technical Lead

Databricks Tech Lead – Job Description

Job Title: Databricks Tech Lead

Experience: 7–12 years

Location: India / Remote

Employment Type: Full-time/ Contract

Work Mode: Remote / Hybrid

Job Summary

We are looking for an experienced Databricks Tech Lead to lead the design, development, and implementation of scalable data engineering solutions using the Databricks platform. The ideal candidate should have strong hands-on experience with Azure Databricks, Spark/PySpark, SQL, Delta Lake, Azure Data Factory, and cloud data platforms, along with the ability to guide a technical team and work closely with stakeholders.

Key Responsibilities

• Lead the design and development of scalable Databricks-based data engineering solutions.
• Provide technical leadership to data engineers and conduct design/code reviews.
• Develop and optimize data pipelines using PySpark, Spark SQL, SQL, and Delta Lake.
• Design and implement ETL/ELT workflows using Azure Databricks and Azure Data Factory.
• Work with Delta Lake, Unity Catalog, ADLS, Azure Synapse, and other Azure data services.
• Implement Medallion Architecture including Bronze, Silver, and Gold layers.
• Optimize Spark jobs, Databricks workloads, and SQL queries for performance and cost.
• Design reliable data ingestion and transformation frameworks.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Collaborate with architects, data engineers, business stakeholders, and project teams.
• Troubleshoot complex production and data pipeline issues.
• Establish technical standards, best practices, and reusable frameworks.
• Support CI/CD and deployment processes for Databricks workloads.
• Mentor team members and contribute to technical documentation.

Required Skills

• 7+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Strong knowledge of Apache Spark / PySpark.
• Advanced SQL skills.
• Experience with Delta Lake and Lakehouse architecture.
• Strong experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage (ADLS).
• Knowledge of Unity Catalog and Databricks security/governance.
• Experience designing ETL/ELT pipelines.
• Strong understanding of data modeling and data warehousing concepts.
• Experience with Git and CI/CD pipelines.
• Strong debugging, problem-solving, and technical leadership skills.

Good to Have

• Databricks certification.
• Experience with Azure Synapse / Microsoft Fabric.
• Experience with Terraform or infrastructure-as-code.
• Experience with streaming technologies such as Kafka / Event Hubs.
• Knowledge of Delta Live Tables / Lakeflow.
• Experience with data governance and security.
• Experience working in large-scale enterprise data environments