Search by job, company or skills

Lead Data Engineer

Lead Data Engineer

Recro
5-7 Years
Not Disclosed
Early Applicant
  • Posted a day ago
  • Be among the first 10 applicants

Job Description

Job Description: Lead Data Engineer (Level 4)

Location – Chennai

Experience - 5+ Years

Role Summary

We are seeking an experienced Lead Data Engineer (Level 4) to serve as the technical leader responsible for the modernization and future-state architecture of our enterprise Sales Validation platform. In this role, you will lead the end-to-end technical delivery, drive a critical platform migration from Azure Synapse to Databricks, establish engineering excellence standards, and mentor high-performing data engineering teams.

Key Responsibilities

● Architecture & Modernization: Lead Databricks architecture, future-state design, and delivery across major platform workstreams.

● Platform Migration: Direct and execute the end-to-end migration of legacy ETL/ELT workloads from Azure Synapse to Databricks.

● Engineering Standards & Frameworks: Establish robust engineering standards, design reusable frameworks, and implement best practices for Lakehouse architecture.

● Technical Ownership & Optimization: Own technical delivery across key workstreams, driving performance tuning, scalability, and system reliability.

● Leadership & Quality Assurance: Conduct thorough architecture/design reviews, enforce code quality via code reviews, and mentor junior to mid-level engineers.

Required Qualifications & Technical Skills

1. Databricks & Lakehouse (Core Focus)

● Deep expertise in Databricks and Lakehouse Architecture.

● Advanced hands-on experience with PySpark, Spark SQL, and Delta Lake.

● Production experience implementing Unity Catalog for governance and access management.

● Strong knowledge of orchestration and automation using Databricks Workflows and Databricks Asset Bundles (DABs).

2. Azure Data Platform

● Proven experience building and orchestrating pipelines using Azure Data Factory (ADF).

● Strong knowledge of Azure Data Lake Storage Gen2 (ADLS Gen2), Azure SQL, and enterprise Azure Security Patterns.

3. Migration & Legacy Modernization

● Hands-on experience modernizing legacy ETL workloads and migrating from Azure Synapse Analytics to Databricks.

● Deep practical knowledge of Synapse components: Synapse Pipelines, Synapse Spark, Synapse SQL Pools, Synapse Notebooks, and Synapse Data Flows.

4. Core Software & Data Engineering

● Advanced SQL and complex Data Modeling capabilities.

● Proven track record in query and job performance tuning and optimization.

● Experience with CI/CD automation and version control using GitLab CI/CD.

More Info

Job Type:
Industry:
Employment Type:

Key Skills

Azure Security Patterns

Synapse Notebooks

Lakehouse Architecture

Azure Data Lake Storage Gen2

Synapse Pipelines

GitLab CI/CD

Unity Catalog

Synapse SQL Pools

Databricks Asset Bundles

Synapse Data Flows

Delta Lake

CI/CD

Synapse Spark

Databricks Workflows

About Company

Similar Jobs

5-7 yrs
Chennai, India
Skills:
Apache Spark, Databricks, Sql, Python, ETL/ELT architectures
8-10 yrs
Chennai, India
Skills:
snowflake , Power Bi, Power Automate, Etl Development, Shell Scripting, Svn, Sql, Git, Gcp, Azure, Python, AWS, Generative AI tools
5-7 yrs
Chennai, India
Skills:
snowflake , Data Modeling, Data Governance, Data Integration, AWS, Etl, dbt, Security
6-10 yrs
Chennai, India
Skills:
model development , Tensorflow, Pytorch, Sql, Python, Machine Learning Algorithms, Time Series, Clustering, Predictive Analytics, R, scikit-learn, Statistical analyses, Statistical methods, Regression, Model training, Classification, Ensemble learning, Optimizations