Job Title: Lead Azure Data Engineer
Location: Bangalore, Chennai, Hyderabad, Pune
Expereinece: 8-15 Years
Work Type: Contract (Extendable)
Job Description
We are looking for an experienced
Lead Data Engineer with 8
+ years of experience in designing, developing, and maintaining enterprise-scale data engineering solutions. The ideal candidate should have strong expertise in
Microsoft Fabric, Databricks, PySpark, SQL/T-SQL, Delta Lake, Azure Data Factory, and Power BI. You will play a key role in modernizing legacy data platforms by building scalable
Bronze, Silver, and Gold Medallion Architecture pipelines and migrating SQL Server ETL workloads to Microsoft Fabric.
Roles & Responsibilities
- Design, develop, and maintain scalable Bronze, Silver, and Gold data pipelines using Microsoft Fabric and Databricks.
- Implement Cascade 1.0 and Cascade 2.0 merge logic within the Silver layer, managing schema evolution, new attributes, and null value handling.
- Migrate legacy SQL Server objects including tables, views, stored procedures, and ETL processes into Microsoft Fabric Pipelines, Fabric Notebooks, and Databricks.
- Develop high-performance PySpark transformations and build optimized Delta Lake tables for enterprise data processing.
- Write and optimize complex SQL/T-SQL queries, stored procedures, views, functions, CTEs, and performance-tuned scripts.
- Design and develop Azure Data Factory (ADF) pipelines for orchestrating enterprise data ingestion and transformation.
- Build and maintain scalable Lakehouse solutions using Microsoft Fabric and OneLake.
- Perform data ingestion, cleansing, transformation, validation, and reconciliation to ensure data quality and consistency.
- Establish data quality checks, lineage, reconciliation (apple-to-apple validation), and governance across enterprise datasets.
- Develop and optimize semantic models and support Power BI datasets for enterprise reporting and analytics.
- Collaborate with Data Engineers, BI Developers, QA teams, and business stakeholders to deliver high-quality data solutions.
- Optimize pipeline performance, resource utilization, and overall processing costs.
- Participate in Agile ceremonies including sprint planning, daily stand-ups, sprint reviews, retrospectives, and code reviews.
- Prepare technical documentation for data pipelines, migration activities, architecture, and operational support.
Required Skills
- Microsoft Fabric (Lakehouse, OneLake, Fabric Pipelines)
- Databricks
- PySpark
- SQL / T-SQL
- Delta Lake
- Azure Data Factory (ADF)
- Power BI (Datasets & Semantic Models)
- Data Engineering
- Medallion Architecture (Bronze, Silver, Gold)
- ETL Development
- Data Migration
- Data Quality & Data Lineage
- Agile / Scrum
