Associate Delivery Manager - Data Engineer

Location Name: Pune Corporate Office - Mantri

Job Purpose

To effectively design, develop, and manage data solutions using ETL technologies such as Azure Databricks (ADB) , Azure Data Factory (ADF) and SQL along with leading 3 to 5 members of developers

Duties And Responsibilities

KEY RESPONSIBILITIES

•  Lead the end-to-end design, development, and delivery of scalable data engineering solutions using Azure Databricks, ADF, PySpark, SQL, and Delta Lake.
•  Convert business requirements into robust technical designs, architecture documents, data models, and implementation plans.
•  Own technical delivery of data integration, ETL, lakehouse, semantic layer, and AI/BI enablement initiatives.
•  Guide and mentor data engineers on coding standards, design best practices, performance optimization, and reusable framework development.
•  Review technical designs, code, pipelines, and deployment plans to ensure quality, scalability, maintainability, and compliance.
•  Drive architecture decisions for batch and near-real-time data pipelines across Bronze, Silver, and Gold layers.
•  Ensure data quality, reconciliation, anomaly detection, and timely resolution of production issues through effective RCA and permanent fixes.
•  Optimize data pipelines, Databricks jobs, SQL queries, and storage usage to improve performance and reduce cost.
•  Implement CI/CD practices, version control, automated deployments, and environment management across Dev, QA, and Production.
•  Collaborate with PMO, business stakeholders, BI teams, InfoSec, DevOps, and external partners for smooth project execution.
•  Establish SOPs, engineering standards, reusable components, monitoring frameworks, and documentation practices.
•  Track delivery progress, manage technical dependencies, prioritize work, and ensure timely closure of project milestones.
•  Support adoption of modern data platforms, semantic modeling, metrics layer design, and GenAI/BI capabilities.
•  Ensure compliance with data governance, security, access control, audit, and enterprise data management standards.
•  Act as the technical escalation point for critical issues, complex solutioning, and cross-team dependency resolution.
  
  

Key Decisions / Dimensions


KEY DECISIONS / DIMENSIONS

•  Define semantic layer design and metric definitions
•  Prioritize data vs AI optimization trade-offs
•  Handle production issues with RCA and long-term fixes
•  Drive architectural decisions for lakehouse + Data integration
  
  

Major Challenges


MAJOR CHALLENGES

•  Ensuring Data Delivery within TAT
•  Driving adoption of GenAI-based BI over traditional dashboards
•  Balancing performance, cost, and scalability
•  Managing dependencies across data engineering, AI, and business teams
  
  

Required Qualifications And Experience


REQUIRED SKILLS & EXPERIENCE

Must Have

•  Azure Databricks – PySpark, SQL, Delta Lake
•  Strong experience in Semantic Modeling & Metrics Layer design
•  Hands-on with Databricks workflows
•  Pyspark (Pandas, PySpark, FastAPI)
•  Azure Data Factory (ADF) for ETL pipelines
•  Strong SQL and data modeling skills
  
  

Good to Have

•  Cosmos DB / MongoDB (NoSQL concepts)
•  Azure Data Explorer (KQL)
  
  

DATA STACK (MANDATORY FOR SCREENING)

SNo Data Platform / Concepts Associated Technologies

1 Databricks Lakehouse PySpark, SQL, Delta Lake

2 AI for BI Databricks Genie, Genie Rooms, Instructions, Agents

4 ETL & Orchestration Azure Data Factory

5 Programming Pyspark

6 Cloud Platform Azure (Preferred)

10 DevOps CI/CD Pipelines, Git