Job description:

Job Title: Azure Data Engineer – Azure Data Factory, Azure Data Lake, Azure Databricks

Experience: 3+ years

Location: Pune, India
(Hybrid/Remote as per project need)

Shifts: 6:30 AM to 3:30 PM IST

(Client shift may apply)

Role Summary

You will build and support Azure-based data platforms.
You will create pipelines for ingestion, transformation, and analytics.
You will manage data lake and warehouse layers with strong data modeling.
You will enable AI/ML workloads by preparing quality datasets and supporting Azure ML.

Primary Skills (Must Have)

Azure Data Factory (ADF) – pipeline design, triggers, monitoring, error handling
Azure Databricks (Spark / PySpark) – transformations, performance tuning, Delta (if used)
Azure Data Lake Storage (ADLS Gen2) – lake design, folder structure, partitioning
Azure Synapse Analytics – analytics/warehouse concepts and data serving
SQL (Advanced) – complex queries, validation, tuning
Python – data processing + scripting (ML exposure is a plus)
Data Modeling & ETL – strong warehouse and dimensional modeling understanding
Integration of multiple Azure services end-to-end

Key Responsibilities

1) Data Ingestion & Orchestration (Azure Data Factory)

Design and build scalable ADF pipelines for batch and incremental loads.
Configure linked services, datasets, triggers, and integration runtime.
Implement retry logic, alerts, and failure handling.
Maintain pipeline standards, parameters, and reusable templates.
Monitor daily runs and fix failures with proper RCA.

2) Data Lake Design & Storage Management (ADLS + Azure SQL)

Design data lake layers: raw, staged, curated, consumption.
Ensure correct formats like Parquet/Delta/CSV based on need.
Apply partitioning and naming standards for performance and clarity.
Manage curated datasets in Azure SQL Database when required.
Ensure data availability, retention, and lifecycle policies.

3) Data Transformation & Big Data Processing (Databricks)

Develop transformations using PySpark / Spark SQL in Databricks.
Implement data quality checks and reconciliation rules.
Optimize cluster usage, caching, and job performance to reduce cost.
Implement incremental processing and upsert patterns (MERGE) if needed.
Schedule and run Databricks jobs through ADF or job workflows.

4) Data Warehousing & Analytics (Synapse)

Build and support analytics solutions using Azure Synapse.
Design warehouse objects and implement loading strategies.
Support query tuning and performance improvement.
Publish curated, trusted datasets for BI and downstream apps.

5) Data Modeling & ETL Design

Create logical and physical data models for reporting and analytics.
Apply star schema / dimensional modeling where needed.
Maintain source-to-target mapping and transformation rules.
Ensure data consistency across lake, warehouse, and BI layers.

6) AI/ML Enablement (Azure Machine Learning)

Support ML pipelines through feature preparation and dataset readiness.
Work with Data Scientists for training and deployment support.
Build Python scripts for model experiments when required.
Use libraries like Scikit-learn (preferred), and TensorFlow/PyTorch (good to have).
Track model inputs, outputs, and repeatable pipeline execution.

7) SQL, Python & Engineering Practices

Write optimized SQL for validation, reconciliation, and transformations.
Write clean Python code for automation and data processing.
Use Git with good branching and PR review practices.
Support CI/CD practices for data pipelines if project has it.

8) Security, Compliance & Governance

Follow best practices for secure data handling and access control.
Work with RBAC, managed identity, and Key Vault where applicable.
Ensure compliance with client policies and audit needs.
Implement encryption, access boundaries, and safe data sharing.

9) Agile Delivery & Production Support

Work in Agile/Scrum mode and deliver stories on time.
Provide estimates and daily updates to stakeholders.
Support production issues and perform RCA with prevention steps.
Maintain runbooks and operational documents.

Secondary Skills (Good to Have)

Power BI – dataset modeling, dashboards, refresh, performance basics
Azure Functions / Logic Apps – automation and integration support
Azure Cognitive Services – awareness for AI use cases (optional)
Big data background: Hadoop basics, strong Spark understanding
Monitoring tools: Log Analytics / Azure Monitor (as used in project)
DevOps exposure: Azure DevOps pipelines for data workloads

Tools / Technologies (Typical)

Azure: ADF, ADLS Gen2, Databricks, Synapse, Azure SQL, Azure ML
Languages: Python, SQL, PySpark
Dev Tools: Git, Azure DevOps / Jira (as applicable)
Monitoring: ADF monitor, Databricks job runs, Azure Monitor (if enabled)

Qualification

BE/BTech/BCA/MCA or equivalent practical experience

Soft Skills

Clear communication and strong ownership.
Good problem solving and troubleshooting mindset.
Good documentation habit and disciplined delivery.

Works well with business, platform, and security teams

Azure Data Engineer - Databricks