Azure Datafactory

Responsibilities :

Key Responsibilities:

Design, develop, and maintain end-to-end data pipelines using Azure Data Factory for batch and scheduled workloads.

Build and orchestrate data transformations and processing workflows using Databricks aligned to business requirements.

Implement robust pipeline monitoring, alerting, and failure handling to ensure reliable and repeatable executions.

Perform data validation and reconciliation checks to ensure accuracy, completeness, and consistency across sources and targets.

Optimize pipeline performance by tuning ADF activities, improving orchestration logic, and streamlining transformations in Databricks.

Collaborate with cross-functional teams to gather requirements, estimate effort, and deliver solutions within timelines.

Maintain technical documentation for pipelines, datasets, schedules, dependencies, and operational runbooks.

Support deployments and environment promotions by following structured release and change management practices. Minimum Qualifications:

Education: BTECH, MTECH, MCA, MSC.

Overall experience: 3â€5 years in data engineering / data integration roles.

Hands-on experience with Azure Data Factory including pipeline development, scheduling, triggers, and integration patterns.

Hands-on experience with Databricks for data processing and transformation workflows.

Ability to troubleshoot pipeline failures, analyze logs, and implement corrective actions to improve stability.

Additional Responsibilities:

Experience designing scalable orchestration patterns in ADF (parameterization, reusable components, dependency handling).

Strong understanding of data transformation best practices and implementing efficient processing in Databricks.

Exposure to building data quality checks and operational dashboards for pipeline health and SLA tracking.

Experience collaborating with stakeholders to translate business requirements into technical pipeline designs and delivery plans.

Familiarity with performance tuning approaches for cloud data pipelines and distributed processing workloads.

Technical and Professional Requirements:

Azure Synapse Analytics, Azure Data Lake Storage (ADLS), PySpark, Delta Lake, CI/CD for data pipelines, AZURE DATAFACTORY