ADF (Azure Data Factory)Databricks+Pyspark

Technology->Big Data - Data Processing->PySpark Technology->Cloud Integration->Azure Data Factory (ADF) Technology->Data Engineering->Databricks

Key Responsibilities: Data Engineering & Delivery

• Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing
• Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability
• Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing
• Build reusable frameworks, templates, and standards for pipeline development and deployment Architecture & Performance
• Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices
• Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing)
• Establish monitoring, alerting, and operational runbooks for production pipelines Leadership & Collaboration
• Provide technical leadership, code reviews, and mentoring to ensure high engineering standards
• Collaborate with stakeholders to translate business requirements into scalable data solutions
• Drive delivery planning, estimation, and risk management for data engineering initiatives Minimum Qualifications:
• BTECH, MTECH, MCA, MSC (or equivalent) in Computer Science, Engineering, or related field
• 7–9 years of experience in data engineering with strong hands-on delivery ownership
• Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns
• Strong expertise in Databricks for building scalable data processing solutions
• Hands-on proficiency with PySpark for building and optimizing distributed data transformations
• Experience building production-grade pipelines with logging, error handling, and operational support readiness Preferred Qualifications:
• Experience designing medallion/layered data architectures and implementing reusable transformation patterns in Databricks
• Strong understanding of data modeling concepts and building curated datasets for analytics consumption
• Experience implementing CI/CD practices for data pipelines and notebooks, including automated testing and deployment
• Proven ability to lead technical discussions, mentor team members, and drive engineering best practices
• Experience improving observability (metrics, alerts, dashboards) and reducing pipeline failures through proactive monitoring