ADF (Azure Data Factory)Databricks+Pyspark

Job Description:

• You ll lead the design and delivery of modern data engineering solutions that turn raw data into trusted analytics ready assets
• Working at the intersection of orchestration scalable processing and cloud native platforms you ll partner closely with product owners analysts and engineering teams to build reliable pipelines that power business decisions
• This role is ideal for someone who enjoys owning end to end delivery shaping architecture guiding implementation and mentoring engineers while continuously improving performance quality and operational excellence
• If you re excited by solving complex data challenges enabling self serve analytics and building a collaborative culture that values craftsmanship and learning this is the place to make a meaningful impact

Key Responsibilities:

• Key Responsibilities
• Data Engineering Delivery
• Lead end to end development of data pipelines using ADF for orchestration and Databricks for scalable processing
• Design and implement robust ETL ELT workflows ensuring data quality reliability and maintainability
• Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing
• Build reusable frameworks templates and standards for pipeline development and deployment
• Architecture Performance
• Define solution architecture for ingestion transformation and serving layers aligned to platform best practices
• Tune Spark jobs for performance and cost efficiency partitioning caching shuffle optimization file sizing
• Establish monitoring alerting and operational runbooks for production pipelines
• Leadership Collaboration
• Provide technical leadership code reviews and mentoring to ensure high engineering standards
• Collaborate with stakeholders to translate business requirements into scalable data solutions
• Drive delivery planning estimation and risk management for data engineering initiatives
• Minimum Qualifications
• BTECH MTECH MCA MSC or equivalent in Computer Science Engineering or related field
• 7 9 years of experience in data engineering with strong hands on delivery ownership
• Strong expertise in Azure Data Factory ADF for pipeline orchestration scheduling and integration patterns
• Strong expertise in Databricks for building scalable data processing solutions
• Hands on proficiency with PySpark for building and optimizing distributed data transformations
• Experience building production grade pipelines with logging error handling and operational support readiness

Technical Requirements:

• Technology Big Data Data Processing PySpark
• Technology Cloud Integration Azure Data Factory ADF
• Technology Data Engineering Databricks

Additional Responsibilities:

• Preferred Qualifications
• Experience designing medallion layered data architectures and implementing reusable transformation patterns in Databricks
• Strong understanding of data modeling concepts and building curated datasets for analytics consumption
• Experience implementing CI CD practices for data pipelines and notebooks including automated testing and deployment
• Proven ability to lead technical discussions mentor team members and drive engineering best practices
• Experience improving observability metrics alerts dashboards and reducing pipeline failures through proactive monitoring

Preferred Skills:

Technology->Cloud Integration->Azure Data Factory (ADF),Technology->Data Engineering->Databricks,Technology->Big Data - Data Processing->PySpark