Role description

Deliver and operate scalable reliable data engineering solutions on Databricks to support analytics AI and downstream consumption

Key Responsibilities

Design and build ETLELT pipelines using Databricks PySpark and SQL

Implement batch and incremental data processing patterns for enterprise datasets

Apply data engineering best practices for performance scalability and reliability

Develop and optimize Spark jobs notebooks and workflows in Databricks

Ensure data quality validation and reconciliation across pipelines

Support data modeling and transformation for analytics and reporting use cases

Integrate Databricks pipelines with upstream and downstream systems

Participate in production support monitoring and incident resolution

Core Skills Required

Strong handson experience with Databricks and Apache PySpark

Programming skills in Python PySpark and SQL for efficient data transformation and processing ETL

Solid understanding of ETLELT patterns and data pipeline orchestration

Big Data Processing Experience with big data frameworks specifically Apache Spark for processing large datasets

Data Warehousing Modelling Understanding of data warehousing principles and scalable data modelling techniques

Experience with data engineering best practices and optimization techniques

Handson exposure to enterprise data engineering workflows

Experience supporting production data pipelines at scale

Familiarity with analytics reporting or AIdriven data consumption use cases

Good to have

Experience in working with Opensource storage formats such as Apache Delta Apache Parquet would be beneficial

Awareness of compliance regulations relevant to data handling and processing

Knowledge of data security practices including encryption access control and data governance

Knowledge of other Azure services that integrate with ADF and ADB such as Azure Functions Azure Logic Apps and Azure Event Hubs

Senior Specialist - Data Engineering