g large datasets and delivering enterprise grade data solutions on the Azure platform

Key Responsibilities:

Design develop and optimize large scale data pipelines using Azure Databricks and PySpark
Build and maintain ETL ELT workflows using Azure Data Factory
Develop scalable data processing frameworks for batch and near real time data ingestion
Implement data transformation cleansing and enrichment processes
Work with large scale structured and unstructured datasets in Azure Data Lake
Optimize Spark jobs for performance scalability and cost efficiency
Collaborate with business stakeholders architects and analytics teams to deliver data solutions
Ensure data quality security governance and compliance standards are met
Provide production support troubleshoot issues and drive continuous improvements
Mentor junior team members and contribute to technical solution design

Technical Requirements:

12 16 years of IT experience with strong Data Engineering expertise
Hands on experience in Azure Databricks and PySpark development
Strong knowledge of Azure Data Factory ADF
Expertise in SQL and Data Warehousing concepts
Experience with Azure Data Lake Storage ADLS Gen2
Strong understanding of Spark architecture and performance tuning
Experience designing and implementing enterprise scale data pipelines
Knowledge of CI CD and Agile delivery practices

Preferred Skills:

Technology->Data Engineering->Databricks,Technology->Big Data - Data Processing->PySpark,Technology->Cloud Platform->Azure Networking Services->Azure Bastion

Azure Databricks & PySpark Principal Consultant