Azure Databricks & PySpark Principal Consultant

Job Description:

• We are looking for an experienced Data Engineer with 12 16 years of overall IT experience and strong expertise in Azure Databricks and PySpark
• The ideal candidate should have extensive experience in building scalable data pipelines processing large datasets and delivering enterprise grade data solutions on the Azure platform

Key Responsibilities:

• Design develop and optimize large scale data pipelines using Azure Databricks and PySpark
• Build and maintain ETL ELT workflows using Azure Data Factory
• Develop scalable data processing frameworks for batch and near real time data ingestion
• Implement data transformation cleansing and enrichment processes
• Work with large scale structured and unstructured datasets in Azure Data Lake
• Optimize Spark jobs for performance scalability and cost efficiency
• Collaborate with business stakeholders architects and analytics teams to deliver data solutions
• Ensure data quality security governance and compliance standards are met
• Provide production support troubleshoot issues and drive continuous improvements
• Mentor junior team members and contribute to technical solution design

Technical Requirements:

• 12 16 years of IT experience with strong Data Engineering expertise
• Hands on experience in Azure Databricks and PySpark development
• Strong knowledge of Azure Data Factory ADF
• Expertise in SQL and Data Warehousing concepts
• Experience with Azure Data Lake Storage ADLS Gen2
• Strong understanding of Spark architecture and performance tuning
• Experience designing and implementing enterprise scale data pipelines
• Knowledge of CI CD and Agile delivery practices

Preferred Skills:

Technology->Data Engineering->Databricks,Technology->Big Data - Data Processing->PySpark,Technology->Cloud Platform->Azure Networking Services->Azure Bastion