Job Summary
We are seeking a highly skilled Senior Azure Data Engineer to design, develop, and optimize scalable data solutions on the Microsoft Azure platform. The ideal candidate will have strong hands-on experience with Azure Databricks, Delta Lake, SQL, PySpark, and Azure Data Services, along with expertise in building robust ETL/ELT pipelines and modern data lake architectures. Exposure to AI/ML data platforms will be an added advantage.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure services.
- Build and optimize data processing workflows using Azure Databricks and PySpark.
- Implement and manage Delta Lake-based data architectures.
- Develop and optimize SQL queries, stored procedures, and data models.
- Ingest, transform, and process large volumes of structured and semi-structured data.
- Work with ADLS Gen2, Azure Data Factory (ADF), and Azure Synapse Analytics.
- Ensure data quality, governance, security, and performance optimization.
- Collaborate with BI, Analytics, and Data Science teams to deliver reliable datasets.
- Implement CI/CD processes and support deployment automation.
- Troubleshoot production issues and optimize data pipelines for performance and scalability.
Mandatory Skills
- Azure Databricks
- Delta Lake
- SQL
- PySpark
- Azure Data Factory (ADF)
- ADLS Gen2
- Azure Data Engineering
- ETL/ELT Pipeline Development
- Data Warehousing Concepts
- Azure Synapse Analytics
Preferred Skills
- AI/ML Data Engineering Exposure
- MLflow
- Feature Store
- Microsoft Fabric
- Data Governance Tools
- Power BI
- Azure DevOps & CI/CD
Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- Minimum 5 years of hands-on experience in Azure Data Engineering.
- Strong expertise in Azure Databricks, Delta Lake, SQL, and PySpark.
- Experience working with large-scale enterprise data platforms.
- Excellent analytical and problem-solving skills.
