We are looking for a highly skilled Lead Data Engineer with expertise in Azure, Python, Databricks, and ETL to drive data engineering initiatives. The ideal candidate will have strong leadership experience, hands-on technical skills, and a deep understanding of cloud-based data solutions.
Key Responsibilities:
- Lead the design, development, and implementation of scalable data pipelines using Azure Data Services, Databricks, and Python.
- Architect and optimize ETL workflows to ensure data integrity and efficiency.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and provide robust solutions.
- Manage and mentor a team of data engineers, providing guidance on best practices and performance optimization.
- Implement and enforce data governance, security, and compliance standards.
- Troubleshoot and resolve data pipeline issues, ensuring high availability and reliability.
- Optimize data storage, transformation, and retrieval for large datasets in a cloud environment.
Required Skills & Experience:
- 8+ years of experience in Data Engineering, with at least 2+ years in a lead role.
- Strong hands-on experience with Azure Data Services (Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake).
- Expertise in Databricks and Spark for big data processing.
- Proficiency in Python for data manipulation and pipeline development.
- Deep understanding of ETL/ELT processes and best practices.
- Experience with CI/CD pipelines and DevOps practices for data engineering.
- Strong knowledge of data modeling, warehousing concepts, and performance tuning.
- Excellent problem-solving, analytical, and communication skills.
