Key Responsibilities:
- Design and develop scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
- Implement and manage data storage solutions using Azure Data Lake Storage (ADLS).
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
- Optimize data workflows for performance, reliability, and cost-efficiency.
- Develop and maintain CI/CD pipelines for data solutions using tools like Azure DevOps or GitHub Actions.
- Ensure data security, governance, and compliance standards are met.
- Troubleshoot and resolve issues related to data quality, performance, and system integration.
Required Skills:
- 6–8 years of overall experience in data engineering or related roles.
- Strong hands-on experience with Azure Data Factory (ADF), Azure Databricks, and ADLS.
- Proficient in PySpark, SQL, and data modeling techniques.
- Experience in implementing CI/CD pipelines in Azure environments.
- Solid understanding of ETL/ELT processes and best practices.
- Familiarity with Azure DevOps, Git, and monitoring tools like Azure Monitor or Log Analytics.
- Good problem-solving skills and attention to detail.
- Excellent communication and collaboration abilities.
Preferred Qualifications:
- Azure certifications such as DP-203: Azure Data Engineer Associate.
