We are looking for a Senior Data Engineer with strong hands-on experience in Azure-based data engineering tools and technologies. This role will focus on building and optimizing scalable data pipelines using Azure Databricks, ADF, ADLS Gen2, and PySpark, while collaborating with cross-functional teams to deliver enterprise-grade data solutions.
Key Responsibilities:
- Design and implement scalable ETL pipelines and robust data solutions in the Azure cloud environment.
- Manage data orchestration using Azure Data Factory (ADF) and Azure Databricks.
- Build and maintain secure and efficient Data Lake architectures using ADLS Gen2 and Azure Key Vaults.
- Translate business data requirements into technical specifications and deliverables.
- Create and manage CI/CD pipelines for data processes using Azure DevOps.
- Ensure data quality, monitor performance, and optimize pipelines for scalability and efficiency.
- Write clean, reusable PySpark code following best practices in a fast-paced Agile environment.
- Produce documentation for pipelines, architectural decisions, and reusable components.
Must-Have Skills:
- 6+ years of experience in Data Engineering.
- Strong proficiency in:
- SQL, Python, PySpark, Spark
- Azure Databricks, ADF, ADLS Gen2, Azure DevOps, Key Vaults
- Proven experience with:
- Data Warehousing
- ETL & Data Pipeline Development
- Data Modeling & Data Governance
- Understanding of Agile methodologies, SDLC, and containerization (e.g., Docker).
- Emphasis on clean code, modularity, and performance tuning.
