Job Description
Azure Data Engineering is responsible for leading the design, delivery, and operational excellence of enterprise-scale data platforms on Microsoft Azure. This role combines hands-on technical leadership, team management, and stakeholder engagement, with a strong focus on enabling AI and Generative AI (GenAI) initiatives through high-quality, secure, and reliable data pipelines.
The Assistant Manager oversees data engineering teams, defines technical standards, and ensures that data platforms effectively support LLM training, embeddings, inference pipelines, and AI-enabled applications, while meeting enterprise security, governance, and performance expectations.
Interact with multiple stakeholders and manage the end-to-end data project delivery from project requirements, solutioning and execution and hand off to operations.
Responsibilities
- Own the architecture, design, and delivery of scalable data storage and processing solutions using Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, and Azure Databricks.
- Define and enforce data engineering standards, best practices, and design patterns across teams.
- Provide technical leadership and oversight for complex ETL/ELT pipelines.
- Lead data engineering support for AI and GenAI workloads, including preparation and management of training datasets, embeddings, and inference pipelines.
- Partner with data scientists, AI engineers, and solution architects to meet AI performance and governance requirements.
- Manage, mentor, and grow a team of Azure Data Engineers.
- Plan capacity, allocate work, and ensure timely delivery of initiatives.
- Drive adherence to Agile delivery practices.
- Drive optimization of storage, compute, and pipelines for performance and cost efficiency.
- Review and approve designs with long-term sustainability in mind.
Qualifications
- Bachelor's degree in computer science, Engineering, Information Systems, or a related technical field, or equivalent practical experience.
- Experience with Azure cloud services, specifically data storage and data processing solutions.
- Strong knowledge of data warehousing and ETL (Extract, Transform, Load) concepts.
- Handson experience with SQL, Python and PySpark.
- Understanding of GenAI and LLM data concepts, including training data, embeddings, and inference pipelines.
- Strong understanding of data security and privacy best practice.
- Experience with monitoring and troubleshooting data pipelines.
- Strong problemsolving and analytical skills.
- Effective communication skills to collaborate with crossfunctional teams.
- Experience with data governance is a plus.
- Proven experience leading and mentoring teams.
- Strong communication and stakeholder management skills.
#KGS
