1. Position Summary
The Data Engineering Lead is responsible for designing, building and operating data platforms and pipelines required for AI, analytics and automation use cases. The role leads a team of Data Engineers to implement the data architecture defined by the Architecture Lead and Solution Architect, ensuring reliable, high-quality and well-governed data for use cases such as Manufacturing Control Tower, Golden Batch, Yield Improvement, OOS prediction and pricing models.
2. Roles Played
- Data Engineering team lead for AI and analytics initiatives.
- Owner of data pipelines, transformations and data availability SLAs.
- Key contributor to data modelling, data quality and governance practices.
3. Key Platforms / Technologies
- Azure Data Lake, Data Warehouse, Synapse, Databricks (or equivalent).
- ETL/ELT tools and orchestration frameworks.
- Streaming and batch data ingestion frameworks.
- BI environments (Power BI, Tableau) as data consumers.
- Integration with SAP ECC or S/4 HANA , MES, LIMS, Empower, CTMS and other enterprise sources.
4. Overall Job Responsibilities
A. Data Platform & Pipeline Design ( 25%)
- Design data ingestion, transformation and storage solutions as per architecture guidelines.
- Build data models and structures to support AI, analytics and reporting use cases.
- Ensure solutions are modular, reusable and performance optimized.
B. Data Engineering Delivery & Operations ( 25%)
- Lead the implementation of data pipelines (batch and streaming) into the data lake/warehouse.
- Set up and maintain orchestration, monitoring, alerting and logging for data flows.
- Own day-to-day data operations and incident resolution related to pipelines.
C. Data Quality, Governance & Security ( 20%)
- Implement data quality rules, validations and reconciliations.
- Ensure data lineage and metadata capture for key datasets.
- Work with Security and Governance teams to enforce access control, privacy and data integrity requirements.
D. Performance, Optimization & Cost Management ( 10%)
- Optimize pipelines and queries for performance and cost (compute, storage).
- Periodically review resource utilization and recommend improvements.
E. Collaboration & Stakeholder Engagement ( 10%)
- Collaborate with Solution Architect, Data Scientists, Full Stack Developers and Business Analysts.
- Understand data needs of each use case and translate them into engineering tasks.
F. Team Leadership & Capability Building ( 10%)
- Lead and mentor Data Engineers and ETL developers.
- Promote best practices in coding, testing, documentation and DevOps for data.
5. External Interfaces
- Data platform vendors and implementation partners.
6. Internal Interfaces
- Solution Architect and Architecture Lead.
- BI Developers, Data Scientists and Modelers.
- Application team (SAP, MES, LIMS etc.), Infrastructure and Security.
8. Education
- Bachelor's degree in engineering / computer science / IT mandatory.
- Postgraduate qualification in Data Engineering / Data Science preferred.
9. Experience
- 812 years of experience in data engineering, ETL or data warehouse roles.
- Minimum 35 years leading data engineering teams or complex data projects.
- Hands-on experience with cloud data platforms (preferably Azure).
- Pharma or manufacturing data experience is an advantage.
10. Knowledge & Skills (Functional / Technical)
- Strong skills in SQL/NoSQL, Spark/Databricks and ETL tools (ADF, Data Build Tool, SSIS SQL Server Integration Services); streaming (Kafka/Event Hubs)
- Experience with Azure data lake / Storage and Microsoft Fabric Delta Lake, data warehouse architecture
- Familiarity with Python/Scala or other scripting for data pipelines.
- Understanding data quality, governance and metadata concepts.
- Exposure to integration with SAP, MES, LIMS or similar systems.
11. Leadership / Managerial Attributes
- Hands-on technical leader with a delivery focus.
- Good communication and coordination with multiple teams.
- Ability to mentor junior engineers and enforce engineering discipline.
12. Other Requirements
- Relevant cloud/data certifications (e.g., Azure Data Engineer) preferred.
- Willingness to support critical data operations outside normal hours when required.
