Job Description for Lead Data Engineer
Rank Manager
Objectives and Purpose
The Lead Data Engineer leads large scale solution architecture design and optimisation to provide streamlined insights to partners throughout the business. This individual leads the team of Mid- and Senior data engineers to partner with visualization on data quality and troubleshooting needs.
The Lead Data Engineer will:
- Implement data processes for the data warehouse and internal systems
- Lead a team of Junior and Senior Data Engineers in executing data processes and providing quality, timely data management
- Manage data architecture, design ETL processes
- Clean, aggregate and organize data from disparate sources and transfer it to data warehouses
- Lead development, testing and maintenance of data pipelines and platforms, to enable data quality to be utilized within business dashboards and tools
- Support team members and direct reports in refining and validating data sets
- Create, maintain, and support the data platform and infrastructure that enables the analytics front-end; this includes the testing, maintenance, construction, and development of architectures such as high-volume, large-scale data processing and databases with proper verification and validation processes
Your Key Responsibilities
Data Engineering
- Lead the design, development, optimization, and maintenance of data architecture and pipelines adhering to ETL principles and business goals
- Develop and maintain scalable data pipelines, build out new integrations using AWS native technologies and Databricks
- Define data requirements, gather and mine large scale structured and unstructured data, and validate data in the Big Data Environment
- Lead ad hoc data analysis, support standardization/customization and develop mechanisms to ingest, analyze, validate, normalize, and clean data
- Write unit/integration/performance test scripts and troubleshoot data-related issues
- Implement processes and systems to drive data reconciliation and monitor data quality, ensuring accurate and available production data
- Lead evaluation and deployment of emerging tools for analytic data engineering
- Develop communication and education plans for data engineering capabilities
- Solve complex data problems to deliver business insights
- Partner with Business Analysts and Enterprise Architects for strategic technical architecture
- Coordinate with Data Scientists, visualization developers, and other data consumers
Relationship Building and Collaboration
- Collaborate across technical and business teams to develop scalable analytics and machine learning solutions
- Support Data Scientists in data sourcing/preparation for visualization and insights
- Mentor other data and analytics professionals on best practices
- Foster a collaborative culture of reusable design and scalable solutions
Skills and Attributes for Success
Technical/Functional Expertise
- Advanced experience in Big Data, data integration, data modeling, AWS, cloud technologies
- Business acumen, preferably in Pharmaceutical, Healthcare, or Life Sciences
- Strong in SQL, data transformations, metadata management
- Familiar with Agile methodologies
Leadership
- Mentor Senior and Junior Engineers
- Strategic thinker aligned with long-term business goals
- Promote psychological safety and team collaboration
Decision-making and Autonomy
- Promote data-driven decisions over manual decisions
- Strong problem-solving and critical thinking skills
Interaction
- Proven ability to develop trust-based relationships with stakeholders
- Work across IT functions and vendors
Innovation
- Advocate for disruptive technologies and advanced data solutions
- Lead R&D efforts in data engineering
- Embrace a growth mindset and continuous learning
Complexity
- Culturally sensitive and effective across global teams
- Skilled in large-team coordination and problem solving
To Qualify for the Role, You Must Have the Following:
Essential Skillsets
- Bachelor's in Engineering, Computer Science, Data Science, or related
- 9+ years in software development, data engineering, ETL, and reporting
- Expert in dimensional data modeling, ETL pipelines
- Hands-on with data mesh, data fabric, data product design
- Knowledge of structured/unstructured data frameworks
- Proven complex data solutions design and implementation
Technical Experience With:
- Python, PySpark
- Kubernetes, AWS (Lambda, S3, DMS, Step Functions, Event Bridge, CloudWatch, RDS)
- Databricks, IICS/DMS, GitHub, SQL
- Lakehouse/Data Lake architecture
- CI/CD using GitHub Actions
- Optimizing Python/Spark scripts and AWS/Databricks cloud costs
- Strong communication and organizational skills
Desired Skillsets
- Master's degree in a relevant field
- Global working environment experience
Travel Requirements
- Access to transportation for meetings
- Willingness to travel regionally and globally
About EY
EY | Building a better working world
- EY exists to create long-term value for clients, people, and society
- Operating in over 150 countries
- Offering services in assurance, consulting, law, strategy, tax, and transactions
- Driving transformation with data, technology, and diversity
