As a Lead Data Engineer, you will provide technical leadership, incident management expertise, and strategic oversight in ensuring the reliability, security, and scalability of mission-critical data platforms. You will be responsible for defining and implementing best practices, influencing key stakeholders, and guiding the team in solving complex, high-priority technical challenges.
Job responsibilities
- Work closely with cross-functional teams, including client systems and data engineers, to diagnose and resolve incidents impacting data operations, ensuring minimal business disruption.
- Monitor, troubleshoot, and optimize data pipelines to ensure reliable and timely data delivery, addressing any issues encountered in production environments.
- Use incident management tools, data monitoring metrics, and reports to detect issues and take corrective actions.
- Enforce and monitor data quality metrics to maintain consistency, accuracy, and completeness across systems.
- Leverage logging techniques for alerting, monitoring, and identifying root causes in data platforms and pipelines.
- Optimize and modify data pipelines for scalability and performance improvements.
- Support deployment of new features, bug fixes, and enhancements, ensuring smooth production integration and minimal risk.
- Maintain documentation (troubleshooting guides, workflows, best practices) and mentor junior peers through technical and leadership guidance.
- Follow standards and best practices to enhance operational efficiency, system stability, and availability.
- Apply continuous delivery practices to support and deliver high-quality data solutions and customer value early and often.
Job qualifications
Technical Skills
- Experience with data projects, including data frameworks, ETL processes, and data visualization.
- Skilled in working with data models and pipelines to support smooth and efficient operations.
- Proficient in MSSQL databases and AWS cloud-native services.
- Strong hands-on experience with Python and Shell scripting.
- Possess strong debugging and triaging skills to troubleshoot data issues effectively.
- Experience building and supporting APIs within microservice architectures.
- Familiar with data processing frameworks such as Apache Spark and Hadoop.
- Practical experience with data science tools, including AWS SageMaker.
- Comfortable working in Agile environments, including Scrum and/or Kanban.
- (Desirable) Exposure to Java (Spring Boot), frontend frameworks like Angular/React, and knowledge of architectural styles such as monolithic, layered, microservices, and serverless.
Professional Skills
- Strong communicator with the ability to articulate ideas clearly and influence others toward technical excellence.
- Open-minded and adaptable to change while promoting best practices.
- Resilient and able to navigate ambiguity by approaching challenges from different angles.
- Experienced in mentoring and guiding junior consultants.
- Willing to participate in a rotation-based, 24/7 support schedule as needed.
Other things to know
Learning and development
There is no one-size-fits-all career path at Thoughtworks: However you want to develop your career is entirely up to you. But we also balance autonomy with the strength of our cultivation culture. This means your career is supported by interactive tools, numerous development programs and teammates who want to help you grow. We see value in helping each other be our best and that extends to empowering our employees in their career journeys.
AI for recruitment
At Thoughtworks, we use AI tools to support our recruitment team with administrative tasks such as drafting communications, scheduling interviews and writing job descriptions.
Crucially, our AI tools do not screen, assess, rank or make hiring decisions. Every application is reviewed by our team and all selection decisions are made exclusively by our interviewers and hiring managers.
We are committed to fairness and responsible AI. We actively manage our AI systems by testing, monitoring for biased outcomes and implementing mitigation measures. We hold our third-party vendors to these same high standards through a rigorous governance process. For additional information, please see our full Thoughtworks AI Policy for Recruitment.
About Thoughtworks
Thoughtworks is a dynamic and inclusive community of bright and supportive colleagues who are revolutionizing tech. As a leading technology consultancy, we're pushing boundaries through our purposeful and impactful work. For 30+ years, we've delivered extraordinary impact together with our clients by helping them solve complex business problems with technology as the differentiator. Bring your brilliant expertise and commitment for continuous learning to Thoughtworks. Together, let's be extraordinary.
