We are seeking a technically strong and delivery-focused Lead Data Engineer to support and enhance enterprise-grade data and application products. The ideal candidate will serve as the primary technical interface for the client, ensuring high system availability, performance, and continuous improvement. This role requires a hands-on technologist with proven leadership experience, cloud (AWS) expertise, and excellent communication skills to drive technical decisions and manage client interactions.
Main Responsibilities
Support & Enhancement Leadership
- Act as the primary technical lead for support and enhancement of assigned products.
- Ensure incident resolution, problem management, and enhancement delivery within agreed upon SLAs.
- Perform root cause analysis (RCA) and provide technical solutions to recurring issues.
- Design end-to-end data engineering solutions that are scalable and modular.
- Work in Agile implementations and lead all data engineering initiatives.
Technical Ownership
- Provide technical direction and architectural guidance for improvements, optimizations, and issue resolutions.
- Drive best practices in code performance tuning, ETL processing, and cloud-native data management.
- Lead the modernization of legacy data pipelines and applications by leveraging AWS services (Glue, Lambda, Redshift, S3, EMR, Athena, etc.).
- Utilize PySpark, SQL, and Python to manage big data processing pipelines efficiently.
Client Engagement
- Maintain high visibility with client stakeholders and act as a trusted technical advisor.
- Proactively identify and suggest areas for improvement or innovation to enhance business outcomes.
- Participate in daily stand-ups, retrospectives, and client presentations, communicating complex technical concepts clearly.
Team & Delivery Management
- Lead a cross-functional team of engineers, providing mentorship and driving upskilling.
- Monitor team performance, support capacity planning, and ensure timely, high-quality deliveries.
- Enforce adherence to governance, documentation, and change management practices.
Process & Quality Assurance
- Implement and ensure compliance with engineering best practices, including CI/CD, version control, and automated testing.
- Define support procedures and documentation standards.
- Identify risk areas and dependencies, and propose mitigation strategies.
Required Skills & Qualifications
- Extensive experience in Data Engineering or Application Support and Development.
- Deep hands-on expertise in the AWS ecosystem (Glue, Lambda, Redshift, S3, Athena, EMR, CloudWatch).
- Proficiency in PySpark, SQL, and Python, with experience in handling big data pipelines.
- Strong application debugging skills across batch and near real-time systems.
- Good knowledge of incident lifecycle management, RCA, and performance optimization.
- Proven leadership experience leading engineering teams, preferably in support/enhancement environments.
- Excellent communication and client-facing skills.
- Experience with tools like JIRA, Confluence, Git, and Jenkins.
Good to Have
- Experience in on-prem to AWS migration projects.
- Familiarity with legacy technologies and their interaction with modern cloud stacks.
- Good knowledge of designing Hive tables with partitioning for performance.
