Job Description – Senior Data Engineer (Technical Lead)
Position: Senior Data Engineer – Technical Lead
Experience: 10+ Years
Location: Remote (Candidate must collect company assets from the client location.)
Shift Timing: 2:00 PM – 11:00 PM IST
Mandatory Skills
- SQL Server
- Python
- Apache Airflow
- Hadoop
- Google BigQuery
- Data Lake
Job Summary
We are seeking an experienced Senior Data Engineer – Technical Lead to lead a Business-as-Usual (BAU) Data Engineering team responsible for maintaining, enhancing, and supporting production data pipelines on Google Cloud Platform (GCP).
The ideal candidate will combine strong technical expertise with leadership capabilities, ensuring the stability, reliability, and continuous improvement of enterprise data platforms. This role requires hands-on involvement in production support, data pipeline optimisation, mentoring engineers, and driving operational excellence across the complete data lifecycle.
Key ResponsibilitiesTechnical Leadership
- Lead and mentor the offshore Data Engineering team while ensuring effective collaboration with US stakeholders.
- Provide technical guidance, code reviews, and best practices for production support activities.
- Act as the primary escalation point for complex production issues.
Production Support & Incident Management
- Troubleshoot and resolve production pipeline failures and data-related issues.
- Perform root cause analysis and implement permanent solutions to recurring problems.
- Ensure high availability, reliability, and performance of production data pipelines.
Data Pipeline Management
- Manage and optimise data pipelines across the complete lifecycle:
- Data Lake
- Dimensions & Fact Tables
- Enterprise Data Warehouse
- Implement SQL fixes, DAG enhancements, data backfills, and validation processes.
- Optimise Airflow workflows for scalability and performance.
Change & Release Management
- Ensure production deployments follow strict change management processes.
- Validate deployment evidence, testing results, and rollback strategies before production releases.
- Maintain deployment documentation and operational standards.
Continuous Improvement
- Identify automation opportunities to reduce manual effort and technical debt.
- Develop scalable solutions using Python and SQL.
- Improve monitoring, alerting, and operational efficiency of data pipelines.
Required Skills & Experience
- 10+ years of hands-on experience in Data Engineering.
- Proven experience leading Data Engineering or Technical Support teams.
- Strong expertise in Google BigQuery, including:
- Slowly Changing Dimensions (SCD Type 1 & Type 2)
- Partitioning
- Clustering
- Query and cost optimisation
- Extensive experience with Apache Airflow / Cloud Composer.
- Strong knowledge of Jinja templating for Airflow DAG development.
- Advanced SQL development and performance tuning skills.
- Strong proficiency in Python for automation, ETL development, API integrations, and data processing.
- Experience with SQL Server, Hadoop, and enterprise Data Lake architectures.
- Excellent understanding of production support processes, incident management, and ITIL-based change control.
- Strong communication and stakeholder management skills with the ability to collaborate effectively across global teams and support overlap with US business hours.
Preferred Qualifications
- Strong understanding of Kimball Data Warehouse Methodology, including:
- Star Schema Design
- Fact Tables
- Dimension Tables
- Experience working with Salesforce as a source system.
- Prior experience leading offshore BAU or Production Support teams.
- Exposure to enterprise-scale cloud data platforms on Google Cloud Platform (GCP).
Key Competencies
- Technical Leadership
- Production Support
- Data Engineering
- BigQuery
- Airflow
- Python
- SQL Optimisation
- Data Warehousing
- Change Management
- Problem Solving
- Team Mentoring
- Stakeholder Communication
