Job Title: Lead Data Engineer GCP (BigQuery Composer Python PySpark)
Location: Gurgaon
Experience: 8+ years (data engineering / analytics engineering), with previous lead responsibilities
About the Role:
You will lead the design, build and operation of large-scale data platforms on the Google Cloud Platform. You will manage a team of data engineers, collaborate with analytics & data science stakeholders, define best-practices, and ensure robust, production-ready pipelines that support business insights and data products.
Key Responsibilities:
- Architect and implement end-to-end data pipelines using BigQuery, PySpark/Spark, Python.
- Use Cloud Composer (Airflow) to orchestrate workflows, manage DAGs, scheduling, dependencies and monitoring.
- Design and optimise BigQuery usage (partitioning, clustering, cost-control, performance tuning).
- Develop and manage large-scale Spark/PySpark jobs for batch and potentially streaming data.
- Ensure cloud-native architecture on GCP: storage (Cloud Storage), compute (Dataproc/Dataflow), orchestration (Composer), security/iam.
- Lead and mentor a team of data engineers: code reviews, best practices, architecture decisions.
Required Skills & Qualifications:
- Minimum 8 years of experience in data engineering or similar, ideally with lead responsibilities.
- Hands-on experience in GCP servicesespecially BigQuery (data warehousing, optimisation), Cloud Composer (Airflow) for orchestration.
- Strong experience with PySpark/Spark for large-scale data processing.
- Excellent proficiency in Python (scripting, automation, building frameworks).
- Strong SQL skills (in BigQuery or similar cloud data warehouse), including complex queries, optimisation.
- Experience designing data models, data warehouses/lakes, ingestion pipelines, transformations.
- Experience building/operating CI/CD pipelines, version control (Git), test/monitoring practices.
- Excellent communication and leadership skillsable to mentor, lead architecture discussions and engage stakeholders.
Why Join Us:
- Lead a talented team and influence data engineering architecture & tooling.
- Work on high-impact data platforms that enable business growth.
- Use cutting-edge GCP services and modern data engineering practices.
