Job Summary
We are seeking a highly skilled GCP Lead Data Engineer to design, build, and optimize scalable data pipelines and data platforms on Google Cloud Platform (GCP). The ideal candidate will have strong expertise in Python, SQL, and PySpark, along with hands-on experience in GCP services such as BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Cloud Composer. This role involves leading data engineering initiatives, ensuring high-quality data delivery, and enabling advanced analytics and business intelligence solutions.
Key Responsibilities
- Design, develop, and maintain scalable and reliable data pipelines using Python, SQL, and PySpark.
- Build and optimize batch and real-time data processing pipelines using GCP services such as Dataflow and Pub/Sub.
- Develop and manage data warehouses and data lakes using BigQuery and Cloud Storage.
- Implement workflow orchestration using Cloud Composer (Apache Airflow).
- Ensure data quality, integrity, security, and governance across data platforms.
- Optimize query performance, data models, and pipeline efficiency.
- Lead architecture discussions and provide technical guidance to data engineering teams.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver scalable data solutions.
- Monitor, troubleshoot, and resolve pipeline and infrastructure issues.
- Implement best practices for CI/CD, automation, and infrastructure-as-code where applicable.
Required Skills & Qualifications
- 7+ years of experience in Data Engineering, with strong exposure to GCP.
- Hands-on experience with Python and SQL for data processing and transformation.
- Strong experience with PySpark and distributed data processing frameworks.
- Expertise in GCP services, including:
- BigQuery
- Dataflow
- Pub/Sub
- Cloud Storage
- Cloud Composer (Airflow)
- Experience building ETL/ELT pipelines and modern data platforms.
- Strong understanding of data warehousing concepts and data modeling.
- Experience with real-time and streaming data pipelines is a plus.
- Strong problem-solving and performance optimization skills.
- Excellent communication and leadership abilities.
Preferred Qualifications
- Experience leading or mentoring data engineering teams.
- GCP Professional Data Engineer certification.
- Experience with CI/CD, Terraform, or infrastructure automation.
- Knowledge of data governance, security, and compliance best practices.
