Role: Senior Data Engineer – GCP Data Platform
Objective of the Role
We are looking for a hands-on Senior Data Engineer to design and build scalable data solutions and reusable data-platform capabilities on Google Cloud Platform (GCP).
This role combines Data Engineering and Data Platform Engineering. You will provide technical leadership for complex data-processing capabilities while remaining actively involved in architecture, design and implementation.
You will help define how data products are transformed, orchestrated, governed and operated at scale, with a strong focus on reusability, automation, performance, reliability and engineering standards.
You Will
- Own the technical design and architecture of scalable data-processing and transformation solutions on GCP.
- Design and build production-grade batch, incremental and streaming data pipelines.
- Develop and optimise complex transformations using BigQuery, SQL and Dataform / dbt.
- Build reusable, metadata-driven and configuration-driven frameworks that reduce bespoke pipeline development.
- Design orchestration patterns using Cloud Composer / Apache Airflow, including dynamic workflows, dependencies, retries and recovery.
- Develop distributed and streaming processing solutions using Dataflow / Apache Beam and Dataproc / Spark.
- Define standards for data modelling, schema design, schema evolution and data contracts.
- Design and evolve modern analytical storage patterns, including Apache Iceberg.
- Establish engineering patterns for data quality, reconciliation, metadata, lineage and observability.
- Optimise large-scale workloads for performance, scalability, latency and GCP cost.
- Define reusable engineering standards for testing, versioning, CI/CD and production readiness.
- Lead technical design and code reviews and guide engineers on complex implementation decisions.
- Identify systemic technical debt and drive improvements to shared platform capabilities.
- Work closely with Data Engineering, Platform Engineering, DevOps, Architecture, Governance and Product teams.
- Take technical ownership from solution design through production deployment and operational stability.
You Must Have
- 6+ years of hands-on experience designing and building enterprise-scale data platforms and data products.
- Strong production experience with Google Cloud Platform (GCP).
- Deep hands-on experience with BigQuery, including data modelling, partitioning, clustering, query optimisation and cost optimisation.
- Advanced SQL skills for complex transformation and analytical workloads.
- Strong programming experience with Python, including modular development, testing and automation.
- Strong experience with Dataform and/or dbt.
- Hands-on experience with Cloud Composer / Apache Airflow and complex DAG/workflow design.
- Hands-on experience with Google Cloud Dataflow / Apache Beam.
- Experience with Dataproc and Spark / PySpark for distributed processing.
- Experience with Apache Iceberg or modern lakehouse table formats.
- Strong understanding of batch, incremental, idempotent and streaming processing patterns.
- Experience designing reusable data-processing frameworks rather than only individual pipelines.
- Experience with metadata-driven and configuration-driven processing.
- Strong data-modelling skills, including dimensional, normalized and denormalized models.
- Experience defining and implementing data contracts and schema-management patterns.
- Strong understanding of data quality, metadata, lineage and observability.
- Experience with Git, automated testing and CI/CD for data workloads.
- Experience designing solutions for reliability, recoverability and production operations.
- Ability to independently make architectural decisions and communicate technical trade-offs.
Technical Skills
Cloud & Storage
- Google Cloud Platform
- BigQuery
- Google Cloud Storage
- Apache Iceberg
Transformation & Orchestration
- Advanced SQL
- Dataform / dbt
- Cloud Composer / Apache Airflow
Distributed & Streaming Processing
- Dataflow
- Apache Beam
- Dataproc
- Spark / PySpark
Programming & Engineering
- Python
- Git
- CI/CD
- Automated testing
- Monitoring and observability
Data Platform Capabilities
- Batch and incremental processing
- Streaming and event processing
- Data modelling
- Data contracts and schema management
- Metadata-driven processing
- Configuration-driven processing
- Data quality and reconciliation
- Metadata and lineage
- Performance and cost optimisation
- Reusable platform frameworks
Good to Have
- Practical experience with Data Product / Data Mesh principles.
- Experience developing self-service or internal data-platform capabilities.
- Experience with Change Data Capture and event-driven architectures.
- Exposure to semantic and metric modelling.
- Experience with Terraform / Infrastructure as Code.
- Experience working with multi-domain or multi-market enterprise data platforms.
Strong Interpersonal Skills
- Strong technical ownership and ability to drive complex initiatives from concept to production.
- Ability to lead technical discussions and communicate architectural decisions clearly.
- Strong problem-solving and production troubleshooting skills.
- Ability to mentor engineers and improve engineering practices across a team.
- Comfortable collaborating with engineering, architecture, product and business stakeholders.
- Ability to work effectively within distributed and multicultural teams.
