Required Qualifications

Expert-level Python for backend/data engineering
Hands-on GCP: Dataflow, BigQuery, Cloud Composer/Airflow, Cloud Functions, Cloud Run, GCS, IAM
Design and development of ETL/ELT pipelines (batch and streaming)
Apache Spark for large-scale processing
Apache Kafka for messaging/streaming
Orchestration with Airflow/Cloud Composer (DAG design, scheduling, monitoring)
Strong SQL with enterprise RDBMS (SQL Server, Oracle, PostgreSQL)
Git/GitHub and CI/CD for data projects
Deployments on GKE and Cloud Run; understanding of autoscaling and load balancers
API development with Python FastAPI; APIGEE proxy management
Data quality frameworks, validation rules, monitoring/observability
Security: IAM roles/policies, data redaction, DLP
Data ingestion from varied sources; transformation and cleansing
Documentation of pipelines, data flows, and operations
Experience migrating on-prem data to cloud

Responsibilities

Design, develop, test, and maintain scalable ETL/ELT data pipelines in Python
Architect data solutions using Kafka, GCP services (Dataflow, BigQuery, Composer/Airflow, Functions, Cloud Run/GKE, GCS, IAM), dbt, and related tools
Implement streaming and batch processing; ensure autoscaling, reliability, and cost efficiency
Develop and manage Airflow/Cloud Composer DAGs for orchestration (scheduling, monitoring, alerting)
Build APIs with FastAPI and manage APIs via APIGEE; integrate load balancers when required
Ingest data from diverse sources; apply transformation, cleansing, and enrichment
Implement data quality checks, validation rules, observability, and monitoring
Apply security best practices including IAM, data redaction, and DLP
Write and optimize complex SQL for extraction, validation, and analytics
Manage code in GitHub; implement CI/CD pipelines for data workloads
Deploy and operate workloads on GKE and Cloud Run
Collaborate with data science/analytics teams to deliver fit-for-purpose data products
Create and maintain documentation of pipeline designs, data flows, and operational runbooks
Lead or contribute to on-prem to cloud data migration initiatives

Pyspark Professional

GCP Data Engineer