GCP & PySpark with ETL - Lead (Kochi)

Job description

Key Responsibilities

• Design, develop, and optimize ETL pipelines using PySpark on Google Cloud Platform (GCP).
• Work with BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration.
• Develop and optimize Spark-based ETL processes for large-scale data processing.
• Implement best practices for data governance, security, and monitoring in a cloud environment.
• Collaborate with data engineers, analysts, and business stakeholders to understand data requirements.
• Troubleshoot performance bottlenecks and optimize Spark jobs for efficient execution.
• Automate data workflows using Apache Airflow or Cloud Composer.
• Ensure data quality, validation, and consistency across pipelines.
• 5+ years of experience in ETL development with a focus on PySpark.
• Strong hands-on experience with Google Cloud Platform (GCP) services, including:
• BigQuery
• Cloud Dataflow / Apache Beam
• Cloud Composer (Apache Airflow)
• Cloud Storage
• Proficiency in Python and PySpark for big data processing.
• Experience with data lake architectures and data warehousing concepts.
• Knowledge of SQL for data querying and transformation.
• Experience with CI/CD pipelines for data pipeline automation.
• Strong debugging and problem-solving skills.
• Experience with Kafka or Pub/Sub for real-time data processing.
• Knowledge of Terraform for infrastructure automation on GCP.
• Experience with containerization (Docker, Kubernetes).
• Familiarity with DevOps and monitoring tools like Prometheus, Stackdriver, or Datadog.

Skills

Gcp,Pyspark,Etl