GCP Data Engineer

Job Description:

• Strong hands on GCP Data Engineering experience across scalable cloud data platforms
• Expertise in Google Cloud Storage BigQuery Dataflow Dataproc Pub Sub Cloud Composer Cloud Functions Cloud Monitoring IAM and Secret Manager
• Strong experience in batch and real time streaming using Pub Sub Dataflow Apache Beam Spark Structured Streaming or Kafka
• Hands on coding proficiency in Python PySpark and SQL including debugging and performance optimization
• Experience building high volume low latency ingestion and transformation pipelines with retry idempotency deduplication and dead letter handling
• Strong experience in Databricks Delta Lake Structured Streaming Auto Loader and Spark performance tuning
• Experience handling TB PB scale datasets partitioning clustering file compaction schema evolution and data lifecycle management
• Strong understanding of Data Lake Lakehouse and modern cloud data platform architectures
• Practical data modeling experience for analytics reporting and downstream consumption
• Experience implementing CI CD using Cloud Build GitHub Actions Jenkins or equivalent with Terraform for Infrastructure as Code
• Strong troubleshooting observability production support and cloud cost optimization skills
• Excellent client communication stakeholder management and problem solving skills

Key Responsibilities:

• Responsibilities and JD in brief along with additional criteria to be considered
• Design and develop scalable GCP Data Engineering solutions and cloud native data platforms
• Build batch and real time pipelines using BigQuery Cloud Storage Dataflow Pub Sub Dataproc Python PySpark and Databricks
• Develop and optimize Databricks Delta Lake Spark and Structured Streaming workloads for large scale processing
• Implement high throughput streaming solutions using Pub Sub and Dataflow including windowing watermarking late arriving data duplicate handling and dead letter queues
• Handle high volume TB PB scale data with focus on performance scalability reliability and cost optimization
• Develop complex SQL transformations data models and analytics ready BigQuery datasets using partitioning and clustering
• Orchestrate pipelines using Cloud Composer Airflow and implement CI CD and Terraform based Infrastructure as Code
• Ensure data quality governance security monitoring alerting and production support
• Collaborate with architects stakeholders and cross functional teams in an Agile delivery model
• Troubleshoot and optimize end to end GCP Databricks and streaming data pipelines

Technical Requirements:

• GCP BigQuery Cloud Storage Dataflow Pub Sub Dataproc batch and real time streaming Databricks Pyspark
• Python strong SQL
• Databricks with Delta Lake and Structured Streaming
• Responsibilities and JD in brief along with additional criteria to be considered
• Design and develop scalable GCP Data Engineering solutions and cloud native data platforms
• Build batch and real time pipelines using BigQuery Cloud Storage Dataflow Pub Sub Dataproc Python PySpark and Databricks
• Develop and optimize Databricks Delta Lake Spark and Structured Streaming workloads for large scale processing
• Implement high throughput streaming solutions using Pub Sub and Dataflow including windowing watermarking late arriving data duplicate handling and dead letter queues
• Handle high volume TB PB scale data with focus on performance scalability reliability and cost optimization
• Develop complex SQL transformations data models and analytics ready BigQuery datasets using partitioning and clustering
• Orchestrate pipelines using Cloud Composer Airflow and implement CI CD and Terraform based Infrastructure as Code
• Ensure data quality governance security monitoring alerting and production support
• Collaborate with architects stakeholders and cross functional teams in an Agile delivery model
• Troubleshoot and optimize end to end GCP Databricks and streaming data pipelines

Additional Responsibilities:

• Infosys is a global leader in next generation digital services and consulting
• Within the Data Analytics unit you will work on cutting edge data engineering and analytics initiatives for global clients across industries
• The role offers opportunities to work with cloud native data platforms modern analytics ecosystems AI driven solutions and large scale enterprise data modernization programs
• Employees benefit from continuous learning programs certifications and career growth opportunities within Infosys data and analytics practice

Preferred Skills: