Data Engineer

Job Title: Data Engineer — Cloudera to BigQuery Migration

Budget: 1.10 LPM + GST

Location: Remote

Experience: 4-5 years

Role Overview

We are looking for a skilled Data Engineer to lead and support migration initiatives from Cloudera (Hadoop/Hive) to Google Cloud Platform (BigQuery). The ideal candidate will have strong expertise in data warehousing, ELT pipeline design, and cloud-native architectures.

Key Responsibilities

Lead end-to-end migration from Cloudera (Hive/HDFS) to BigQuery

Convert and optimize Hive/HiveQL queries into BigQuery-compatible SQL

Design and implement scalable Medallion Architecture (Bronze, Silver, Gold layers)

Build robust ELT pipelines supporting both incremental and full snapshot load patterns

Develop and manage orchestration workflows using Python-based DAGs (Airflow/Cloud Composer)

Implement data ingestion pipelines via SFTP, JDBC, and other connectors

Work with Parquet and other columnar formats for efficient data processing

Ensure secure data access using VPC Service Perimeters and private networking

Collaborate with cross-functional teams to ensure data quality, governance, and performance optimization

Required Skills & Experience

Strong hands-on experience with GCP BigQuery

Proven experience in migrating data from Hive/Cloudera to cloud platforms

Expertise in SQL transformation (HiveQL to BigQuery SQL)

Solid understanding of data modeling and Medallion architecture

Experience with Apache Airflow / Cloud Composer for orchestration

Proficiency in Python for data pipeline development

Experience with Infrastructure as Code tools like Terraform

Familiarity with data ingestion techniques (SFTP, JDBC)

Understanding of networking concepts like VPC, security perimeters

Good to Have

Experience with large-scale data migration projects

Knowledge of data governance and security best practices

Exposure to performance tuning in BigQuery

Education

Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience)