Software Engineer

Description As a GCP Data Engineer, you will be instrumental in modernizing our data infrastructure by building robust, automated ETL/ELT pipelines and leveraging Google Cloud Platform's powerful suite of big data tools. Collaborating closely with Enterprise Architects, you will spearhead the migration from legacy Hadoop/CDP systems to BigQuery. Your daily focus will involve architecting historical and incremental data loads, orchestrating tasks with Apache Airflow, and creating scalable, data-driven solutions that empower the organization with clean, accessible, and organized data.

Responsibilities

• Pipeline Development: Develop and maintain efficient ETL/ELT pipelines, implementing and automating tasks using Apache Airflow.
• Data Architecture: Architect historical and incremental data loads, continuously evaluating and refining the data architecture to ensure optimal performance.
• Data Warehousing: Support, organize, and optimize data within Google BigQuery data warehouses.
• System Migration: Work alongside Enterprise Architects to lead and deliver the migration of data from legacy Hadoop/Cloudera Data Platform (CDP) systems to BigQuery.
• Enterprise Solutions: Design and build end-to-end, GCP data-driven solutions for enterprise data warehouses and data lakes.

Must Have:

• Certification: Professional GCP Data Engineer Certification or equivalent.
• Development Experience: 2+ years of coding experience in Java/Python and Infrastructure as Code using Terraform.
• GCP Expertise: 2+ years of experience working in GCP-based Big Data deployments (both Batch and Real-Time), specifically leveraging BigQuery, Bigtable, Google Cloud Storage, Pub/Sub, Data Fusion, Dataflow, Dataproc, and Airflow.
• Data Design: Proven history of designing and delivering comprehensive data lake and data warehousing solutions.
• Data Processing: Strong hands-on experience in extracting, loading, transforming (ETL/ELT), cleaning, and validating data, as well as designing pipelines and architectures for data processing.
• Database Skills: Proficiency in at least one SQL language.
• Big Data Tools: Practical experience working with Spark services.
• Methodologies: Experience working within Agile and Lean methodologies.

Good to Have (Preferred):

• Data Visualization: Experience using visualization tools such as Qlik or Looker Studio.
• Migration Experience: Proven experience in migrating legacy systems (preferably Big Data Cloudera Data Platform) into GCP technologies.
• Large-Scale Systems: Experience working with either a MapReduce or an MPP (Massively Parallel Processing) system at any size or scale.