Data Engineer

Role Overview

We are looking for a hands-on Data Engineer to design, build, and optimize scalable data pipelines and platforms. You will be responsible for creating robust batch and streaming data processing frameworks that enable advanced analytics and AI solutions. [1, 2]

Key Responsibilities

• Data Pipeline Development: Design and maintain scalable ETL/ELT pipelines using Scala and Spark (Core, SQL, Streaming).
• Real-time Streaming: Implement and manage real-time data ingestion using Apache Kafka or GCP Pub/Sub.
• Cloud Infrastructure: Build and optimize data solutions on Google Cloud Platform using services like BigQuery, Dataflow, Dataproc, and Cloud Storage.
• Performance Tuning: Optimize Spark jobs for speed and scalability through partitioning, caching, and shuffle tuning.
• Data Modeling: Design efficient data models for large-scale datasets in SQL and NoSQL environments.
• Orchestration & DevOps: Manage data workflows using tools like Airflow (Cloud Composer) and implement CI/CD for data pipelines.

Required Skills & Qualifications

• Programming: Deep proficiency in Scala is mandatory.
• Big Data Frameworks: Strong hands-on experience with Apache Spark and Hadoop ecosystem components.
• Messaging Systems: Practical experience with Apache Kafka (producers/consumers, streaming APIs).
• Cloud Platform: Experience with GCP (preferred) or other major cloud providers like AWS/Azure.
• Databases: Mastery of SQL and experience with cloud data warehouses like BigQuery.
• Experience: Typically requires 3–10 years of professional experience in data engineering roles.

Preferred Qualifications

• Google Professional Data Engineer Certification.
• Experience with Infrastructure as Code (e.g., Terraform).
• Familiarity with containerization using Docker or Kubernetes.