Data Engineer

Job Title: Data Engineer – Airflow, Spark, dbt

Role Overview

We are seeking a skilled Data Engineer with strong expertise in Apache Spark, Apache Airflow, and dbt (Data Build Tool) to design, build, and optimize scalable data pipelines. The role involves working on modern data platforms, enabling analytics, reporting, and advanced data transformations across cloud or hybrid environments.

Key Responsibilities

· Design, develop, and maintain end-to-end data pipelines using Spark (batch & streaming).

· Build and manage workflow orchestration using Apache Airflow (DAGs, scheduling, monitoring).

· Develop modular and scalable transformations using dbt following best practices.

· Implement ELT frameworks for data ingestion, transformation, and processing.

· Ensure high data quality through validation, testing, and monitoring frameworks (dbt tests, Airflow alerts).

· Optimize performance of Spark jobs and SQL transformations.

· Work with data warehouses/lakehouses (Snowflake, BigQuery, Redshift, Databricks).

· Collaborate with analysts, data scientists, and business teams to translate requirements into data solutions.

· Maintain documentation and adhere to data governance, security, and compliance standards.

· Support CI/CD integration for data pipeline

Required Skills

Core Technologies

· Apache Spark (PySpark / Scala / SQL)

· Apache Airflow (DAG design, scheduling, troubleshooting)

· dbt (Data Build Tool) – models, macros, testing, documentation

· Advanced SQL & Python

Data Engineering Concepts