Data Engineer

Overview

This role sits within the data engineering team reporting to the Engineering Manager. The primary mandate is to lead and accelerate the migration from Azure Synapse pipeline orchestration to Dagster, establishing scalable, production- grade data pipelines and CI/CD best practices. This individual will initially operate as a lead/solo engineer, with future team augmentation planned. Dagster has already been implemented and a team of six is currently learning the platform, but this person will help accelerate their training of Dagster and be more of the SME for the team.

Experience

3–5+ years in data engineering roles

Proven experience operating in production environments and scaling data platforms

Required Skills (Priority Order)

Dagster – Strong experience with asset-based orchestration, sensors, scheduling, and ideally Dagster+ Cloud deployments

Python – Advanced proficiency in production-grade data engineering (scripting, packaging, pipeline development)

PySpark – Experience building and tuning distributed pipelines (DataFrames, Spark SQL, performance optimization)

Data Engineering Fundamentals – Data modeling, ETL/ELT, orchestration, containerization, and lake architecture (ADLS Gen2)

Azure Synapse – Working knowledge of pipelines, notebooks, linked services, and integration runtimes (migration experience preferred)

Key Responsibilities

Lead migration from Azure Synapse pipelines to Dagster (Dagster+ Cloud preferred)

Design, build, and optimize asset-based data pipelines using Dagster (sensors, schedules, orchestration patterns)

Develop and maintain scalable data pipelines using Python and PySpark

Establish and standardize CI/CD processes for data workflows

Collaborate with Data Scientists and MLOps engineers to support downstream analytics and AI use cases

Implement and maintain data architecture within Azure (ADLS Gen2, SQL, Synapse)

Optimize distributed data processing jobs and ensure performance and reliability

Contribute to data modeling, ETL/ELT design, and containerized deployment strategies

Provide technical guidance and upskill team members on Dagster adoption