Data Engineer(ETL, SQL, SSIS, Python, AWS)

Job Description: Senior Data Engineer (AWS & Databricks)

Role Overview

We are looking for a Senior Data Engineer to architect and execute the migration of our data estate from SQL Server to an AWS-based Databricks Lakehouse. You will lead the transition from legacy stored procedures and manual processes into scalable PySpark pipelines. This role is critical for establishing our Medallion Architecture (Bronze/Silver/Gold) on AWS.

Key Responsibilities

Migration Leadership: Lead the technical migration strategy for moving data and logic from on-prem/RDS SQL Server to AWS S3 and Databricks.

Pipeline Engineering: Build high-performance ETL/ELT pipelines using PySpark and Scala, leveraging Databricks Workflows and Unity Catalog.

AWS Integration: Configure and optimize data ingestion patterns using AWS Glue (Crawlers), S3, and IAM for secure data access.

Code Quality: Set standards for Python/Scala development, including unit testing and CI/CD integration via AWS CodePipeline or GitHub Actions.

Required Skills & Qualifications

8+ years in Data Engineering with at least 3 years focused on AWS.

Expert level: PySpark and Databricks (Delta Lake, Photon engine, Liquid Clustering).

Strong Proficiency: SQL Server (T-SQL) for reverse-engineering legacy logic and optimizing source extraction.

Languages: Mastery of Python and SQL; professional experience with Scala.

Cloud Infrastructure: Hands-on experience with AWS S3, IAM, and Secrets Manager.

Nice to Have

Experience reading/decoding SSIS packages to extract business logic.

Databricks Certified Data Engineer Professional.