Data Engineer(ETL, SQL, SSIS, Python), AWS

Job Description: Data Engineer (Mid-Level, AWS & Databricks)

Role Overview

As a Mid-Level Data Engineer, you will develop and maintain the data pipelines that form the backbone of our AWS Data Lakehouse. Your mission is to translate business requirements into efficient PySpark code and ensure our SQL Server data is accurately migrated and synchronized with Databricks.

Key Responsibilities

ETL Development: Develop and debug data pipelines in Databricks using PySpark and SparkSQL.

Data Ingestion: Implement data movement from SQL Server to Amazon S3 using efficient batch and incremental loading techniques.

Database Operations: Query and manage SQL Server environments to validate data consistency during the migration phase.

Modernization: Refactor existing SQL-based transformations into modular Python scripts or Scala functions.

Required Skills & Qualifications

4-6 years of experience in Data Engineering.

Hands-on Experience: Building production pipelines in Databricks on AWS.

Core Competencies: Strong Python, PySpark, and complex SQL (T-SQL preferred).

AWS Basics: Familiarity with S3 bucket management and basic IAM roles.

Adaptability: Ability to work across Python and SQL comfortably, with an interest in learning/using Scala for performance tuning.

Nice to Have

Prior exposure to SSIS (this is a plus, not a requirement).

Knowledge of AWS Glue or AWS Lambda for lightweight orchestration.