Data Engineer - CB2

Data Engineer

Overview Of Role

• Design, develop, and maintain scalable, high-performance data engineering solutions using AWS,
  
  

Databricks, Spark, and PySpark, leveraging strong technical acumen to solve complex data and

engineering challenges.

• Build robust data pipelines following Medallion Architecture (Raw, Silver, and Gold layers), covering data
  
  

ingestion, transformation, processing, data modeling, validation, quality checks, and reconciliation.

• Develop and optimize data solutions using AWS services including S3, Glue, Aurora/RDS, Lambda, and
  
  

Step Functions, along with Databricks, Spark, and PySpark to improve performance, scalability, reliability,

and cloud cost efficiency.

• Implement reliable orchestration and operational workflows covering scheduling, dependencies, retries,
  
  

error handling, monitoring, and failure recovery, while troubleshooting complex data and production

issues and driving root-cause resolution.

• Develop reusable, maintainable, and production-ready code following engineering standards and best
  
  

practices; contribute to code reviews, testing, CI/CD, deployment, automation, and continuous

improvement.

• Support data migration and modernization initiatives across AWS and Databricks, including legacy
  
  

platform migrations, source-to-target mapping, data validation, reconciliation, and production readiness.

• Collaborate with Product, Business, Architecture, QA, API, and Engineering teams in a cross-functional
  
  

Agile environment, contributing to technical design discussions, estimation, sprint planning, backlog

refinement, and delivery.

• Provide technical guidance and mentorship to other engineers as applicable, promote reusable
  
  

frameworks and engineering standards, and contribute to resolving complex technical challenges.

Required Qualifications

• 5+ years of experience in Data Engineering, ETL/ELT, and developing large-scale data pipelines, with
  
  

experience providing technical guidance or leadership as applicable.

• Strong hands-on experience with AWS services including S3, Glue, Aurora/RDS, Lambda, and Step
  
  

Functions, with a strong understanding of cloud-native data engineering practices.

• 2+ years of hands-on experience with Databricks, Spark, and PySpark, including Spark performance
  
  

optimization, partitioning, joins, caching, file formats, data skew, and efficient job design.

• Strong SQL and Python/PySpark skills, with experience in data modeling, database design, schema
  
  

mapping, ETL/ELT, and source-to-target transformations.

• Experience designing and optimizing data pipelines and orchestration workflows, including scheduling,
  
  

dependencies, retries, error handling, monitoring, failure recovery, and performance optimization.

• Strong understanding of software engineering practices including coding standards, code reviews, testing,
  
  

Git, CI/CD, Azure DevOps, reusable frameworks, and automation.

• Experience with data quality, validation, reconciliation, monitoring, troubleshooting, and root-cause
  
  

analysis.

• Understanding of REST APIs, API request/response flows, and integration with APIs and third-party source
  
  

systems.

• Experience creating, reviewing, and maintaining functional and technical documentation throughout the
  
  

delivery lifecycle.

• Strong communication and collaboration skills, with experience working across Product, Business,
  
  

Architecture, QA, API, and Engineering teams in Agile environments.

• Strong understanding of Agile practices, including sprint planning, backlog refinement, estimation,
  
  

iterative delivery, and production support.

5

Preferred Qualifications

• Experience with API development and integration, including hands-on experience working with API-driven
  
  

data solutions.

• Experience with Redis or other in-memory databases and caching technologies.
• Experience with Databricks and advanced Spark performance optimization techniques.
• Experience developing reusable frameworks, automation, and engineering standards.
• Relevant AWS certifications in Cloud, Data Engineering, or Solutions Architecture.
• Experience developing, debugging, and supporting data solutions within large, cross-functional
  
  

engineering teams.

• Strong analytical and problem-solving skills, with the ability to manage multiple priorities in a deadlinedriven
  
  

environment.

• Strong attention to detail and commitment to data quality, reliability, and engineering excellence.