Data Engineer

Company Profile

Our client is a global IT services company that helps businesses with digital transformation with offices in India and the United States. It helps businesses with digital transformation, provide IT collaborations and uses technology, innovation, and enterprise to have a positive impact on the world of business.

With expertise is in the fields of Data, IoT, AI, Cloud Infrastructure and SAP, it helps accelerate digital transformation through key practice areas - IT staffing on demand, innovation and growth by focusing on cost and problem solving.

Job Profile: Specialist System Engineer (IBM MQ)

Location: Remote

Employment Type: C2H

Preferred experience: 5 - 8 years

The Role:

The Data Engineering Lead will be responsible for designing, developing, and optimizing scalable AWS-based data pipelines for retail data ecosystems, including MMS and POS systems. The role involves building centralized Operational Data Store (ODS) platforms using NoSQL technologies and enabling real-time or near real-time data processing for downstream applications and analytics. The candidate will lead data engineering initiatives using PySpark, AWS Glue, EMR, Redshift, and streaming technologies while ensuring data quality, governance, and performance optimization. The role also requires collaboration with DevOps, microservices, and application teams, along with providing technical leadership and mentorship to the engineering team.

Responsibilities:

·      Design and implement scalable ETL/ELT pipelines for ingesting data from MMS, POS, and third-party systems into AWS-based data platforms.

·      Build robust batch and streaming ingestion frameworks for structured and semi-structured data.

·      Develop and optimize large-scale data processing workflows using PySpark / Apache Spark.

·      Implement CDC merge/upsert logic and idempotent processing frameworks.

·      Ensure data consistency, reconciliation, and reliability across pipelines.

AWS Data Platform Engineering

·      Develop scalable cloud-native solutions using AWS services such as:

o  Amazon S3

o  AWS Glue

o  Amazon EMR

o  Amazon Redshift

o  Amazon Aurora

o  Amazon Kinesis

o  AWS Lambda

·      Optimize data platform performance, scalability, and cost efficiency.

·      Support real-time and near real-time data processing architectures.

Operational Data Store (ODS) & NoSQL Engineering

·      Design and maintain centralized ODS layers using Couchbase or similar NoSQL technologies.

·      Create denormalized, API-ready data models for downstream microservices and applications.

·      Support high-performance, low-latency application access patterns.

Preferred exposure to:

·      Couchbase

·      MongoDB

·      Amazon DynamoDB

CI/CD, DevOps & Automation

·      Collaborate with DevOps teams to build and maintain CI/CD pipelines.

·      Implement deployment automation using:

o  GitLab

o  AWS CodePipeline

o  Similar DevOps tools

·      Ensure version control, deployment consistency, and release management best practices.

Data Governance & Quality

·      Define and enforce data governance frameworks.

·      Implement metadata management and data quality standards.

·      Ensure secure, compliant, and auditable data operations.

Must - Have Qualifications:

·      5+ years of Data Engineering experience

·      Minimum 2+ years in Technical Lead / Team Lead role

·      Strong hands-on experience in AWS Data Services

·      Expertise in AWS Glue

·      Strong experience with Amazon S3

·      Experience with Amazon EMR

·      Hands-on experience with Amazon Redshift

·      Experience with Amazon Aurora

·      Strong proficiency in PySpark / Apache Spark

·      ETL / ELT Pipeline Development experience

·      Experience building Batch & Streaming Data Pipelines

·      CDC (Change Data Capture) / Upsert Processing experience

·      Data Modeling & Data Transformation expertise

·      Experience with NoSQL Databases

·      Hands-on experience with Couchbase / DynamoDB / MongoDB

·      Experience building Operational Data Store (ODS)

·      Experience working with MMS (Merchandise Management Systems)

·      Experience working with POS (Point of Sale) Data

·      Retail / Retail Transaction Data experience

·      Experience with Amazon Kinesis and/or AWS Lambda

·      CI/CD Pipeline implementation experience

·      Hands-on experience with GitLab CI/CD / AWS CodePipeline

·      Experience supporting Microservices-based Architectures

·      Data Governance & Data Quality exposure

·      Strong Performance Optimization & Troubleshooting skills

·      Experience with AI-assisted coding tools like: Claude Code, OpenAI Codex, Similar AI developer tools

Preferred Qualifications:

·      Experience supporting microservices architectures

·      Strong understanding of distributed systems

Application Method

Apply on LinkedIn or email your resume to: [Confidential Information]