Senior Data Engineer

Senior Data Engineer – Spark / Scala

Location: Bangalore, India

Industry: Retail & E-commerce

About the Role

We are looking for an experienced Senior Data Engineer to join our data engineering team supporting a large-scale US retail environment. The ideal candidate will have strong hands-on experience building and optimizing data platforms, developing high-volume data pipelines, and working with distributed data processing technologies.

This is a hands-on role for someone who can ramp up quickly, troubleshoot complex production issues, and independently drive technical deliverables from development through deployment.

Key Responsibilities

• Design, develop, and optimize large-scale data pipelines using Apache Spark and Scala.
• Build and maintain high-volume batch and streaming data pipelines.
• Develop data processing solutions using Spark, Hive, Kafka, and GCP services.
• Work with BigQuery, GCS, Dataproc, and Pub/Sub as part of cloud-based data platforms.
• Perform Spark job optimization, performance tuning, and troubleshooting.
• Design and maintain scalable ETL/ELT workflows and data pipelines.
• Troubleshoot complex data, application, performance, and production issues.
• Support cloud migration and modernization of legacy/on-premise data platforms.
• Develop reusable, scalable, maintainable, and production-ready data solutions.
• Implement and maintain CI/CD pipelines using Git, Jenkins, and related DevOps practices.
• Participate in code reviews, testing, production deployments, and operational support.
• Collaborate with engineering, analytics, product, and platform teams to deliver end-to-end data solutions.
• Analyze data quality issues and implement appropriate validation and reconciliation processes.
• Quickly understand existing systems and take ownership of assigned technical deliverables.

Required Qualifications

• 7+ years of experience in Data Engineering, Big Data, or related engineering roles.
• Strong hands-on experience with Apache Spark and Scala.
• Strong experience with Hive and distributed data processing.
• Hands-on experience with Kafka and streaming/event-driven architectures.
• Strong programming experience in Scala and/or Python.
• Proven experience designing and troubleshooting large-scale ETL/data pipelines.
• Strong understanding of distributed systems, data modeling, data quality, and performance optimization.
• Experience with at least one major cloud platform: GCP, AWS, or Azure.
• Experience with Git and CI/CD practices.
• Experience with workflow/orchestration tools such as Oozie or equivalent.
• Strong production troubleshooting and problem-solving skills.
• Ability to work independently in a fast-paced engineering environment.

Preferred Qualifications

• Strong experience with Google Cloud Platform (GCP).
• Experience with BigQuery, GCS, Dataproc, and Pub/Sub.
• Experience migrating on-premise Spark/Hive workloads to GCP.
• Experience with large-scale data migration, validation, and reconciliation.
• Knowledge of data quality, monitoring, and observability frameworks.
• Experience supporting complex retail, e-commerce, or high-volume transactional data environments.
• Experience working with multiple upstream and downstream data integrations.
• Experience with modern data engineering and cloud modernization initiatives.

Apply now or reach out with your resume to learn more about the opportunity at [Confidential Information]

#Hiring #DataEngineer #SeniorDataEngineer #BigData #ApacheSpark #Scala #Kafka #Hive #GCP #BigQuery #Dataproc #DataEngineering #CloudComputing #ETL #Python #Jenkins #CICD #RetailTech #TechJobs #USJobs