Job Summary
Location: AIA-Chennai
We are seeking
Senior Data Engineer resources to work on the migration of applications from our legacy Cloudera environment to the new Kubernetes-based data platform. The role requires strong
hands-on development skills & Testing in data engineering, with the ability to deliver high-quality pipelines under guidance from internal leads.
Key Responsibilities
- Develop and optimize data pipelines using Spark 3.5 and Python/Scala.
- Migrate existing Hive, Spark, and Control-M jobs to Airflow and DBT-based workflows.
- Integrate data pipelines with messaging systems (Kafka, Solace) and object stores (S3, MinIO).
- Troubleshoot and optimize distributed jobs running in Kubernetes environments.
- Collaborate closely with internal leads and architects to implement best practices.
- Design and implement migration/acceleration framework to automate end to end migration.
- Continuous enhancements to the frameworks to ensure the stability, scalability and support for diverse use cases and scenarios.
- Work with various data applications to enable and support the migration process.
- Deliver assigned migration tasks within agreed timelines.
Required Skills
- 69 years of hands-on data engineering experience.
- Strong expertise in Apache Spark (batch + streaming) and Hive.
- Proficiency in Python, Scala, or Java.
- Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
- Experience working with Kafka, Solace, and object stores (S3, MinIO).
- Exposure to Docker/Kubernetes for deployment.
Hands on experience of
data Lakehouse formats (Iceberg, Delta Lake, Hudi).
