Job Title: Senior Data Engineer (Kafka & AWS)
Location: Bangalore, Hyderabad, Kochi, Trivandrum, Pune
Experience: 5+ Years
Primary Skills: Python, AWS, Apache Kafka, PySpark
Job Summary
We are seeking a highly skilled Senior Data Engineer with strong expertise in Apache Kafka and AWS data services to design, build, and optimize scalable real-time and batch data pipelines. The ideal candidate will have hands-on experience with distributed data systems, cloud-based architectures, and modern ETL/ELT frameworks.
Key Responsibilities
- Design, develop, and maintain real-time data pipelines using Apache Kafka (MSK/Confluent) and AWS services
- Configure and manage Kafka connectors, ensuring seamless integration across multiple systems
- Work extensively with the Kafka ecosystem including producers, consumers, brokers, topics, and schema registry
- Build and optimize ETL/ELT pipelines for large-scale data processing using PySpark and AWS tools
- Develop and enhance data lake and data warehouse solutions using AWS services such as Lambda, S3, and Glue
- Implement monitoring, logging, and observability frameworks to ensure reliability and performance
- Ensure adherence to data security, governance, and compliance standards
- Collaborate with cross-functional teams including Data Scientists, Analysts, and DevOps teams
Required Skills & Qualifications
- Minimum 5+ years of experience in Data Engineering or related roles
- Strong hands-on experience with Apache Kafka (MSK/Confluent)
- Deep understanding of AWS data ecosystem (Glue, Lambda, S3, etc.)
- Proficiency in Python, SQL, and Java (preferred)
- Experience with PySpark for large-scale data processing
- Familiarity with Infrastructure-as-Code (IaC) tools such as CloudFormation
- Experience in building and managing CI/CD pipelines
- Strong problem-solving, analytical, and communication skills
- Ability to write production-quality code in both Python and Java
