Senior Data Engineer

Job Title: Senior Data Engineer

Department: Engineering

Location: Ahmedabad

Job Summary:

Senior Data Engineer to design, build, and maintain scalable data pipelines and databases that support our intelligent automation products. The role requires strong expertise in Python, modern data systems, cloud platforms, and API development. The ideal candidate will be responsible for ensuring data reliability, high performance, and seamless integration across various platforms while working closely with cross-functional teams.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines for large-scale data processing.

• Optimize and troubleshoot existing data pipelines for improved efficiency and reliability.

• Build, manage, and optimize databases using MongoDB, Elasticsearch, Redis, and SQL systems.

• Ensure high availability, data integrity, and scalability of all database environments.

• Implement cloud-based data infrastructure on AWS, Azure, or Google Cloud as per project requirements.

• Configure and manage CI/CD pipelines for automated deployment and testing of data workflows.

• Work with containerization tools like Docker; exposure to Kubernetes will be advantageous.

• Develop and maintain RESTful APIs using Flask for internal and external data integration.

• Collaborate with frontend, backend, data science, and analytics teams to deliver robust data solutions.

• Contribute to architectural discussions, sprint planning, code reviews, and process improvements.

• Monitor system performance, manage logging, and address production issues proactively.

• Support continuous improvement by mentoring junior engineers and sharing best practices.

Required Skills & Qualifications:

• Strong proficiency in Python for building data pipelines and APIs.

• Hands-on experience with MongoDB, Elasticsearch, Redis, and SQL databases.

• In-depth understanding of ETL/ELT processes, distributed data systems, and data engineering frameworks.

• Experience in developing and maintaining APIs using Flask.

• Working knowledge of cloud platforms such as AWS, GCP, or Azure.

• Familiarity with CI/CD tools like Jenkins or GitHub Actions.

• Knowledge of Docker; Kubernetes knowledge is a plus.

• Strong analytical, debugging, and problem-solving abilities.

• Excellent communication and collaboration skills, with the ability to work independently.

Preferred Skills:

• Experience working with data streaming technologies such as Kafka.

• Understanding of machine learning workflows or MLOps.

• Familiarity with Infrastructure as Code tools including Terraform or Ansible.

Experience:

• 5 to 8+ years of experience in Data Engineering or a similar technical role.

• Proven experience managing large-scale, production-grade data systems.

Educational Qualification:

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.