We are seeking a highly experienced and versatile Lead Data Engineer to join our engineering team. The ideal candidate will be proficient in the full software development lifecycle, with strong expertise in building micro services, Python, PySpark, and AWS cloud services. This role involves developing scalable enterprise applications within a Microservices Architecture, integrating databases, and ensuring high code quality through rigorous testing and engineering best practices.

Key Responsibilities:

Contribute across the entire technology stack: frontend, backend, and database layers.
Design and develop enterprise-grade services using Python, Pyspark and AWS.
Implement microservices-based architecture to build scalable and maintainable solutions.
Apply software engineering principles to enhance the reliability, scalability, and maintainability of the codebase.
Incorporate automated testing and ensure robust code coverage as a part of daily development.
Collaborate closely with cross-functional teams in an Agile development environment.
Design and implement ETL/ELT pipelines for automated data extraction, transformation, and loading
Develop parameterized queries, job configurations, and data ingestion logic for diverse datasets
Build and maintain data orchestration workflows using tools like Apache Airflow
Partner with the Full Stack team to integrate backend APIs with ETL jobs and job status monitoring
Support integration and load testing, and optimize job execution across environments
Work on metadata ingestion, job audit logging, and traceability for compliance
Participate in production deployment, job monitoring, and troubleshooting post go-live
Document data models, pipelines, and configurations; support training and knowledge transfer

Must-Have Skills:

10+ years of experience in data engineering and ETL development in complex enterprise environments
Strong object-oriented programming knowledge
Proficient in Python 3.6+. Strong debugging and performance tuning skills across the stack.
Strong experience in Python and Django for backend API development
Hands-on experience with AWS services (EC2, S3, Lambda, etc)
Experience integrating and managing SQL databases.
Hands-on experience with Apache Airflow for workflow orchestration.
Hands-on experience with Apache Spark for big data processing and analytics.
Hands-on experience in building ETL pipelines with tools like Python, SQL, or similar
Exposure to containerization and orchestration tools (eg, Docker, Kubernetes).
CI/CD pipeline experience with tools like Jenkins, GitLab CI/CD, or AWS Code Pipeline.
Expertise in SQL Server: complex queries, stored procedures, performance tuning
Familiarity with RESTful API integration for triggering and monitoring jobs
Proficiency with job parameterization, scheduling, and configuration-driven pipelines
Exposure to data validation, quality checks, and error handling mechanisms
Experience with Git-based version control, CI/CD, and deployment processes
Knowledge of metadata management and data cataloging concepts
Familiarity with unit tests (Preferably with PYtest)
Proven track record of working in fast-paced, agile teams.

Our Perks and Benefits:

Our benefits and rewards program has been thoughtfully designed to recognize your skills and contributions, elevate your learning/upskilling experience and provide care and support for you and your loved ones. As an Apexon Associate, you get continuous skill-based development, opportunities for career advancement, and access to comprehensive health and we'll-being benefits and assistance.

We also offer:

Group Health Insurance covering family of 4
Term Insurance and Accident Insurance
Paid Holidays & Earned Leaves
Paid Parental Leaveo Learning & Career Development
Employee we'llness

Job Location : Bengaluru, India

Lead - Big Data Engineer

Lead Software Engineer - Java, Big Data, Hybrid cloud