Lead Data Engineer

Lead Data Engineer (PySpark) - Tech Lead

Location: Pune (Work from Office – 5 Days a Week)

Job Summary:

We are looking for a highly skilled and experienced Lead Data Engineer to join our team in Pune. The ideal candidate will have 8–10 years of experience in designing, building, and optimizing scalable data pipelines and data platforms. The role requires strong expertise in Python, PySpark, SQL, cloud technologies, and big data frameworks to support large-scale data processing and analytics initiatives.

Key Responsibilities

• Design, develop, and maintain scalable and high-performance data pipelines using Python, PySpark, and SQL.
• Build and optimize ETL/ELT processes for large-scale data ingestion, transformation, and storage.
• Process and manage large datasets containing millions of records with a focus on performance and scalability.
• Develop and maintain data models, data warehouses, and data architecture solutions.
• Work with cloud platforms such as AWS, Azure, or Google Cloud and leverage cloud-native data services.
• Implement and manage workflow scheduling and orchestration tools.
• Collaborate with cross-functional teams, including Data Analysts, Data Scientists, and Product teams.
• Monitor, troubleshoot, and optimize existing data pipelines and infrastructure.
• Develop automation scripts using Shell Scripting and Linux tools.
• Stay updated with emerging technologies and industry trends in data engineering.

Required Skills & Experience

• 8-10 years of experience in Data Engineering or a related field.
• Strong hands-on experience with Python, PySpark, SQL, Pandas, and MongoDB.
• Experience working with Apache Spark for large-scale distributed data processing.
• Strong understanding of ETL processes, data warehousing, and data modeling concepts.
• Experience with big data technologies such as Hadoop, Hive, and related ecosystems.
• Experience with workflow orchestration and scheduling tools such as Airflow or similar.
• Proficiency in Shell Scripting and Linux environments.
• Knowledge of modern data engineering technologies such as Elasticsearch, Druid, etc.
• Experience working with cloud platforms (AWS, Azure, or GCP).
• Strong analytical, problem-solving, and debugging skills.
• Excellent communication and collaboration abilities.

Preferred Qualifications

Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

Cloud certifications such as AWS Certified Data Analytics, Google Cloud Professional Data Engineer, or equivalent.

Exposure to AI/ML/LLM-based data solutions is a plus.

What We Offer

Opportunity to work on large-scale data engineering projects.

Exposure to modern cloud and big data technologies.

Collaborative and growth-oriented work environment.

Direct impact on business-critical data initiatives.