Role Overview

The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities

· Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.

· Define technical strategy, architecture, and best practices for development and data engineering.

· Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.

· Construct infrastructure for efficient ETL processes from various sources and storage systems.

· Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.

· Lead the implementation of algorithms and prototypes to transform raw data into useful information.

· Review code for quality, scalability, and performance.

· Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.

· Develop analytical tools, programs, and reporting mechanisms.

· Create data validation methods and data analysis tools.

· Interpret data trends and patterns to establish operational alerts.

· Conduct complex data analysis and present results effectively.

· Prepare data for prescriptive and predictive modeling.

· Ensure compliance with data governance and security policies.

· Troubleshoot, debug, and resolve complex technical issues.

· Drive continuous improvement in software and data development processes, tools, and methodologies.

· Mentor and guide engineers through code reviews, technical discussions, and training.

· Ensure timely delivery of projects while maintaining high engineering standards.

· Continuously explore opportunities to enhance data quality and reliability.

· Apply strong programming and problem-solving skills to develop scalable solutions.

· Demonstrate passion for testing strategy, problem-solving, and continuous learning.

· Willingness to acquire new skills and knowledge.

· Possess a product/engineering mindset to drive impactful data solutions.

· Experience working in distributed environments with global teams.

· Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

· Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

· High proficiency in Python, Scala, and Spark for applied large-scale data processing.

· Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.

· Solid understanding of batch and streaming data processing techniques.

· Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.

· Expert-level ability to write complex, optimized SQL queries across extensive data volumes.

· Experience with RDBMS and OLAP databases such as MySQL and Snowflake.

· Familiarity with Agile methodologies.

· Obsession for service observability, instrumentation, monitoring, and alerting.

· Knowledge or experience in architectural best practices for building data lakes.

Qualifications

· Bachelor's degree in computer science, Engineering, Information Systems, or related field

· Relevant Experience: 8+ Years (as Python and Data Engineer)

· Overall IT Experience: 8–12 Years

Azure - Lead Data Engineer

Lead Azure Data Engineer / Data Engineer (Python)