Lead Data Engineer
Youwin Technologies Private Limited- Posted 9 hours ago
- Be among the first 30 applicants
Job Description
Job Title: Lead Data Engineer
Experience: 8–12 Years
Employment Type: Full-Time
Department: Data Engineering
Job Summary
We are looking for an experienced Lead Data Engineer with strong expertise in Python programming, Databricks, Data Modelling, T-SQL and Data Architecture.
The ideal candidate will have extensive hands-on experience designing, developing and managing production-grade Python applications, scalable ETL/ELT pipelines and modern Lakehouse platforms.
This role requires a strong Python-first engineering approach, combined with deep knowledge of Databricks, Spark, data warehousing and data architecture. The candidate will also be responsible for defining engineering standards, optimizing platform performance, providing technical direction and mentoring a team of Data Engineers.
Key Responsibilities
- Design, develop and maintain production-grade Python applications, reusable packages, frameworks and ETL/ELT pipelines.
- Develop clean, scalable and well-tested Python code using OOP, decorators, generators, multiprocessing and asyncio.
- Implement unit testing, schema validation and automated testing frameworks using pytest and CI/CD practices.
- Profile, debug and optimize Python applications and data pipelines for performance and scalability.
- Establish Python coding standards, conduct code reviews and implement best practices for packaging, versioning, logging and exception handling.
- Design and implement Bronze, Silver and Gold Medallion Architecture pipelines using Databricks.
- Work with Delta Lake, Structured Streaming, Auto Loader and Unity Catalog.
- Optimize Spark workloads, including data skew, cluster sizing, autoscaling and cost performance.
- Design and maintain dimensional data models using Star and Snowflake schemas.
- Implement Slowly Changing Dimensions (SCD) and manage fact and dimension tables.
- Develop and optimize advanced T-SQL queries, including window functions, MERGE/upsert operations and indexing strategies.
- Collaborate with Data Architects and business stakeholders on data architecture, orchestration and ETL/ELT strategies.
- Mentor Data Engineers and Senior Data Engineers through technical guidance, design reviews and code reviews.
Required Skills & Qualifications
1. Python & Data Engineering – Primary Requirement
- Strong hands-on experience in Python programming.
- Excellent understanding of OOP concepts, decorators, generators and exception handling.
- Experience with multiprocessing and asyncio.
- Strong experience developing production ETL/ELT pipelines.
- Knowledge of unit testing using pytest, schema validation and reusable Python frameworks.
- Ability to debug, profile and optimize Python code for large-scale data processing.
2. Databricks & Lakehouse
- Strong hands-on experience with Databricks and Apache Spark.
- Experience with Delta Lake, ACID transactions, Time Travel, OPTIMIZE, ZORDER and VACUUM.
- Knowledge of Structured Streaming, Auto Loader and Unity Catalog.
- Experience in Spark job optimization, cluster management and performance tuning.
3. Data Modelling & Data Warehousing
- Strong understanding of dimensional data modelling.
- Experience with Star Schema and Snowflake Schema.
- Knowledge of SCD Types 1, 2 and 3.
- Understanding of fact tables, dimension tables, grain and conformed dimensions.
- Working knowledge of Data Vault modelling and Kimball vs. Inmon methodologies.
4. T-SQL
- Strong experience in advanced T-SQL.
- Knowledge of window functions and MERGE/upsert operations.
- Experience with indexing strategies and execution-plan-based query optimization.
- Ability to write and optimize complex SQL queries.
5. Data Architecture
- Experience designing Medallion Architecture (Bronze, Silver and Gold).
- Understanding of ETL vs. ELT and batch vs. streaming processing.
- Experience with orchestration tools such as Databricks Workflows, Apache Airflow or Azure Data Factory.
- Experience working with Azure or AWS cloud environments.
- Strong understanding of scalable data platform architecture and performance optimization.
Experience Requirements
- 8–12 years of overall experience in Data Engineering.
- Minimum 2 years of experience in a technical leadership or mentoring capacity.
- Proven experience leading technical discussions, reviewing code and mentoring engineering teams.
Good to Have
- PCEP / PCAP or equivalent Python certification.
- Databricks Certified Data Engineer Professional certification.
- Experience with data governance and compliance practices such as GDPR/HIPAA.
- Knowledge of FinOps and cloud cost optimization.
- Previous experience leading or mentoring a team of 3 or more engineers.
Educational Qualification
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering or a related discipline, or equivalent practical experience.
Required Soft Skills
- Strong communication and stakeholder management skills.
- Ability to explain complex technical concepts and trade-offs to technical and non-technical stakeholders.
- Strong analytical, problem-solving and decision-making abilities.
- Ability to take ownership of enterprise-level technical decisions.
- Effective leadership, mentoring and collaboration skills.
