Lead Data Engineer

Role:

Job Summary

We are looking for a strong Data Engineer / Lead Data Engineer with 5+ years of solid experience in Data Engineering to design, build, optimize, and maintain scalable data platforms and pipelines. The ideal candidate should be well versed in all core aspects of modern data engineering, including data ingestion, transformation, modeling, orchestration, governance, performance tuning, and production support.

This role requires strong hands-on experience with at least one major cloud platform such as Azure, AWS, or GCP, along with solid coding skills in Python and Spark. Experience with Snowflake and/or Databricks is a strong advantage. The candidate should have a good understanding of medallion architecture, orchestration frameworks such as Airflow or cloud-native alternatives, and working knowledge of DevOps practices. Exposure to RAG architecture and Vector Databases will be a major plus.

Key Responsibilities

• Design, develop, and maintain scalable, reliable, and secure data pipelines and data platforms
• Build batch and near real-time ingestion and transformation frameworks
• Implement robust data models and data processing layers using modern architectural patterns, including medallion architecture
• Work with structured, semi-structured, and unstructured data from multiple source systems
• Develop and optimize ETL/ELT workflows using Python, Spark, SQL, and cloud-native services
• Orchestrate and monitor workflows using Airflow or equivalent cloud orchestration tools
• Build and maintain data solutions on cloud platforms such as Azure, AWS, or GCP
• Work with platforms such as Snowflake and/or Databricks for scalable analytics and engineering workloads
• Ensure data quality, observability, lineage, reliability, and performance across the platform
• Collaborate with architects, analysts, data scientists, application teams, and business stakeholders to translate requirements into technical solutions
• Support CI/CD, infrastructure automation, deployment pipelines, and environment management as part of engineering and DevOps practices
• Troubleshoot production issues, perform root cause analysis, and implement preventive fixes
• For Lead Data Engineer candidates: provide technical leadership, guide design decisions, mentor junior engineers, and enforce engineering best practices

Required Skills and Experience

• 5+ years of strong hands-on experience in Data Engineering
• Strong understanding of data engineering concepts across ingestion, transformation, storage, orchestration, modeling, and optimization
• Deep knowledge of at least one cloud platform: Azure or AWS or GCP
• Strong programming experience in Python
• Strong hands-on experience with Apache Spark / PySpark
• Strong SQL skills and experience working with large-scale datasets
• Experience designing and implementing ETL/ELT pipelines
• Good understanding and practical experience with medallion architecture
• Experience with workflow orchestration tools such as Airflow or equivalent cloud-native tools
• Experience with DevOps practices, including CI/CD, code versioning, deployment automation, and environment management
• Understanding of data security, access control, and performance optimization in enterprise data platforms
• Strong problem-solving, debugging, and communication skills

Preferred

• Domain experience – Manufacturing, Energy, Oil and Gas, Utilities is nice to have.