Job Title: Senior /Lead Data Engineer
Location: Velachery ,Chennai
Employment Type: Full-Time (onsite, 5days )
Experience Level: 7-12 years
Department: Data Engineering
Job Summary:
We are looking for a Lead Data Engineer with strong expertise in Azure, advanced skills in Spark, SQL, PySpark, and data modeling/design. The role involves leading the design and development of scalable data pipelines, driving best practices, mentoring the team, and managing complex real-time/streaming data solutions.
Key Responsibilities:
- Lead the design, development, and optimization of scalable and efficient ETL/ELT data pipelines.
- Oversee data cleaning, transformation, and validation from diverse structured and unstructured sources.
- Collaborate closely with Data Scientists, Analysts, and stakeholders to define and deliver robust data solutions aligned with business goals.
- Ensure data quality, consistency, and integrity across all data platforms and pipelines.
- Develop and maintain comprehensive documentation for data architecture, processes, and best practices.
- Drive code reviews, enforce engineering standards, and promote continuous improvement across the team.
- Lead the implementation of streaming data technologies and manage real-time data processing solutions.
- Mentor and coach junior and mid-level data engineers, fostering technical growth and ensuring successful project delivery.
- Provide technical leadership in architectural decisions and contribute to the overall data strategy.
Required Skills & Qualifications:
- 7+ years of experience in data engineering or related roles, with proven leadership in complex projects.
- Strong expertise in Azure Data Services (ADF, Databricks, ADLS Gen2, Key Vault, Event Hub).
- Advanced proficiency in SQL and programming languages such as Python or Scala.
- Deep understanding of Apache Spark and Spark Core concepts.
- Extensive experience with Azure Databricks for large-scale data engineering workloads.
- Strong knowledge of data warehousing, data modeling, and relational database concepts.
- Exposure to cloud platforms (Azure preferred; AWS or GCP is a plus).
- Hands-on experience with ETL tools and enterprise data integration frameworks.
- Excellent analytical, debugging, and problem-solving skills.
- Proven ability to lead teams, mentor engineers, and collaborate across functions.
- Knowledge of streaming technologies such as Apache Flink and/or programming in Java is an added advantage.
- Familiarity with version control and CI/CD tools like Git and Azure DevOps.
- Strong communication skills with the ability to drive technical discussions and influence stakeholders.
Note : Immediate Joiners/ Candidates who can join within 15days are preferred.
