Senior Data Engineer

Job Title: Senior Data Engineer

Location: Chennai

Employment Type: Full-Time

Experience Level: 7-10 years

Department: Data Engineering

Job Summary:

We are looking for a Lead Data Engineer with strong expertise in cloud platforms (Google Cloud Platform preferred), advanced skills in Spark, SQL, PySpark, and data modeling/design. The role involves leading the design and development of scalable data pipelines, driving best practices, mentoring the team, and managing complex real-time/streaming data solutions.

Key Responsibilities:

• Lead the design, development, and optimization of scalable and efficient ETL/ELT data pipelines.
• Oversee data cleaning, transformation, and validation from diverse structured and unstructured sources.
• Collaborate closely with Data Scientists, Analysts, and stakeholders to define and deliver robust data solutions aligned with business goals.
• Ensure data quality, consistency, and integrity across all data platforms and pipelines.
• Develop and maintain comprehensive documentation for data architecture, processes, and best practices.
• Drive code reviews, enforce engineering standards, and promote continuous improvement across the team.
• Lead the implementation of streaming data technologies and manage real-time data processing solutions.
• Mentor and coach junior and mid-level data engineers, fostering technical growth and ensuring successful project delivery.
• Provide technical leadership in architectural decisions and contribute to the overall data strategy.

Required Skills & Qualifications:

• 7+ years of experience in data engineering or related roles, with proven leadership in complex projects.
• Strong expertise in cloud data services on a major cloud platform (e.g., Google Cloud Dataflow, Dataproc, Cloud Storage, Secret Manager, Pub/Sub, or equivalent AWS/Azure services).
• Advanced proficiency in SQL and programming languages such as Python or Scala.
• Deep understanding of Apache Spark and Spark Core concepts.
• Extensive experience with cloud-based big data processing (e.g., Google Cloud Dataproc, Databricks) for large-scale data engineering workloads.
• Strong knowledge of data warehousing, data modeling, and relational database concepts.
• Exposure to major cloud platforms (GCP preferred; AWS or Azure is a plus).
• Hands-on experience with ETL tools and enterprise data integration frameworks.
• Excellent analytical, debugging, and problem-solving skills.
• Proven ability to lead teams, mentor engineers, and collaborate across functions.
• Knowledge of streaming technologies such as Apache Flink and/or programming in Java is an added advantage.
• Familiarity with version control and CI/CD tools like Git, Google Cloud Build, or similar CI/CD pipelines.
• Strong communication skills with the ability to drive technical discussions and influence stakeholders