Senior Data Engineer

Senior Data Engineer – Job Description

Location - Bangalore

Role Overview

We are looking for a Senior Data Engineer who provides technical leadership and hands-on ownership for complex, high-impact data engineering solutions across client engagements.

This role sits between Data Engineer and Principal Engineer, extending senior execution into cross-project design decisions, standards, and accelerators, while still delivering critical components end-to-end.

The Principal Data Engineer reduces ambiguity, drives technical direction, and raises engineering quality across teams.

Key Responsibilities

Technical Leadership & Solution Design

• Lead design for complex data subsystems (ingestion, transformation, curation, serving).
• Drive key design choices such as batch vs streaming, lakehouse vs warehouse, orchestration approach, and partitioning strategy.
• Define and enforce standards for pipeline design, coding practices, documentation, and production readiness.

Hands-on Data Engineering Ownership

• Design, build, and optimize ETL / ELT pipelines using Python, SQL, and distributed processing frameworks (Spark / Databricks).
• Own complex integrations across multiple source systems (APIs, files, CDC/event streams).
• Optimize pipelines for performance, reliability, and cost efficiency.

Orchestration, Reliability & Operations

• Architect and implement orchestration patterns (Airflow or equivalent), including retries, backfills, SLAs, and dependency management.
• Establish monitoring, alerting, and data quality checks.
• Lead triage and resolution of high-severity data pipeline issues.

Platform Enablement (Applied, Not Architectural-Heavy)

• Implement CI/CD practices for data pipelines.
• Build reusable components and templates (ingestion patterns, validation utilities, standard DAGs) to accelerate delivery.
• Partner with Architects to align solutions with modern cloud data platform patterns where needed.

Mentoring & Collaboration

• Mentor Senior and mid-level engineers through design and code reviews.
• Clearly communicate design decisions and trade-offs to engineers, product teams, and clients/stakeholder.

Design & Architecture Articulation

• Produce and maintain design docs and Architecture Decision Records (ADRs) including trade-offs, risks, and rollout plans.
• Communicate design intent and responsibilities across teams clearly.

Must-Have Skills

• Strong proficiency in Python and SQL.
• Deep hands-on experience with Spark / PySpark / Databricks.
• Solid experience designing production-grade ETL / ELT pipelines and orchestration workflows.
• Strong understanding of data modeling, data warehouses, and data lakes.
• Practical experience with cloud platforms (AWS / Azure / GCP).
• Proven ability to lead technical decisions, mentor engineers, and deliver under ambiguity.

Nice-to-Have Skills

• Experience with streaming or event-driven data pipelines (Kafka, Kinesis, Pub/Sub).
• Exposure to data observability, governance, or catalog tools.
• Familiarity with containers, IaC, or platform enablement practices.
• Relevant cloud or data engineering certifications.

What Success Looks Like in This Role

• You deliver the hardest and most ambiguous parts of client data platforms.
• You reduce ambiguity by turning requirements into clear designs and execution plans.
• Your pipelines are reliable, observable, cost-aware, and production-ready.
• You uplift the DE practice through standards, reusable components, and mentorship.