Senior Data Engineer

Role - Data Engineer

Experience - 3+Yrs

Location - Bangalore

Technical Priorities

• Real-time CDC: Ownership of high-throughput ingestion from RDBMS to Lakehouse using Debezium, PeerDB.
• Lakehouse Architecture: Designing and optimizing table formats (Iceberg, Delta, Hudi) for both performance and storage efficiency.
• Unified Compute: Developing robust ETL/ELT frameworks in PySpark and Flink (handling both batch and streaming workloads).
• Infrastructure & Ops: Managing data workloads on AWS (EMR, EKS, MSK, S3) and automating everything via Gitlab/Github Actions.
• Query & BI: Tuning Trino or Clickhouse to power real-time dashboards in Metabase, Superset, and PowerBI.

Requirements

• Experience: 3–5 years in Data Engineering, specifically with distributed systems and cloud-native architectures.
• Coding: Expert-level Python/PySpark and SQL.
• Familiarity with Go/Java/Scala is a plus
• Infrastructure: Hands-on experience with AWS (S3, EKS, MSK) and Infrastructure-as-Code.
• Orchestration: Experience with Airflow or Temporal for complex workflow management.
• AI-Native: Proficiency in using AI tools (Claude, Codex, Copilot) to write, test, and document code efficiently.
• Systems Thinking: Ability to explain the trade-offs between different storage formats and processing frameworks.
• Tech LeaderShip : Drive key tech initiatives by preparing TRD and actively involve in design reviews.
• Domain Modelling - Should be hands on in designing Domain models for OLAP like Fact, Dimension and types of SCD's and OBT pattern tables.
• Self Starter - Lead the team technically and bring in new ideas to contribute to the growth of the charter.
• Customer First - Interact with the Product & Key Stakeholders & help them by adding value to the business workflow with data & analytics.

Our Tech Stack

• Ingestion: Debezium, PeerDB, Olake
• Storage: Delta, Iceberg, Hudi (S3-based Lakehouse)
• Compute: PySpark, Flink, EMR, EKS
• Streaming: MSK (Kafka)
• Query Engines: Trino, Clickhouse
• Orchestration: Airflow, Temporal
• DevOps: Gitlab, Github Actions, Terraform

Visualization: Metabase, Superset, Tableau, PowerBI

Key Responsibilities

• Execute predefined command playbooks to perform first-level incident investigation.
• Run diagnostic commands to identify root causes across application, database, and infrastructure layers.
• Follow standard operating procedures (SOPs) to handle 5–7 probable causes per incident type.
• Apply first-level mitigation:
• Update configuration values as per playbooks.
• Adjust alert thresholds and temporarily silence non-critical alerts.
• Document incident steps, findings, and resolutions clearly in ticketing/monitoring tools.
• Collaborate with engineering/SRE teams for escalation and deeper troubleshooting.
• Contribute feedback to improve and automate playbooks over time.

Must-have Qualifications

• 2–5 years of experience in technical support / product support / L1–L2 operations roles.
• Strong understanding of incident/ticket lifecycle and customer-impact awareness.
• Ability to follow technical runbooks and work with command-line tools.
• Good troubleshooting mindset and structured approach to problem solving.
• Clear written and verbal communication skills.