Job Description

Role- Lead Data Platform Engineer

Experience- 8 to 15 years

Location- Pune, Hyderabad, Bangalore

Required Skill set- Spark, Kafka, Flink, Airflow, Iceberg, Polaris/Nessie, Kubernetes

Required Skills & Experience

Must Have

8-15 years of data engineering or data platform experience, with substantial recent hands-on responsibility for distributed processing platforms.
Deep hands-on experience with Apache Spark, including DataFrame or SQL workloads, performance tuning, packaging, configuration, and production troubleshooting.
Strong Apache Kafka experience covering topics, partitions, consumer groups, offsets, delivery semantics, schemas, security, and operations.
Practical Apache Flink experience covering state, checkpoints, savepoints, event time, recovery, and backpressure.
Strong understanding of Apache Iceberg, Google Cloud Storage, schema and partition evolution, snapshots, compaction, and table maintenance.
Experience with Apache Polaris, Project Nessie, or REST-based Iceberg catalogs.
Hands-on experience with Apache Airflow for distributed data-processing orchestration.
Working knowledge of GKE-based execution, containers, Helm, service accounts, Workload Identity Federation, secrets, networking, storage, and observability.
Proficiency in Python, Java, or Scala, SQL, Git, automated testing, CI/CD, and reusable software engineering practices.
Strong technical leadership, troubleshooting, documentation, mentoring, and stakeholder communication skills.

Good to Have

Experience with Trino, OpenMetadata, JupyterHub, Dataproc, data-quality frameworks, lineage, or schema registries.
Exposure to Debezium, Kafka Connect, Datastream, database logs, or comparable CDC patterns.
Experience with Spark Operator, Flink Kubernetes Operator, Strimzi, KubernetesExecutor, or operator-based GKE integration.
Understanding of data security, encryption, masking, governance, data contracts, and regulated enterprise controls.

Experience with multi-tenant data platforms, regulated environments, and platform cost optimisation on GCP.