Principal Data Platform Engineer

Principal Data Engineer:

Experience: 9+ Years  

Work Mode: Onsite 

Location: Bangalore 

Principal Data Platform Engineer 

Architecture: Lakehouse (Medallion: Bronze/Silver/Gold)  

Compute: Apache Spark (Expert level)  

Storage/Table Format: Delta Lake (Required), Iceberg (Strong Plus)  

Transformation: dbt (Expert level)  

Orchestration: Airflow, Cosmos  

Infrastructure: Cloud-native (GCP preferred) + Databricks/Commercial tooling  

Patterns: Microservices, Event-driven, CI/CD, IaC (Terraform)  

Shape  

Core Technical Requirements  

1. Data Engineering & Spark Internals  

Deep Spark: You must understand RDDs, DataFrames, Spark SQL, and internals (Shuffle, 

Partitioning, Memory Management, Catalyst Optimizer).  

Pipeline Mastery: Building idempotent, self-healing ELT/ETL pipelines. Experience with Schema 

Evolution and handling late-arriving data.  

Lakehouse ACID: Expert knowledge of transaction logs, time travel, and file compaction in 

Delta/Iceberg.  

2. Software Architecture & Design 

Engineering First: This isn't just SQL and scripts. You apply SOLID principles, design patterns, 

and write production-grade Python/Scala/Java.  

Integration: Experience building and consuming Microservices. Knowledge of API design 

(REST/gRPC) and message brokers (Kafka/PubSub).  

System Design: Experience building a platform from scratch. You know how to design for 99.9% 

availability and horizontal scalability.  

3. Data Modeling & dbt  

Modeling: Expert in dimensional modeling (Kimball), Data Vault 2.0, or OBT (One Big Table) for 

high-performance analytics.  

dbt Power User: Advanced dbt usage (Macros, Packages, Custom Tests, dbt Mesh). You treat 

dbt projects like software repositories (version control, PR reviews, CI).  

4. Cloud & Platform 

Cloud Native: Deep understanding of IAM, VPCs, Object Storage, and serverless compute.  

Migrations: Proven track record of moving petabyte-scale data from legacy systems (On-prem, 

Redshift, Snowflake) to a Lakehouse without data loss.  

Shape  

Key Deliverables (First 6-12 Months)  

Platform Zero: Evaluate, select, and deploy the foundational Lakehouse infrastructure.  

Core Frameworks: Build the reusable libraries/templates for the rest of the engineering team to 

build pipelines.  

Legacy Decommission: Design the technical map to migrate all high-priority finance/business 

data to the new stack.  

Performance Baseline: Optimize Spark/Cloud costs by at least 20% through better resource 

management.  

Shape  

The Plus List  

MLOps: Building feature stores and model deployment triggers.  

GCP Specialization: BigQuery (as a Lakehouse layer), Dataproc, and Cloud Composer.  

Observability: Implementing Data Quality monitoring (Great Expectations, Monte Carlo) and 

OpenTelemetry.