Senior Data Engineer

Role Summary

NTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.

The role requires a strong hands-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves.

This is not a coordination-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers.

Experience

10+ years of Data Engineering experience, including:

. 5+ years hands-on Apache Spark/PySpark.

. Strong Hadoop/Hive experience.

. Experience leading Data Engineering teams.

. Large-scale data migration/data modernization experience.

. Strong data modelling and source-to-target mapping experience.

Multi-hundred-TB/PB-scale migration experience is strongly preferred.

Key Responsibilities

Technical Leadership

. Lead a team of Data Engineers through assessment, development, migration, validation and production execution.

. Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.

. Own technical estimates, engineering backlog, work allocation and technical delivery.

. Conduct design and code reviews and enforce engineering standards.

. Mentor Data Engineers and provide hands-on technical guidance.

. Act as the primary engineering escalation point for complex migration and production issues.

. Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.

. Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions.

Hadoop-to-Iceberg Migration

. Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.

. Harden and extend the reusable, configuration-driven migration framework developed during the POC.

. Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.

. Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.

. Design execution approaches for small, medium, large and very-large tables.

. Optimize parallel migration across tables and partitions.

. Implement checkpointing, restart/recovery, idempotency and safe retries.

. Lead technical execution and troubleshooting of migration waves.

Data Modelling & Schema Engineering

. Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.

. Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.

. Perform source-to-target data mapping and schema/data-type mapping.

. Define and implement table structures consistent with approved Iceberg standards.

. Analyze:

o entities and relationships,

o primary/foreign-key relationships,

o data types,

o partition keys,

o business keys,

o reference/cross-reference structures.

. Identify schema incompatibilities and recommend remediation.

. Work with the Data Architect on partitioning, file-layout and model-design decisions.

. Ensure migrated models preserve required business relationships and downstream compatibility.

. Support future Bronze/Silver/Gold modelling requirements where applicable.

Important: enterprise modelling standards remain owned by the Data Architect the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards.

Spark/PySpark Engineering & Performance

. Develop and review production-grade Python/PySpark components.

. Tune Spark jobs for PB-scale migration.

. Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.

. Optimize table/partition/chunk parallelism.

. Tune workloads to meet migration-throughput objectives.

. Identify bottlenecks across:

Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation

. Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework.

Apache Iceberg Engineering

. Implement approved Iceberg table standards.

. Develop/configure Iceberg tables and Spark→Iceberg processing.

. Implement:

o partitioning,

o schema evolution,

o snapshots,

o file sizing,

o compaction,

o metadata maintenance,

o small-file remediation.

. Troubleshoot Iceberg catalog/write/commit issues.

. Optimize Iceberg table performance and storage characteristics.

Cyber-3 / Tokenization

. Implement PNC-approved TaaS/Protegrity tokenization and retokenization patterns.

. Integrate tokenization into Spark migration pipelines.

. Implement batching, retry, timeout and recovery handling.

. Troubleshoot tokenization failures/performance bottlenecks.

. Measure impact of tokenization on end-to-end migration throughput.

. Support Cyber/tokenization validation based on PNC-defined rules.

Migration Controls & Reconciliation

. Lead implementation of PNC Data Movement Controls, including:

o checkpointing,

o idempotency,

o control totals,

o restart/recovery,

o retry,

o audit logging,

o exception handling.

. Build reusable source-to-target reconciliation capabilities covering:

o row counts,

o schema validation,

o partition completeness,

o control totals,

o aggregates,

o hash/checksum where appropriate,

o null/duplicate validation,

o tokenization validation.

. Isolate discrepancies at table/partition level.

. Enable selective reprocessing rather than full-table reruns.

. Produce technical evidence required for migration acceptance.

Mandatory Skills

Strong hands-on expertise in:

. Python

. PySpark

. Apache Spark / Spark SQL

. Hadoop / HDFS

. Hive / HiveQL / Hive Metastore

. SQL

. Apache Iceberg

. Data modelling

. Source-to-target mapping

. Parquet

. Linux

. Git

. Large-volume ETL/data migration

. Data validation/reconciliation

. Spark performance tuning

. Distributed-system troubleshooting.

Data Modelling

Strong experience with:

. logical and physical data modelling,

. source-to-target mapping,

. schema/data-type mapping,

. entity relationships,

. keys and referential relationships,

. dimensional/relational modelling concepts,

. large-scale analytical data models.

Technical Leadership

Demonstrated experience:

. leading Data Engineering teams,

. technical estimation/planning,

. detailed solution design,

. code/design reviews,

. mentoring engineers,

. production troubleshooting,

. root-cause analysis,

. communicating technical issues with architects/customer teams.

Preferred Skills

. Apache Polaris/Nessie/Glue.

. OpenShift/Kubernetes/Spark Operator.

. S3-compatible object storage.

. NetApp StorageGRID/ONTAP.

. TaaS/Protegrity or equivalent tokenization technology.

. Trino.

. PostgreSQL.

. Control-M/Airflow.

. CI/CD for Spark/Python applications.

. Banking/financial-services experience.

. PB-scale data migration.

. Bronze/Silver/Gold modelling and implementation.

. HiveQL→Spark SQL modernization.