Role: Senior Developer - Modern Data Platform Architect

Role Summary

We are seeking a highly skilled Senior Developer / Data Platform Engineer to design and build scalable data pipelines and modern Lakehouse solutions on Databricks-based platforms. The candidate should have strong expertise in PySpark, ETL/ELT frameworks, Data Quality, Apache Iceberg, and job orchestration, with exposure to asset management domain considered a strong advantage.

Key Responsibilities

1. Data Platform Development & Engineering

Design and develop scalable data ingestion and transformation pipelines using PySpark and SQL.
Build and manage batch and streaming ETL/ELT pipelines on Databricks.
Implement data processing frameworks supporting high-volume enterprise datasets.

2. Lakehouse Architecture & Iceberg Implementation

Design and maintain modern data Lakehouse architecture (Bronze/Silver/Gold layers).
Work on Apache Iceberg / open table formats including schema evolution, partitioning, and compaction strategies.
Ensure interoperability across engines and optimize data storage formats for performance.

3. Data Ingestion & Pipeline Management

Design and monitor end-to-end ingestion pipelines (batch + near real-time).
Build reusable ingestion frameworks for structured and semi-structured data.
Manage dependencies, data lineage, and pipeline orchestration workflows.

4. Data Transformation & dbt (Nice to Have)

Implement transformation logic using dbt or similar ELT frameworks.
Standardize transformation pipelines for maintainability and versioning.

5. Data Quality & Observability

Implement robust data quality checks, validation rules, and reconciliation processes.
Develop monitoring frameworks for pipeline failures, SLA tracking, and alerting.
Ensure data governance and audit compliance across datasets.

6. Job Orchestration & Workflow Management

Implement and manage orchestration tools (Airflow / Databricks Jobs / other schedulers).
Handle pipeline scheduling, retries, dependency chaining, and failure recovery.

7. CI/CD & DevOps Enablement

Build and maintain CI/CD pipelines for data workflows and infrastructure.
Automate deployment of Databricks jobs, notebooks, and configurations.
Manage code versioning, release management, and environment promotion.

8. Performance Optimization

Optimize Spark jobs for cost, performance, and scalability.
Tune queries, partitioning strategies, and resource utilization.

9. Collaboration & Stakeholder Engagement

Work with business teams, data architects, and upstream/downstream systems.
Translate business requirements (especially in asset management domain) into technical solutions.
Support data modeling (Data Vault / Star Schema exposure is a plus).

10. Platform Governance & Best Practices

Follow enterprise standards for data security, access control, and governance.
Implement Unity Catalog / RBAC / metadata management best practices.
Ensure compliance with platform architecture and operational standards.

Required Skills

Must Have

Strong experience in Databricks and PySpark
ETL/ELT pipeline design and development
Data ingestion and pipeline orchestration
SQL and distributed data processing
Data quality and validation frameworks
Experience with Apache Iceberg / open table formats
Cloud experience (AWS preferred)

Good to Have

Exposure to Asset Management / Financial Data domain
dbt (Data Build Tool)
Data modeling (Data Vault / Star Schema)
Knowledge of Snowflake / Unity Catalog
Experience with CI/CD (Terraform, Git-based deployments)

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you'll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at

Read our

Senior Data Engineer - Senior Manager