Role Description

We are seeking a Data Engineer with strong Databricks expertise to support the NGP (Next Generation Platform) data ecosystem. The role focuses on building scalable data pipelines, optimizing Databricks jobs, and enabling high-performance data processing frameworks within a cloud-based architecture.

The candidate will work closely with Snowflake, Azure, and distributed data platforms to deliver robust, production-grade data solutions supporting financial analytics and reference data systems.

Databricks & Data Engineering

Design, develop, and maintain Databricks-based ETL pipelines using Apache Spark.
Build scalable data processing frameworks for ingestion, transformation, and validation
Optimize Databricks jobs, clusters, and workflows for performance, reliability, and cost efficiency.
Develop and manage job pipelines, scheduling, and orchestration in Databricks environments.

Troubleshoot Databricks job failures, performance bottlenecks, and pipeline issues.
Improve execution efficiency through cluster tuning, job parallelization, and batching strategies.
Support production stability and L4-level troubleshooting for critical pipelines.
The role requires close collaboration with cross-functional teams in an Agile environment to deliver data-driven solutions. Exposure to AI tools such as GitHub Copilot and modern development practices will be considered an added advantage

Core/Must have skills

Strong experience in:
Databricks + Apache Spark
SQL / PySpark / Scala
Hands-on with:
ETL pipeline development and optimization
Distributed data processing frameworks
Experience with:
Snowflake and data warehousing concepts
Azure cloud platform
Strong analytical and debugging skills in complex data systems

Good to have skills

Experience in:

Delta Lake / Unity Catalog
CI/CD pipelines (Harness / Azure DevOps)
Data quality frameworks / validation pipelines
Exposure to:
Financial domain / performance analytics / reference data platforms
Knowledge of AI-assisted development tools (e.g., GitHub Copilot)

Work Schedule

Regular shift

Keywords (If any)

Why this role is important to us

Our technology function, Global Technology Services (GTS), is vital to State Street and is the key enabler for our business to deliver data and insights to our clients. We're driving the company's digital transformation and expanding business capabilities using industry best practices and advanced technologies such as cloud, artificial intelligence and robotics process automation.

We offer a collaborative environment where technology skills and innovation are valued in a global organization. We're looking for top technical talent to join our team and deliver creative technology solutions that help us become an end-to-end, next-generation financial services company.

Join us if you want to grow your technical skills, solve real problems and make your mark on our industry.

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you'll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at

Read our

Data Services Engineering - Assistant Vice President