Role Summary:

We are seeking a Senior QA Engineer specializing in AI and Multi-Agent Systems to ensure the quality, reliability, and governance of next-generation AI platforms. This role focuses on validating agentic AI architectures, LLM-driven systems, and enterprise AI pipelines, combining traditional QA expertise with AI-specific testing strategies.

The ideal candidate has strong experience in test automation and distributed systems, and has evolved into validating AI-driven workflows, including RAG systems, MPC's, agent orchestration, and decision-making models in regulated environments such as banking and financial services.

Key Responsibilities:

1. AI / Multi-Agent System Testing

• Design and implement testing strategies for:
  • Multi-agent architectures (cooperative and autonomous agents)
  • Agent workflows involving planning, reasoning, and tool usage
• Validate:
  • Agent decision-making logic
  • Orchestration across multiple agents and APIs
  • Memory handling (context retention, long-term memory consistency)
• Test AI agent behavior across:
  • Edge cases, failure scenarios, and fallback mechanisms
  • Multi-step workflows and distributed execution

2. LLM & GenAI Testing

• Validate LLM-powered applications, including:
  • Retrieval-Augmented Generation (RAG) systems
  • Prompt engineering workflows and tuning strategies
• Perform:
  • Prompt testing and validation
  • Hallucination detection and response quality checks
  • Output accuracy benchmarking aligned with business expectations
    
• Build automated test frameworks for:
  • Response validation
  • Context relevance
  • Semantic correctness

3. Test Automation & Framework Development

• Develop AI-aware automation frameworks:
  • Extend traditional automation (Selenium, API testing) with AI validation layers
• Automate:
  • Test case generation using AI
  • Regression suites with dynamic test data
• Integrate with tools such as:
  • qTest, JIRA, CI/CD pipelines
• Enable intelligent test case reuse and generation
  

4. Data & Pipeline Validation

• Validate:
  • Data ingestion pipelines
  • Embedding processes and vector databases
  • Retrieval quality from knowledge sources
• Ensure correctness of:
  • Search results
  • Context injection into prompts
  • Data lineage and integrity

5. Enterprise AI Quality & Governance

• Ensure AI systems meet enterprise requirements:
  • Accuracy, explainability, consistency
  • Auditability and traceability
• Support:
  • Responsible AI (RAI) compliance
  • Model risk validation and governance reviews
• Implement monitoring for:
  • Prompt execution traces
  • Agent actions and outcomes
  • Failure patterns and anomalies

6. Performance & Reliability Testing

• Conduct:
  • Load and scalability testing for AI services
  • Latency validation for real-time decision systems
• Validate:
  • Cost efficiency (LLM token usage optimization)
  • System resilience and fallback mechanisms

7. Collaboration & Leadership

• Work closely with:
  • AI engineers, architects, product teams, and business stakeholders
• Mentor QA engineers in:
  • AI testing practices
  • Automation frameworks
• Contribute to:
  • Test strategy definition for enterprise AI programs
  • Architecture reviews and quality governance

Required Qualifications:

Education

• Bachelor's or Master's in Computer Science, Engineering, or related field

Experience

• 10+ years of QA / SDET experience
• 2-4 years of experience in AI/ML or GenAI testing
• Experience in enterprise platforms (preferably banking / financial systems)

Technical Skills:

AI / GenAI Testing

• Understanding of:
  • LLMs, RAG architectures, agentic AI
  • Prompt engineering and evaluation
• Experience validating:
  • Multi-agent workflows
  • AI reasoning systems

Test Automation

• Strong expertise in:
  • Selenium / Playwright / API automation
  • Test frameworks (BDD, Data-Driven)
• Programming:
  • Java / Python

Data & Validation

• Experience with:
  • SQL and data validation
  • API testing and microservices validation
• Exposure to:
  • Vector databases and embeddings (preferred)

DevOps & Tools

• CI/CD: GitHub Actions, Jenkins, Harness
• Containers: Docker, Kubernetes
• Tools: qTest, JIRA, Postman

Preferred Qualifications:

• Experience testing:
  • AI copilots / enterprise assistants
  • Agent-based automation platforms
• Familiarity with:
  • Regulatory compliance (RBI, Basel, data governance)
  • Observability tools (Grafana, Prometheus)

Key Competencies:

• Analytical Thinking: Strong ability to identify AI failure patterns
• Quality Mindset: Focus on reliability, accuracy, and robustness
• Innovation: Adoption of AI-driven testing approaches
• Collaboration: Strong stakeholder communication

Success Metrics:

• High-quality AI releases with minimal production issues
• Improved:
  • Test coverage for AI workflows
  • Accuracy and reliability of AI outputs
• Reduction in:
  • Manual testing effort via AI-driven automation
• Faster validation cycles for AI features

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you'll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at

Read our