QA Automation Architect AI/LLM Systems | 13+ years only | PAN India

What impact will you make

Every day, your work will make an impact that matters, while you thrive in a dynamic culture of inclusion, collaboration and high performance. As the undisputed leader in professional services

Deloitte is where you will find unrivaled opportunities to succeed and realize your full potential.

The Team

Deloitte's Technology & Transformation practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning.

QA Automation Architect – AI/LLM Systems

Experience: 13+ Years

Location: PAN India

Employment Type: Full-Time

About the Role

We are looking for a highly experienced QA Automation Architect to lead the quality engineering vision for AI-driven and Generative AI applications. This role goes beyond traditional QA—focusing on LLM validation, AI safety, governance, and scalable automation architecture, including integrations with enterprise platforms such as Adobe Experience Manager (AEM).

You will play a critical role in ensuring that AI systems are accurate, reliable, safe, and compliant, while driving enterprise-wide standards for AI quality engineering.

What You'll Do

Architect & Lead Quality Engineering

• Define and drive the end-to-end QA strategy for AI/LLM-based platforms
• Design and implement scalable automation frameworks using Playwright (TypeScript) and BDD (Python/Behave)
• Establish reusable test architecture, patterns, and best practices across teams
• Drive test strategies across UI, API, and CMS-driven applications (including AEM-based platforms)
• Champion shift-left testing and automation-first culture

AI/LLM Testing & Quality Validation

• Build frameworks to validate:
• Accuracy, relevance, and contextual correctness
• Bias, fairness, and ethical AI behavior
• Safety, toxicity, and harmful prompt handling
• Fallback and edge-case scenarios
• Design and implement LLM evaluation strategies, including automated scoring and human-in-loop validation
• Create feedback loops and quality dashboards for continuous improvement

Automation & API Engineering

• Architect robust end-to-end and API test automation suites
• Drive high coverage across UI, API, backend, and CMS layers (including AEM components and workflows)
• Implement test data strategies for deterministic and repeatable AI testing

Governance, Compliance & Observability

• Define and enforce Responsible AI testing standards and governance frameworks
• Ensure traceability, auditability, and regulatory compliance
• Establish standards for prompt versioning, dataset control, and experiment tracking
• Validate content workflows and publishing pipelines in systems like AEM
• Leverage logs, traces, and observability tools to validate system behavior

CI/CD & DevOps Integration

• Integrate automation into CI/CD pipelines with quality gates
• Enable test orchestration, parallel execution, and environment consistency (Docker/Kubernetes)
• Define and track quality KPIs and SLAs

Performance & Reliability

• Drive performance testing (p50/p95 latency, throughput, scalability) for AI systems
• Validate performance across APIs, AI services, and CMS platforms (e.g., AEM)
• Identify bottlenecks in model inference and service layers

Leadership & Influence

• Mentor QA engineers and SDETs; elevate team capabilities
• Collaborate with Engineering, Product, Data Science, CMS/AEM teams, and Security teams
• Act as a thought leader in AI quality engineering and automation

Must-Have Skills & Experience

• 13+ years in QA/Automation with architect-level experience
• Strong expertise in Playwright + TypeScript (mandatory)
• Proven hands-on experience in AI/LLM testing and validation
• Deep experience in building scalable automation frameworks
• Strong experience with:
• API testing (Playwright, Postman, RestAssured)
• BDD (Python + Behave)
• CI/CD tools (Jenkins, GitHub Actions, Azure DevOps, etc.)
• Solid understanding of Agile, DevOps, and modern engineering practices

Good to Have

• Experience with Generative AI systems in production
• Knowledge of Responsible AI / AI Governance frameworks
• Exposure to Cloud (AWS/Azure/GCP) environments
• Experience with Docker, Kubernetes, microservices
• Understanding of prompt engineering and versioning
• Experience working with ML/Data Science teams
• Hands-on experience or exposure to Adobe Experience Manager (AEM) (component testing, workflows, content validation)
• Background in regulated domains (Fintech, Healthcare, SaaS)

Why Join Us

• Work on cutting-edge AI/LLM platforms
• Opportunity to shape enterprise AI quality standards
• High-impact role with architectural ownership and visibility
• Collaborative environment with cross-functional innovation

Ideal Candidate Profile

A seasoned QA leader who:

• Thinks architecturally, not just functionally
• Has a passion for quality in AI systems and digital experience platforms (like AEM)
• Balances technical depth with strategic vision
• Drives innovation while ensuring safety and compliance

How you will grow

At Deloitte, our professional development plan focuses on helping people at every level of their career to identify and use their strengths to do their best work every day. From entry-level employees to senior leaders, we believe there is always room to learn. We offer opportunities to help build excellent skills in addition to hands-on experience in the global, fast-changing business world. From on-the-job learning experiences to formal development programs at Deloitte University, our professionals have a variety of opportunities to continue to grow throughout their career. Explore Deloitte University, The Leadership Centre.

Benefits

At Deloitte, we know that great people make a great organization. We value our people and offer employees a broad range of benefits. Learn more about what working at Deloitte can mean for you.

Our purpose

Deloitte is led by a purpose: To make an impact that matters.

Every day, Deloitte people are making a real impact in the places they live and work. We pride ourselves on doing not only what is good for clients, but also what is good for our people and the

Communities in which we live and work—always striving to be an organization that is held up as a role model of quality, integrity, and positive change. Learn more about Deloitte's impact on the world

Recruiter tips

We want job seekers exploring opportunities at Deloitte to feel prepared and confident. To help you with your interview, we suggest that you do your research: know some background about the

organization and the business area you are applying to. Check out recruiting tips from Deloitte professionals.