Overview
We are looking for an experienced
QEIII (Senior Quality Engineer) to lead quality validation for our
agentic AI models product to assess compliance, reasoning outcomes, and evaluation status.
This role requires strong expertise in
test automation using Selenium WebDriver,
CI/CD enablement, and solid experience validating
AI-driven systems, including model outputs, confidence scores, and decision traceability. The QEIII will act as a
quality owner, partnering closely with Product, Engineering, and Data Science teams.
Responsibilities
AI & Compliance Quality Ownership
- Own endtoend quality strategy for the AI agent models/ Agentic AI platform, covering UI, backend services, and AI model outputs.
- Validate AI agent behavior, including:
- Decision correctness
- Evaluation status
- Confidence thresholds
- Explainability / traceability of responses
- Define test approaches for nondeterministic AI behavior (golden datasets, tolerance-based assertions, confidence validation).
- Partner with Data Science teams to validate model inference results, regression drift, and version-to-version consistency.
Automation & Framework Development
- Design, develop, and maintain robust Selenium WebDriver automation frameworks (Java preferred).
- Implement automation for:
- Web UI workflows
- Compliance dashboards
- Agent evaluation flows
- Admin & configuration screens
- Ensure automation follows industry best practices: Page Object Model (POM), reusable utilities, proper wait strategies, and stable selectors.
- Page Object Model (POM)
- Datadriven and keyworddriven testing
- Reusable libraries and utilities
- Robust synchronization and wait strategies to reduce flakiness
- Integrate automation suites into CI/CD pipelines using:
CI/CD & Quality Engineering Practices
- Own Jenkins pipelines for automated test execution (smoke, regression, AI evaluation suites).
- Enable automation execution as part of:
- Pull request validation
- Nightly regression
- Release readiness checks
- Analyze test failures, improve flakiness, and ensure high signaltonoise ratio in automated results.
- Produce actionable test metrics (coverage, defect leakage, automation ROI).
Functional, Integration & Regression Testing
- Drive functional, integration, and regression testing across multiple services and AI agents.
- Validate:
- API integrations (REST)
- Data flows across agent pipelines
- Evaluation status lifecycle (queued processed evaluated completed).
- Perform exploratory testing on new AI capabilities and edge cases.
Leadership & Collaboration
- Act as QE SPOC for the pod / product area.
- Mentor junior engineers on automation, testing strategies, and AI validation approaches.
- Participate in requirement reviews and proactively identify quality risks early in the SDLC.
Qualifications
Core Experience
- 69+ years of experience in Software Quality Engineering.
- Strong handson experience in test automation using Selenium WebDriver.
- Proficiency in Java (or similar OOP language) for automation.
- Solid experience working with:
- Maven (build & dependency management)
- Jenkins (CI pipelines)
- GitHub (version control, PR workflows)
AI / DataDriven Systems (MustHave)
- Experience testing AI/MLdriven applications, analytics platforms, or rule/decision engines.
- Understanding of:
- Model inference pipelines
- Confidence scores
- Evaluation metrics
- Handling nondeterministic outputs
- Ability to design validation strategies for AI agents, beyond simple pass/fail testing.
Testing & Tooling
- Strong experience with:
- API testing (REST)
- TestNG / JUnit
- Agile/Scrum methodologies
- Familiarity with test management and defect tracking tools (Azure DevOps TFS, etc.).
- Experience validating complex workflows across UI + API layers.
NicetoHave
- Experience in compliance, audit, or regulatory systems.
- Exposure to performance testing (e.g., JMeter) for dashboards or AI pipelines.
- Knowledge of logging, monitoring, and observability tools for troubleshooting AI behavior.
- Experience validating data privacy and security aspects of AI systems.
