Title QA Engineer
Location India (Remote)
Duration Contract 6 to 12 months
Job Description
Overview
The QA Engineer II plays a crucial role in ensuring the quality and reliability of our Conversational AI solutions. Unlike deterministic software, the AI agents under test are non-deterministic, so benchmarking emphasizes semantic/behavioral evaluation in addition to exact-match assertions.
The ideal candidate has a strong foundation in software testing methodologies, experience with test case design and execution, and familiarity with automation frameworks. This role is ideal for detail-oriented professionals who are eager to learn, contribute to test automation, and help drive continuous improvements in software quality.
Responsibilities
- Agentic Benchmark Design: Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behavior and responses.
- Functional & Regression Testing: Grow benchmarks and create new ones as new functionality or issues are uncovered.
- Benchmarking Automation Support: Contribute to streamlining the benchmarking process in alignment with product development work.
- Collaboration: Work closely with developers, product managers, and senior QA engineers to ensure test coverage aligns with product requirements.
- Defect Identification & Reporting: Log detailed defect reports and work with engineering teams to ensure timely resolution.
- Process & Traceability: Maintain requirement-to-test traceability and document detailed test plans and coverage to support audits/reviews for OEM programs.
- Release Reporting: Produce per-release quality reports as benchmarks evolve and grow over time.
- Performance & Load Testing Support: Assist in load and performance testing using tools such as JMeter.
- Agile Testing & Continuous Improvement: Participate in Agile development cycles, contributing to continuous process improvements.
Qualifications
Basic Qualifications (Non-Negotiable)
- 3+ years of experience in software/product quality assurance.
- Confident usage of Bash and Python.
- Experience with manual testing and test case execution.
- Basic familiarity with automated testing frameworks such as Playwright or Selenium.
- Experience with testing web protocols (WebSocket message payload)
- Basic knowledge of load and performance testing (e.g., JMeter).
- Strong analytical and problem-solving abilities.
- Systems thinking.
- Familiarity with prompt evaluation, function-calling, and tool-use testing.
Preferred Skills
- Exposure to test automation scripting using Python.
- Experience with CI/CD integration.
- Familiarity with Conversational AI platforms and NLP-based applications.
- Experience with test management and result tracking.
- Experience testing conversational-agent systems and evaluating non-deterministic outputs.
General Requirements
- Eager to learn and grow in the field of QA and test automation.
- Strong collaborative mindset and ability to work effectively in a global team environment.
- Ability to adapt to fast-paced development cycles and evolving requirements.
Title
Location:
Duration
SUMMARY:
RESPONSIBILITIES; 
