Roles & Responsibilities :
- Design and implement test automation strategies for backend APIs, agent orchestration logic, and Angular-based agent UIs.
- Build and maintain Pytest-based test suites for Python backend services and agent workflow validation.
- Implement Playwright automation for Angular-based UI, covering interaction flows, approval queues, and streaming response handling.
- Validate multi-step agent workflows by asserting on intermediate states, tool call outputs, and handoff results, not just final API responses.
- Use Langfuse traces to validate agent reasoning chains, confidence scores, and execution paths.
- Build and maintain LLM evaluation harnesses covering prompt regression testing, RAG output validation, and model version change detection.
- Integrate test suites into Azure DevOps CI/CD pipelines as automated quality gates.
- Verify audit log integrity, including immutable log structure, correlation ID chains, and SOX-compliant event emission.
Test Automation - Primary Gate
- 5+ years of QA automation experience for API and web-based applications.
- Pytest as the primary test framework for Python backend and agent logic validation.
- Playwright for UI automation targeting Angular-based interfaces.
- REST API testing using Postman, HTTPX, or equivalent tools for contract and schema validation.
- Python for test scripting, strongly preferred given the stack.
- SQL/NoSQL for test data setup, teardown, and database state validation.
- Azure DevOps CI/CD pipeline integration for automated quality gates.
LLM / Agent Testing Awareness - Required
- Understands that LLM outputs are non-deterministic and knows why standard assertion patterns must be adapted, even without prior hands-on production experience.
- Familiarity with multi-step agent graph testing, including asserting on intermediate states, tool calls, and handoff outputs rather than just end results.
- Understanding of what Langfuse traces contain and how they can be used for agent validation. Hands-on production experience is a plus but not required.
- Awareness of model version regression risks and how to detect quality degradation when model versions change.
