Senior Automation QA Engineer
About The Role
We are looking for a
Senior Automation QA Engineer to own end-to-end test automation across API and UI layers for our AI-native platform, including agent workflow validation.
Key Responsibilities
- Design and execute test strategies for API and UI layers using Pytest and Playwright (Angular).
- Own CI/CD quality gates in Azure DevOps, ensuring reliable build promotion and release readiness.
- Build and maintain coverage for agent workflow testing, including non-determinism, agent graph validation, and confidence score assertions.
- Develop reusable automation frameworks and data setup utilities backed by PostgreSQL.
- Contribute to LLM evaluation and observability workflows using Langfuse or similar tooling.
Required Qualifications
- 6+ years of experience in QA automation.
- Strong hands-on expertise with Pytest and Playwright, with proven depth in both API and UI automation.
- Solid proficiency in Python and experience working with PostgreSQL for test data and validation.
- Hands-on experience with Azure DevOps CI/CD pipelines and quality gates.
- Awareness of LLM/agent testing challenges (non-determinism, agent graph validation, confidence score assertions); exposure to Langfuse or LLM evaluation frameworks is a plus and can be built on the job.
