We're building an agentic AI platform that orchestrates intelligent automation across enterprise systems. This role owns LLM reasoning pipelines end-to-end, from prompt design to production deployment on Azure Kubernetes.
You will own the system's quality, milestone payments depend on acceptance tests.
What You'll Do
- Build pytest automation frameworks for API, integration, and E2E workflows
- Validate LLM structured outputs against expected baselines
- Implement PII/PHI leakage detection tests for AI outputs
- Enforce schema validation (Pydantic/JSON) across APIs
- Integrate tests with CI/CD (GitHub Actions)
- Add security scanning (Gitleaks, Trivy, Bandit)
- Run internal dry-runs before milestone demos
Must-Have
- 3+ years Python test automation (pytest preferred)
- Strong REST API testing experience
- Experience with JSON/schema validation
- CI/CD testing in GitHub Actions or Azure DevOps
- Comfortable validating complex JSON payloads
- Basic Docker knowledge
Strong Plus
- Testing LLM applications
- Security scanning tools
- Azure services (AKS, CosmosDB)
- API contract testing or load testing
- Claude Code + Cursor tooling
- No demo without a green test suite
