AI/LLM Test Automation Engineer (SDET)
Location: Bangalore (Hybrid preferred)
Experience: 5-8 Years
Job Summary
We are seeking a Senior Test Automation Engineer (SDET) with expertise in AI/LLM testing and Azure DevOps CI/CD to build robust automation frameworks for cutting-edge AI applications. The role combines deep programming skills (Java/Python), modern DevOps practices, and specialized LLM testing to ensure high-quality AI product delivery.
Key Responsibilities
- Design, develop, and maintain automation frameworks using Java/Python for web, mobile, API, and backend testing.
- Create and manage YAML-based CI/CD pipelines in Azure DevOps for end-to-end testing workflows.
- Perform AI/LLM testing including prompt validation, content generation evaluation, model behavior analysis, and bias detection.
- Write and maintain BDD Cucumber feature files integrated with automation suites.
- Execute manual + automated testing across diverse application layers with focus on edge cases.
- Implement Git branching strategies, code reviews, and repository best practices.
- Track defects and manage test lifecycle using ServiceNow or similar tools.
- Conduct root-cause analysis, troubleshoot complex issues, and drive continuous quality improvements.
Mandatory Skills & Experience
5+ years SDET/Automation experience
Java/Python scripting for test frameworks (Selenium, REST Assured, Playwright)
Azure DevOps YAML pipelines (CI/CD end-to-end)
AI/LLM testing (prompt engineering, model validation, RAG testing)
Cucumber BDD (Gherkin feature files + step definitions)
Git (branching, PRs, GitFlow)
ServiceNow/Jira defect tracking
Manual + Automation testing (web/mobile/API/backend)
Technical Stack
Programming: Java, Python, JavaScript
CI/CD: Azure DevOps, YAML Pipelines
Testing: Selenium, Playwright, REST Assured, Postman
BDD: Cucumber (Gherkin), JBehave
AI/ML: Prompt validation, LLM APIs (OpenAI, LangChain)
Version Control: Git, GitHub/GitLabDefect
Tracking: ServiceNow, Jira, Azure Boards
Preferred Qualifications
- Exposure to AI testing frameworks (LangSmith, Promptfoo)
- Experience with containerization (Docker) and Kubernetes
- Knowledge of performance testing for AI workloads
- AWS/GCP cloud testing experience
- ISTQB or relevant QA certifications
What We Offer
- Work on next-gen AI/LLM products with global impact
- Modern tech stack with Azure-native DevOps
- Flexible hybrid/remote work model
- Continuous learning opportunities in AI testing
Skills:- yaml, Azure devops, Large Language Models (LLM) tuning, Java, Python, CI/CD, Cucumber and Artificial Intelligence (AI)
