Specialist - Quality Engineering

Job Title GenAI Tester

Role Overview

We are seeking a GenAI Tester who can design build and validate enterprisegrade RetrievalAugmented Generation RAG Agentic AI and Generative AI applications on Azure This hybrid role combines handson development with rigorous testing and evaluation to ensure solutions are reliable grounded and production ready The ideal candidate will have strong technical expertise in Azure AI services Python frameworks and automated testing methodologies

Experience Requirements

7 years of overall experience in core IT and software development

3 years of handson experience with Azure Cloud with AI technologies

2 years of experience in Generative AI RAG LLMs Azure OpenAI LangChain and related technologies in building AI Agents

Key Responsibilities

Build RAG bots Agentic AI bots and GenAI applications using

oAzure OpenAI Azure AI Search Python Flask Azure Functions HTTP Queue Blob Timer and APIM with secure configuration via environment variables

Implement scalable data layers using Cosmos DB and Azure Storage Blob Queue Table optimize chunking embeddings and hybrid vector search pipelines

Lead prompt engineering and tuning across system prompts retrieval prompts and tooluse prompts design guardrails to minimize hallucinations and improve grounding

Establish automated testing with pytest and RAG evaluation using Ragas faithfulness precision recall answer correctness

Ensure observability with Application Insights advanced logging and endtoend traceability

Orchestrate workflows using LangChain and optionally LangGraph for stateful agent flows

Deliver production SLAs by tuning rate limits scaling AI Search tiers and optimizing costperformance

MustHave Skills

Strong proficiency in Python Flask FastAPI Azure Functions APIM Azure AI Search Azure OpenAI Cosmos DB and Azure Storage

Proven experience with prompt engineering and tuning techniques

Expertise in test automation using pytest and evaluation with Ragas knowledge of grounding techniques hybrid search semantic ranking and context filtering

NicetoHave Skills

Experience with LangGraph for agentic flows

Familiarity with CICD pipelines Azure DevOps GitHub Actions

Exposure to structured output validation and finetuning

Preferred Certifications

Microsoft Certified Azure AI Engineer AI102