Job Title: QA Tester – AI Observability & Monitoring

Duration: 6 Months Contract with a possibility for an extension

Location: Offshore - India / 100% Remote

Rate: INR 1200 per hour

Experience Required: 8+ years in QA and at least 2 years in AI ML related projects. Should have Observability and Monitoring experience.

Role Overview

We are seeking a QA Tester specializing in AI Observability and Monitoring to support the validation and continuous monitoring of AI/ML solutions in a regulated enterprise environment.

This role will focus on ensuring that AI systems are traceable, explainable, and continuously performing as expected by validating observability frameworks, monitoring pipelines, and model performance metrics. The candidate will work closely with AI engineers, data scientists, and validation teams to ensure AI solutions meet quality, compliance, and audit readiness standards.

Key Responsibilities

AI Observability Validation
Validate observability instrumentation across AI systems, including:

Input/output tracing
Telemetry data (latency, token usage, cost, etc.)

Ensure all observability signals are captured, linked, and auditable
Verify traceability and explainability of model behaviour across workflows
AI Model Monitoring & Drift Testing
Validate monitoring frameworks for:

Model performance (accuracy, confidence, consistency)
Drift detection and threshold-based alerts

Test alerting mechanisms and escalation workflows for:

Performance degradation
Anomalous outputs

Support continuous monitoring validation in production environments
AI Behavior & Functional Testing
Design and execute test scenarios covering:

Edge cases and ambiguous inputs
Prompt variations and response consistency
Bias and fairness validation

Validate model outputs against expected results and SME benchmarks
Perform comparative validation (AI vs. baseline/manual outputs)
Observability Tools & Integration Testing
Test integration between AI applications and observability tools (e.g., Langfuse or similar platforms)
Validate data pipelines feeding observability dashboards and KPI metrics
Ensure end-to-end visibility across AI lifecycle (development QA production)
Non-Functional & System Quality Testing
Validate non-functional requirements including:

Performance and latency
Reliability and resilience
Logging and auditability

Ensure monitoring coverage aligns with enterprise quality and governance standards
Audit, Compliance & Documentation
Maintain audit-ready documentation for:

Test cases, execution results, and validation evidence

Ensure alignment with:

SDLC validation processes
AI governance and compliance requirements

Support inspection readiness and audit responses as needed

Required Qualifications

Bachelor's degree in Computer Science, Data Science, Engineering, or related field
3–7 years of experience in QA / Testing / Validation
Experience working with AI/ML systems or data-driven applications
Exposure to monitoring systems, logging frameworks, or observability platforms

Technical Skills

Strong understanding of:

AI/ML concepts (LLMs, model behavior, drift, evaluation metrics)

Experience with:

API testing and backend validation
SQL / data validation techniques

Familiarity with:

Observability tools (e.g., Langfuse, logging/monitoring platforms)
Test management tools (e.g., QTest, ALM tools)

QA & Validation Skills

Experience designing:

Functional and non-functional test scenarios
Edge case and negative testing scenarios

Understanding of:

Test automation concepts (Python preferred)
End-to-end validation lifecycle

Preferred Qualifications

Experience in GenAI / LLM testing
Knowledge of:

Prompt engineering and evaluation methods

Familiarity with:

GxP / regulated industry environments
AI governance, explainability, and Responsible AI frameworks

Tips: Provide a summary of the role, what success in the position looks like, and how this role fits into the organization overall.

Responsibilities

[Be specific when describing each of the responsibilities. Use gender-neutral, inclusive language.]

Example: Determine and develop user requirements for systems in production, to ensure maximum usability

Qualifications

[Some qualifications you may want to include are Skills, Education, Experience, or Certifications.]

Example: Excellent verbal and written communication skills

Skills: validation,logging,skills,data,ml,testing,metrics,platforms,drift,compliance

QA Tester AI Observability & Monitoring