Senior AI Research Engineer

Company Overview

Trianz is an applied AI solutions company that accelerates customer business transformation through AI powered Transformation Services as a Software Model. With 25+ years of transforming enterprises, we've evolved to a product-led, platform-driven organization serving global enterprises across Financial Services, Insurance, Healthcare, Hi-Tech, Manufacturing, and other industries.

With global presence across 4 continents, our platform portfolio under the unified Concierto brand delivers end-to-end transformations including solutions for Migrate, Manage, Maximize, Modernize, Insights & Agentic AI, and SecOps - delivered through strategic partnerships with leading hyperscalers.

We're building the premier innovation-led organization in the digital transformation space through AI-first methodologies and data-driven excellence - RevolutionAIzing Transformations.

Role Overview

Every AI architectural decision at Trianz - which model to deploy, which hardware to run it on, whether a smaller fine-tuned model outperforms a larger API-consumed model on a specific enterprise task — rests on evidence. You produce that evidence.

As Senior AI Research Engineer, your work is not theoretical. It is the empirical foundation that justifies architectural bets with real infrastructure cost, customer-facing latency, and enterprise compliance consequences. When the team decides to deploy a 7B model on CPU rather than calling a 70B model via API, that decision rests on your benchmark data — your evaluation harness, your reproducibility standards, your domain-specific task results, and your recommendation.

This is not a role for someone who runs general benchmarks from leaderboards and presents the numbers. General benchmarks tell you how a model performs on academic tasks. Enterprise AI decisions require evaluation on the specific task types — document extraction, policy reasoning, structured output generation, multi-step planning — that actually run in production. You design the evaluation suite. You run the experiments. You produce the recommendation with the data behind it.

You will also run fine-tuning experiments to test whether a smaller, domain-adapted model can match or exceed a larger general model on specific tasks — a question with direct infrastructure cost and sovereignty implications. You will maintain the model leaderboard as new models are released and evaluate quantization strategies across the full accuracy-versus-performance tradeoff curve.

If you consume existing benchmarks without questioning their applicability to enterprise tasks, this is not your role. If you design evaluation frameworks, produce reproducible results, and drive architectural decisions with data — you are exactly who we are looking for.

Key Responsibilities

LLM Evaluation Framework Design

• Design and implement the LLM evaluation framework: benchmark suite, evaluation harness, and reproducibility standards
• Run systematic evaluations across 5B, 10B, and 30B parameter models on enterprise-specific task types
• Benchmark open-source models (Llama, Mistral, Qwen, Phi) vs closed models (GPT-4, Claude, Gemini) on accuracy, latency, and cost
• Evaluate model performance on CPU-only inference vs GPU inference to validate hardware routing decisions
• Design domain-specific evaluation tasks: document extraction, policy reasoning, structured output generation, multi-step planning
• Maintain reproducibility standards across all evaluation runs: seed control, dataset versioning, harness version tracking

Model Selection & Architectural Evidence

• Produce model selection recommendations with supporting data for every AI architectural decision
• Design the evidence pack that justifies open-source vs closed model decisions for specific enterprise use cases
• Evaluate cost-per-task across model sizes and inference modalities: API vs self-hosted vs CPU vs GPU
• Maintain the model leaderboard: continuously updated benchmarks as new models are released
• Produce rigorous, reproducible evaluation reports that architects and engineering teams act on

Fine-Tuning Research

• Design and run fine-tuning experiments for smaller models on domain-specific enterprise tasks
• Evaluate whether a fine-tuned 7B model matches or exceeds a 70B general model on specific task types
• Apply parameter-efficient fine-tuning methods: LoRA, QLoRA, DoRA, PEFT
• Design training data curation pipelines for domain-specific fine-tuning
• Evaluate fine-tuned model quality: accuracy, hallucination rate, instruction following, output format fidelity

Quantization & Inference Research

• Evaluate quantization strategies (GPTQ, AWQ, GGUF) and their accuracy vs performance trade-offs
• Benchmark quantized models against full-precision baselines on enterprise task types
• Evaluate throughput, latency, and memory footprint across quantization levels
• Produce quantization selection recommendations for specific model sizes and hardware configurations
• Research emerging quantization and compression techniques for production applicability

Ideal Candidate Profile

Experience: 13+ years (Must-Have)

LLM Evaluation & Research:

• Designed and run LLM evaluation frameworks in a production or research context — not just existing benchmarks
• Hands-on with HuggingFace Transformers, PEFT, and evaluation libraries (lm-eval-harness, HELM, or equivalent)
• Evaluated models on domain-specific tasks, not just general benchmarks (MMLU, HellaSwag, etc.)
• Fine-tuned open-source LLMs for specific downstream tasks
• Strong Python — can build a complete evaluation pipeline from scratch
• 5+ years in ML/AI with at least 2 years focused on LLM evaluation or fine-tuning

Technical Depth

• Deep understanding of LLM architecture: attention mechanisms, tokenization, context window management
• Hands-on with model families: Llama, Mistral, Qwen, Phi, Gemma at different parameter scales
• Strong statistical grounding: significance testing, confidence intervals, evaluation metric selection
• Proficiency in evaluation metric design: beyond accuracy to hallucination rate, faithfulness, output format compliance
• Understanding of hardware-model interaction: memory bandwidth, compute requirements, batching effects
• Familiarity with enterprise AI task types: document QA, structured extraction, policy reasoning, agentic planning
• Python engineering quality: reproducible, versioned, well-documented evaluation pipelines

Research Rigor

• Reproducibility discipline: seed control, dataset versioning, harness version tracking
• Domain-specific evaluation design: task definition, dataset curation, metric selection
• Ablation study design: isolating variables in fine-tuning and quantization experiments
• Results communication: clear, unambiguous recommendation with supporting evidence
• Leaderboard design and maintenance: continuous benchmarking as the model landscape evolves
• Research documentation: reproducible experiment records that others can re-run

Mindset & Fit

• Evidence-first mindset: architectural decisions must rest on data, not intuition
• Skeptical of general benchmarks always asks whether the task distribution matches enterprise reality
• Rigorous and reproducible: results that cannot be reproduced are not results
• Practically grounded: research output drives real infrastructure and cost decisions
• Self-directed: designs the experiment, runs it, and produces the recommendation without handholding
• Curious about the model landscape tracks new model releases and evaluates them systematically

Good to Have

• Inference optimization experience — vLLM, TensorRT-LLM, or Triton in a production context
• CPU inference frameworks — llama.cpp, Intel OpenVINO, or AMD ROCm for CPU-optimized inference evaluation
• Published research or technical writing — on LLM evaluation methodology, fine-tuning, or model compression
• RAGAS or LLM-as-judge evaluation — automated evaluation pipelines for RAG and agentic systems
• Synthetic data generation — for fine-tuning dataset augmentation and evaluation set construction
• Multi-modal model evaluation — vision-language models for document understanding enterprise tasks
• Agentic system evaluation — tool use accuracy, planning quality, multi-step reasoning assessment
• Statistical NLP background — pre-LLM NLP: NER, classification, information extraction

This Is NOT This Role

❌ Someone who runs leaderboard benchmarks and presents the numbers without questioning task applicability

❌ An ML engineer focused on model serving and deployment without evaluation depth

❌ A data scientist who fine-tunes models without designing rigorous evaluation frameworks

❌ A researcher who produces academic results without enterprise applicability

❌ Someone who relies on existing benchmarks without designing domain-specific evaluation suites