Fordel StudiosModel Evaluation Beyond Benchmarks: A Production Framework
Benchmark scores tell you how a model performs on someone else's test suite. Production evaluation tells you how it performs on your users' actual tasks. The gap between these two is where most AI deployments fail.
A model that scores 92% on MMLU and tops the Chatbot Arena leaderboard can still fail catastrophically in your production environment. Benchmarks measure general capability across standardized tasks. Your production system has specific requirements: domain vocabulary, output format constraints, latency budgets, and failure modes that no public benchmark captures.
The organizations deploying AI successfully have shifted from "which model has the best benchmarks?" to "which model performs best on our evaluation suite?" Building that evaluation suite — and the infrastructure to run it continuously — is the single highest-ROI investment in an AI engineering practice.
Building a Production Evaluation Suite
The Five-Layer Evaluation Framework
Individual prompt-response pairs with expected outputs. Test specific capabilities: "Does the model correctly extract dates from this contract clause?" Build a library of 200-500 test cases covering your critical use cases.
End-to-end tests through your full pipeline — RAG retrieval, prompt construction, model inference, output parsing. Catches failures that unit evaluations miss: context window overflow, retrieval of irrelevant documents, output format violations.
Deliberately try to break the model. Prompt injection, out-of-scope queries, contradictory context, edge cases in your domain. These evaluations reveal failure modes before your users find them.
Domain experts rate model outputs on task-specific criteria. This is expensive but irreplaceable for subjective quality dimensions like tone, clinical accuracy, or legal precision. Use a structured rubric, not vibes.
Continuous evaluation on live traffic. Track output quality metrics, user feedback signals, latency distribution, and cost per query. Set alerts for regression beyond acceptable thresholds.
Evaluation Metrics That Matter
| Metric | What It Measures | When to Use | Limitations |
|---|---|---|---|
| Exact match | Output matches expected text | Structured extraction tasks | Too strict for open-ended generation |
| ROUGE / BLEU | N-gram overlap with reference | Summarization, translation | Does not capture semantic correctness |
| BERTScore | Semantic similarity to reference | Open-ended generation | Requires embedding model, slower |
| LLM-as-judge | Another model rates quality | Subjective quality evaluation | Judge model has its own biases |
| Task completion rate | Did the output achieve the goal? | Agent and tool-use evaluation | Requires defining "completion" clearly |
| User feedback rate | Positive/negative user signals | Production quality tracking | Selection bias, sparse signal |
The Model Selection Pipeline
When a new model releases — and they release weekly now — you need a systematic process for evaluating whether it improves your production system. The evaluation suite you built makes this possible: run the candidate model through your suite, compare results against the current production model, and make a data-driven decision.
The key discipline is resisting the urge to deploy a new model based on benchmark improvements alone. A model that scores 5 points higher on a public benchmark but scores 2 points lower on your domain-specific evaluation suite is a regression, not an improvement, for your use case.
- Version-controlled evaluation datasets with clear ownership and update cadence
- Automated evaluation pipeline that runs on every model candidate
- Comparison dashboards showing current vs candidate model across all metrics
- Cost tracking per evaluation run — some evaluations use expensive models as judges
- Regression alerting when production metrics drop below thresholds
- A/B testing infrastructure for gradual production rollouts of new models
“The best AI teams do not chase the latest model release. They have an evaluation suite that tells them, in hours not weeks, whether a new model actually improves their specific use case.”
The Production Evaluation Pipeline: Offline → Shadow → Online
Benchmark scores tell you how a model performs on a curated dataset that was built before your use case existed. Production evaluation tells you how the model performs on your actual users' actual queries. The two numbers are often uncorrelated. A model that scores 92% on MMLU may produce hallucinated citations 30% of the time on your specific legal research queries.
The production evaluation pipeline has three stages, each serving a different purpose and running at a different point in the deployment lifecycle.
Run the candidate model against a versioned evaluation dataset drawn from production traffic (sampled and labelled). Compute your custom metrics: factual accuracy, citation fidelity, instruction following rate, format compliance. This is your pre-deployment gate — the model must pass defined thresholds before proceeding. Offline eval should complete in under 30 minutes to not block your deployment pipeline.
Route a small percentage of production traffic (typically 1-5%) to the candidate model without serving its responses to users. Log both the incumbent and candidate responses, then run automated evaluation on the shadow outputs. This catches distribution shift — cases where your production traffic pattern differs from your eval dataset. Shadow eval runs for 24-72 hours before a full rollout decision.
Continuously sample live traffic and evaluate a subset of responses. Track metric drift: if factual accuracy degrades over a 7-day rolling window, trigger an alert. Online eval is your early warning system for model drift, prompt injection attacks, and distribution shift from changing user behaviour.
The transition from offline to online evaluation is where most teams under-invest. Building an offline eval suite is a one-time engineering project. Building the shadow and online eval infrastructure — traffic routing, response logging, async evaluation jobs, metric dashboards — is a sustained investment. Teams that skip it discover model quality regressions through user complaints rather than metrics. For context on how agentic RAG systems require extended evaluation across multi-hop retrieval chains, see the production retrieval guide.
LLM-as-Judge: When It Works and When It Hallucinates Quality
LLM-as-judge — using a strong model (GPT-4o or Claude 3.5 Sonnet) to evaluate the outputs of a weaker or candidate model — has become the default evaluation approach for subjective quality dimensions. It scales to thousands of samples per day without human labelling costs. But it has failure modes that are systematically underreported.
Mitigation strategies for each bias: position bias — randomise response order and average scores from both orderings. Verbosity bias — include explicit evaluation instructions that penalise padding and reward conciseness. Self-enhancement bias — use a different model family as judge than the model under evaluation, or use multiple judges and require agreement.
LLM-as-judge works well for: coherence scoring, tone evaluation, instruction-following assessment, comparative preference when both responses are plausible. It works poorly for: factual accuracy verification (the judge may hallucinate its own ground truth), citation correctness checking (requires retrieval of the cited source), and mathematical correctness.
For factual accuracy and citation fidelity, use deterministic evaluation: compare extracted claims against a verified knowledge base, or parse citation URLs and check that the retrieved document actually contains the claimed information. These require more engineering than an LLM judge but produce reliable metrics.
Evaluation Tooling Comparison: Braintrust vs Maxim vs Langfuse
Three platforms have emerged as leading evaluation frameworks for production LLM systems. The right choice depends on your team's evaluation maturity, existing observability stack, and whether you need prompt management alongside evaluation. The structured outputs guide is relevant here — evaluation frameworks that can parse and score structured outputs catch a category of failure that free-text evaluation misses.
| Feature | Braintrust | Maxim AI | Langfuse |
|---|---|---|---|
| Primary focus | Eval + prompt management | Eval + guardrails | Observability + eval |
| Deployment | Cloud (self-host available) | Cloud | Cloud + open-source self-host |
| LLM-as-judge built-in | Yes — multiple judge configs | Yes — with guardrail integration | Yes — via scoring functions |
| Dataset versioning | Yes — first-class feature | Yes | Yes — via datasets |
| A/B testing for prompts | Yes — experiment tracking | Partial | Yes — via experiments |
| Custom metrics SDK | TypeScript + Python | Python primary | TypeScript + Python |
| Tracing integration | OpenAI + LangChain + custom | OpenAI + major frameworks | OpenTelemetry + custom |
| Open source | No | No | Yes (MIT license) |
| Pricing (as of 2025) | $0 for 1K events/day, $50+/mo | Custom pricing | $0 self-hosted, $49+/mo cloud |
Braintrust is the strongest choice for teams that need eval tightly integrated with prompt versioning — every prompt change is tracked as an experiment with before/after metrics. Langfuse is the right choice for teams that prioritise open-source, self-hosted infrastructure and need deep observability alongside eval. Maxim is strongest when guardrails and content safety checks are primary requirements alongside quality evaluation.
Custom Eval Metrics That Matter
Generic metrics from frameworks (BLEU, ROUGE, BERTScore) are largely useless for production LLM evaluation. They correlate poorly with human judgement on open-ended generation tasks. The metrics that matter are domain-specific and require custom implementation.
- Factual accuracy: Extract claims from response, verify each against source documents. Score = verified claims / total claims.
- Citation fidelity: Parse cited sources, fetch document, check that cited quote appears verbatim or paraphrase in source. Score = valid citations / total citations.
- Instruction following: Parse response for required elements (bullet points, word count, section headers). Score = satisfied requirements / total requirements.
- Format compliance: Run JSON schema validation, regex checks, or AST parsing for code outputs. Binary pass/fail per response.
- Groundedness: Use an NLI model (e.g., Microsoft DeBERTa-v3 fine-tuned on MNLI) to check that each claim in the response is entailed by the retrieved context.
- Refusal rate: Track how often the model correctly refuses out-of-scope or harmful queries. Both over-refusal and under-refusal are failure modes.
A/B Testing for LLM Outputs
A/B testing LLM outputs is harder than A/B testing click-through rates because the outcome variable (response quality) is not directly observable from user behaviour. Users do not reliably signal response quality through actions like clicking or dwelling — a bad response that sounds plausible may receive no negative signal.
Two approaches that work in practice: explicit preference collection (show users two responses, ask which is better — useful for low-volume internal tools) and implicit behavioural signals (track follow-up clarification questions as a proxy for response inadequacy, track task completion rates for task-oriented assistants). Neither is perfect. The follow-up question signal is noisy — some clarifications are exploratory, not corrective. But at scale, the signal is reliable enough to detect meaningful quality regressions.
Statistical significance in LLM A/B tests requires larger sample sizes than typical web experiments because the outcome variance is high. A minimum of 500-1,000 responses per variant is needed for 80% power to detect a 5% improvement in quality metrics. Plan your rollout accordingly — a 1% canary on a low-traffic product may take weeks to reach significance. For higher-throughput evaluation, the RAG vs fine-tuning decision framework includes guidance on evaluation sample sizes for comparing knowledge retrieval approaches.
Building and Versioning Evaluation Datasets
An evaluation dataset that was good 3 months ago may be misleading today if your user population or use case has evolved. Dataset versioning and contamination prevention are engineering concerns that are consistently underestimated.
Sample 200-500 production queries weekly, covering the distribution of query types. Do not cherry-pick — stratified random sampling across query categories ensures the dataset represents real usage, not your assumptions about usage.
For the first 500 examples, use human reviewers to establish ground truth. Use these human-labelled examples to calibrate your LLM judge — compute judge-human agreement and adjust judge prompts until agreement exceeds 85%. Only then use the judge to label at scale.
Hash each eval example by (query, expected_output) and store in a versioned manifest. When you add or remove examples, generate a new dataset version. This allows you to compare model performance on the same dataset version across time — apples-to-apples comparison.
Never train on your eval dataset, even indirectly. If you use your eval examples to craft few-shot prompts, those examples are contaminated for evaluation purposes. Maintain a strict train/eval split at the data management layer, not just as a convention.
When OpenAI releases GPT-4o-2025-11 and you consider migrating from GPT-4o-2024-11, run both on your full versioned eval dataset before switching. Track per-category score changes, not just aggregate scores — a new model version may improve on average while regressing on your highest-priority query type.
Regression Testing for Model Upgrades
When your model provider releases a new version (GPT-4 → GPT-4.1, Claude 3.5 → Claude 4), you need to know whether the upgrade improves or degrades your specific use case. Public benchmarks tell you about general capability; they tell you nothing about your domain-specific performance. The regression testing approach: maintain a curated test set of 200-500 representative queries with known-good outputs, run the new model against this test set, and compare results across your custom metrics.
The test set must include: common queries (the 80% case), edge cases (unusual inputs, ambiguous queries, adversarial prompts), and regression cases (queries that previously caused failures and were fixed). Version your test set alongside your application code. When you add a new failure case, add it to the test set immediately — this is the LLM equivalent of adding a test case when fixing a bug. Over time, your test set becomes a comprehensive specification of your system's expected behaviour, and running it against a new model version gives you a clear upgrade/downgrade signal within minutes rather than weeks of production monitoring.
Building an Evaluation Culture
The most common failure mode in LLM evaluation is not a tooling problem — it is a cultural problem. Teams that treat evaluation as a one-time validation step ("we tested it, it works") rather than an ongoing practice ("every change is evaluated against our test suite") will ship regressions. The cultural fix: make evaluation a blocking step in your deployment pipeline. No prompt change, no model upgrade, no system prompt modification ships to production without passing the evaluation suite. This is the same discipline as requiring tests to pass before merge — applied to AI systems where "tests" are evaluation metrics and "pass" is defined by your quality thresholds.
The evaluation suite should be owned by the team, versioned in git alongside the application code, and expanded every time a production issue is discovered. When a user reports a bad response, the first action is to add that query to the evaluation suite with the expected correct response. Over time, this builds a comprehensive specification of your system's expected behaviour. For teams building this discipline alongside broader production AI observability, the evaluation suite and the monitoring system are complementary — evaluation catches issues before deployment, monitoring catches issues in production.
Need this kind of thinking applied to your product?
We build AI agents, full-stack platforms, and engineering systems. Same depth, applied to your problem.
Enjoyed this? Get the weekly digest.
Research highlights and AI news, delivered every Thursday. No spam.
