Why Agents Pass Every Offline Eval and Still Fail in Production
Six drift modes quietly age an eval set. A four-axis trace score and a weekly cluster-and-promote loop keep CI honest against what live traffic actually does.
This article is published in English.
Sources: Why Your Agent Passes Every Eval and Still Fails in Production
Picture a customer support agent that clears its CI suite: 47 scenarios, four rubrics, all green, so it ships. A few weeks into production it quotes refund amounts that are off by a factor of ten. It contradicts itself from one turn to the next. In one trace it shows a user the order of a different customer. Yet when the failing conversations get pulled up, each one still scores fine on the per-turn faithfulness rubric. Every conversation failed the user anyway. The eval said ship, and by then the trust was already gone.
The rubric isn't the culprit
The first instinct is to blame the rubric. A better reading is that it was fine in March and getting stale by April, since production never stopped moving. An eval set is a snapshot and production is a river. Evals don't ship, traces do. Each release starts aging the set on the day it lands, and unless new failure modes flow back into the offline set, the distance between passing offline and failing live isn't an accident. It follows from the arithmetic.
Enough incidents like this lead to one takeaway: evaluate production traces, not curated test cases. Offline evals still make a worthwhile regression gate, but they can never be the only thing standing between a change and release, since what they measure disappeared at the moment of freezing. Six kinds of drift wear down any eval set, each ticking at its own pace:
- Dataset drift: curated cases all pass, while production brings user intents the set never contained.
- Tool-API drift: mocks return the same payload shape, while the vendor has changed schemas or error codes.
- Prompt drift: the rubric was frozen for prompt v3, while production runs v17 and the rubric still grades v3.
- Retrieval-corpus drift: the index was frozen at build time, while production's index has doubled and the same query now lands on new chunks.
- User-distribution drift: test inputs were written by hand, while real traffic looks nothing like them.
- Agent-step compounding: each step succeeds 95 percent of the time, yet eight steps in a row multiply out to about 66 percent.
Freezing an eval set freezes a hypothesis about what users will do, how tools will respond, which prompt will be live and what the retriever will surface. Months later each assumption has drifted a different distance, and a rubric that awarded a clean pass is judging a world the agent no longer inhabits.
Nothing is actually wrong with the rubric. It has simply gone stale, like a unit test whose target function was rewritten underneath it: green, and protecting nothing. For software the cure is a fresh test of the new function. For agents the cure is scoring the live trace. A single rubric definition then serves two jobs, gating CI against a versioned dataset and scoring real traces as a span-level attribute. CI sets the minimum bar, the live score tracks the flowing reality, and each week the dataset absorbs the traces the rubric has just flagged.
How each drift shows up and what fixes it
Dataset drift. At launch someone wrote the set, and since then users have come up with intents nobody pictured. Scores never budge because the data never budged. Watch for offline results that stay flat for months while production complaints grow more varied and engineers fail to reproduce most reports against the test set. Each week, sample failed traces, bucket them by segment, intent and score, and tag the toughest 5 to 10 percent with a version before adding them to the set. Every promoted trace then guards against one specific regression.
Tool-API drift. CI uses a mocked tool, and one Tuesday the live endpoint changes its schema, error shape or rate limiting. The agent retries, the loop times out and the agent makes up a convincing reply, but CI stays green because the mock keeps returning yesterday's payload. Rising latency and retry counts, a response rubric that still passes and a cost per success drifting upward are the clues. Give tool-call success its own rubric on live spans, so a broken call sits in the trace next to the broken reply. Mocks catch regressions you caused, and live scoring catches the ones a vendor caused.
Prompt drift. Prompt v17 went live on a Friday, while the rubric still reflects what mattered in v3 back in February. The agent is being measured against an obsolete contract. A good clue is a senior engineer who reads ten traces, disagrees with the judge on six and can't articulate why. Ship the rubric revision in the same change as the prompt revision, treat it as a contract test, and watch judge-human agreement on a small calibration set so a rubric falling behind gets noticed.
Retrieval-corpus drift. The March retriever evaluation covered 12,000 documents and one chunk size. By May the index has 38,000, a re-embed changed how chunks rank and an identical query lands elsewhere. The generator dutifully builds on whatever arrives, so groundedness still shows 0.94 while the source material is wrong. Users calling the bot less helpful than last quarter, with generation rubrics unchanged, is the symptom. Split evaluation by layer. Retrieval rubrics such as context relevance and chunk attribution flag index changes before generation scores mask them. Falling relevance with steady groundedness points at the retriever, and the opposite points at the generator. One bisect replaces three days of guessing.
User-distribution drift. Whoever built the set either wrote examples by hand or took them from the first month of traffic. Real users later bring slang, mixed languages, long prompts, screenshots and strings of follow-ups the set never saw, so a judge tuned on curated data rates live conversations far more kindly than a human would. If spot-checked production traces differ from the judge by 15 points or more, engineers stop trusting the rubric and fall back on reading raw traces. Calibrate the judge on production samples instead of the dataset, and measure judge-human divergence as a metric in its own right.
Step compounding. Suppose each step scores 95 percent and a session involves eight tool calls. Raise 0.95 to the eighth power and the result is 0.66, so roughly two out of three sessions go structurally wrong although every step looked healthy. The rubric never did the multiplication. Strong per-turn numbers next to weak completion, outcome and satisfaction figures are the sign, along with tickets about the bot asking the same thing repeatedly or saying yes and then no. Score whole traces: completeness, role adherence and knowledge retention at conversation level, and an assessment of how the plan was carried out across the span tree. Metrics spanning several turns carry more noise per dollar, yet they follow actual user experience far more closely.
Why a bigger dataset doesn't help
What the six share is timing. Each shows up after the eval set froze, and a fixed dataset can't hold a hypothesis nobody had yet. A 10,000-example set from March knows nothing about May's tool-schema change, June's prompt revision, July's re-embed or August's users who phrase requests differently. A bigger set doesn't fix a snapshot, and only sampling production does, so evaluation belongs where the agent runs. Curated data is the seed for regressions. Live traces are the working material, where failures group up and get scored as they happen. Each named group then turns into tomorrow's dataset entries, and the cycle starts again.
A four-axis trace score
Per-turn faithfulness on the final reply is too coarse to diagnose a drifting agent. A more useful trace score has four axes, each rated one to five:
- Factual grounding: did the agent stay tied to the retrieved context or make things up. This exposes retrieval and dataset drift at the response level.
- Privacy and safety: whether it leaked PII, crossed a tenant boundary or went along with a jailbreak it should have refused, which surfaces permission and refusal regressions.
- Instruction adherence: whether it obeyed the current system prompt, the axis that drops when the prompt reached v17 and behavior stayed behind.
- Plan execution: whether it picked the right tool, in the right order, with no redundant calls or dead ends, which exposes step compounding and tool drift across the call graph.
Four axes cover four kinds of regression and roll up into one composite. When the composite sinks on a trace, the axes show which drift mode struck, and since the same axes run in CI and on live spans, the diagnostic vocabulary matches in both places.
Closing the loop
Narrowing the gap is a workflow and not a feature, and it has five steps:
- Cluster failing traces into named issues, since nobody can triage a flat list of 800 failures.
- Rate each cluster on the four-axis trace score and attach a short note on the fix to make today.
- Move three to ten typical traces from each cluster into the offline set, labeled with route tags and rubric labels.
- Re-gate: the next CI run scores the added entries using the production scorer's rubric, so later edits along that path can't slip back.
- Optimize the prompt against the enlarged set, where the fix must clear the rubric in CI before shipping.
For a product in active use, repeat this weekly, or more often during a turbulent launch. A static set more than a quarter old rarely resembles production, and for a fast-moving agent drift becomes visible within two or three weeks. Each incident handled this way leaves behind a regression test that nobody has to author again.
Costs to plan for
- The loop adds operational surface. Span scores, clustering and a promote step are more machinery than a folder of pytest evals. In return the regression suite keeps getting tougher, and a fresh deployment can launch with tracing and offline evals alone, adding the loop afterward.
- A judge that calibrates itself needs its own safeguard. A rubric tuned against live traffic may wander somewhere nobody intended, so retain a small human-labeled hold-out and raise an alarm if the judge diverges from it by more than humans diverge from each other.
- Scoring attached to traces costs more for each unit of noise than offline scoring does. A four-axis rubric on a 30-second trace is dearer than per-turn scoring of a short case and varies more. Sample according to failure signals, not at random, and place an inexpensive classifier ahead of the costly judge.
The short version
An eval set begins aging the moment it ships, and size doesn't cure a snapshot. Score the current trace instead of the frozen case. Use one rubric in CI and on live spans, grade each trajectory for grounding, safety, instruction adherence and plan execution, cluster the failures and send them back into the offline set weekly. Passing offline is necessary. Passing on traces is what truly protects users. The curated set is only the seed, and production is the dataset you're always trying to catch.