Who Grades the Grader? Validating Agent Eval Suites Before You Trust Them

In 2026 almost every team shipping an AI agent also runs an eval suite, complete with dashboards, pass rates, CI gates and LLM judges that score trajectories against rubrics. All of it signals rigor, and it seems to mark the difference between a production system and a demo that happened to work twice.

Ask the next obvious question, though, and things get quiet. Who verified that the suite is measuring the right thing, measuring it correctly and still telling the truth half a year after it was written? Almost nobody did. The industry spent two years building instruments to grade agents and has barely begun building instruments to grade those instruments. The gap is already showing up in production, in several predictable ways.

The assumption nobody states

Behind most agent eval setups sits an assumption that goes unspoken: a pass means the agent behaved correctly and a fail means it didn't. Accept that and the suite becomes a black box people trust automatically. The dashboard is green, the PR merges and the team moves on.

Research on agent traces has begun to poke holes in that premise. One study examined 58 traces that outcome scoring had marked as successful and found that 83 percent of those runs hid procedural or safety violations that pass/fail scoring never surfaced. The agent produced the right answer and received a perfect mark, yet the route it took was quietly broken. The distance between outcome and process is precisely what an eval suite exists to catch, and precisely what most suites can't see by design.

Four failure patterns deserve separate names. Each needs its own fix, and teams usually notice only the one that has hurt them.

Judge drift. Anything semantic tends to be graded by an LLM, since a rule-based check can't tell whether "next Friday" was converted to the correct date. But the judge is a model too, and models change. Teams with long-running suites often see the entire regression dashboard fall overnight, lose a day investigating and finally discover that scores moved because the judge model got upgraded and not because the agent got worse. Pinning a judge model version and a fixed rubric hash lessens the problem without eliminating it, because hosted judge endpoints keep shifting.

The outcome fallacy. Grading only the end state erases the interesting part of a run. An agent might call the wrong tool, resend the same bad argument four times and still reach the right answer by luck or persistence, and a suite that reads only the final message will mark it as passing.

Private failure modes. Public benchmarks such as BFCL or tau-bench help compare models, but they can't see failures living inside a team's own tool registry: a null in an optional field, a timezone bug in a date argument, a currency mismatch on a money field. Such failures belong to the team's tools and never appear on a general-purpose leaderboard. If coverage is borrowed entirely from public benchmarks, the validator is blind in exactly the place that matters, the parts of the system nobody else has tested.

The cost blind spot. Least discussed, and perhaps most consequential. A harness may rate an agent as accurate on every dimension it follows while never asking whether the agent pays for itself. One well-known case involved an agent that satisfied each of twelve quality metrics yet was removed from production, because each successful resolution cost more than the human process it was meant to replace. Quality scores were accurate. No metric existed for the question that sealed its fate.

A booking agent that scored perfectly and failed

Imagine a booking agent asked to find flights for "next Friday." Out of a registry with dozens of tools it chooses the right one, so tool selection earns a spotless 1.0. Then it sends the date as a literal string without resolving the relative expression first. The API answers with a validation error, and the agent resends the same broken string four times before apologizing and quitting.

That trace yields two figures: a perfect 1.0 for tool selection and a 0 for task completion. Neither identifies which of three things went wrong: faulty argument extraction, an error response the agent never read, or retry logic that loops on unchanged input. A detailed analysis of tool-calling evaluation made the point that checking only whether the right tool was called tests intent instead of execution, and intent wasn't what failed.

That's the whole problem in miniature. The suite emitted two numbers, neither exactly wrong, and both useless for the one question that counted: what should be fixed?

Practices that help

Once the eval suite is treated as a system deserving its own quality checks, a mature setup looks different. A handful of habits distinguish teams that catch problems early from those that find out painfully:

• A meta-evaluation layer of its own. In serious deployments the work now divides into outcome evaluation, evaluation of each trajectory step, and a layer that checks evaluators against one another so that no single judge's score is taken at face value.
• Execution-based verification instead of text matching. The most durable benchmarks inspect what actually changed in the world. tau-bench reads database state, SWE-Bench executes the real tests, and tau2-bench's pass-at-k asks whether an agent succeeds consistently across repeated tries. Homegrown suites should follow suit and confirm real changes, whatever the agent reports.
• Cost next to quality. The finance case argues for tracking cost per successful outcome as a core metric alongside accuracy, worked out from the start instead of later when a budget owner starts asking.
• Tiered CI. Full end-to-end evals cost a lot because each task needs two model calls, the agent's and the judge's, and a hundred-task suite at five trials each roughly doubles the CI bill. Teams that keep this under control separate quick, free, deterministic checks that run on every commit from slow, costly, judge-based evals that fire only when a change touches the agent's surface.
• A suite that stays alive. In a recent industry survey, most respondents said their evals underdeliver because production traffic drifts away from the original test set and keeping up becomes unmanageable. A suite left untouched since launch isn't harmless. It measures a system that has stopped existing.

Four moves for a suite you already have

For a team with an eval suite that wants to validate it and not just trust it, four steps close most of the gap:

1. Take ten to twenty production traces the suite marked as passing and read each whole trajectory manually, not just the answer. How many of those passes hide a procedural flaw is a genuine measure of how much the green dashboard deserves trust.
2. Pin the judge model version and the rubric text, and log both with every score, so a silent judge upgrade appears as a labeled event instead of a mysterious regression.
3. Track cost per successful outcome from day one, even before anyone requests it.
4. Build a small regression set out of real production failures and not just synthetic cases. Public benchmarks set the floor, while a team's own incident history is what predicts the next embarrassing failure.

None of this replaces the suite. It just stops treating the suite as the single component too important to check.

Quick answers

What does eval suite validation mean for agents? It applies the same scrutiny to the evaluation system that the system applies to the agent, asking whether the judge, rubric and scoring pipeline still measure what they claim.

Why does LLM-as-judge scoring drift? The judge is a model. When its provider updates it or it's swapped for a newer version, scores can move even though the evaluated agent hasn't changed.

Can public benchmarks stand in for a custom suite? No. They help compare models on common tasks, but they miss failures rooted in a team's own tools, data and corner cases.

Is a high pass rate enough to keep an agent in production? Not by itself. An agent can do well on accuracy and trajectory quality and still get pulled if cost per successful outcome doesn't justify the value it delivers.

How often should the suite be reviewed or rebuilt? Continuously, as production traffic shifts. Usage moves away from the original test set, so cases drawn from real failures need to arrive on a regular schedule and not only at launch.

The claim behind the dashboard

An eval suite is no proof that an agent works. It's an assertion about what "working" means, written by whoever built it, graded by a judge that may drift, over a test set that starts aging the moment it's written. Taking that assertion as obviously true is how agents get shipped that clear every dashboard and still flunk the question that mattered. A manual review of ten traces, as described above, quickly shows how much the dashboard has been concealing.