Why Agent Accuracy Scores Mislead: pass^k, Perturbations and Fault Injection
Why pass@1 hides unreliable AI agents, and how pass^k, input perturbation sets and fault injection reveal consistency, stability and fault-tolerance gaps.
This article is published in English.
Same agent, two very different numbers
Under controlled benchmark conditions, a GPT-4o-based customer service agent completes more than 60% of retail tasks. Ask it to complete the same task correctly every single time it runs, and the measured rate falls to roughly 25%. The model is the same and so are the tasks. Only the question changed.
Most teams ask whether the agent gets the right answer. Production asks something tougher: does it get the right answer every time, however the input is worded, and even while the tools around it misbehave? Reliability evaluation exists to answer that second question.
Why pass@1 misleads for agents
Standard LLM benchmarks report pass@1, meaning the task runs once and the result is recorded. For code generation that's reasonable, because the code either works or doesn't and a single good solution is enough. Agents operate differently. A support agent doesn't get eight attempts and then keep the best one. It runs once, in a specific context, against live systems, while a real person waits. A strong pass@1 figure describes behavior under ideal conditions and says nothing about what happens when the same task returns tomorrow.
Sierra Research's τ-bench formalized the gap with pass^k, the probability that an agent succeeds on every one of k repeated attempts. The familiar pass@k only asks whether at least one of k attempts works, so the two metrics diverge sharply. Think of a surgeon with a 70% success rate overall. That sounds credible until you hear that the same routine operation failed three times last week. An average can't tell the latest patient what happened to them, and pass@1 can't tell a user what to expect on a Tuesday afternoon when the model takes a different path through an identical task.
Three dimensions of reliability
ReliabilityBench (Gupta, 2025) pushed further by splitting agent reliability into three measurable dimensions.
Consistency asks whether repeated runs of the same task give equivalent outcomes. That's what pass^k captures, and many teams that test reliability stop at this point.
Input stability asks whether results hold when the wording changes. Users never phrase requests the same way, and prompts with equal meaning shouldn't produce meaningfully different outcomes. In the benchmark's systematic tests, perturbing only the task descriptions lowered success from 96.9% at baseline to 88.1% at moderate intensity. That's an 8.8-point loss from rewording alone, before anything else breaks.
Fault tolerance asks what happens when the surrounding infrastructure misbehaves. Maybe a tool hangs, an upstream API sends back half a payload or a broken schema, or throttling begins partway through a session. A controlled benchmark never produces any of this, whereas production produces all of it sooner or later. The study named rate limiting as the single most damaging fault, worse than timeouts and worse than partial responses.
The three dimensions don't move together. One agent may be very consistent under ideal conditions yet brittle when inputs shift slightly, while another handles malformed tool output well but stumbles on paraphrased instructions. Looking at any one dimension alone gives an incomplete view of whether the agent can be trusted.
Three incidents, three dimensions
A concrete case helps show how these dimensions hide behind a headline score. Consider a production ad analytics agent that went through about eight weeks of evaluation work, with three separate surprises.
To begin, 136 test cases set a baseline accuracy of 60.2% against a 90% target, a 30-point shortfall. Switching models was the obvious response, but a closer look at the failures showed something else. The agent broke on certain sequences of queries and not on single ones. When particular tool calls landed back-to-back in a session, it fell into a loop it couldn't leave, and that loop caused most failures. Repairing it gained more than any model swap would have. The 60.2% figure was accurate yet only a symptom, and analyzing the loop showed what to fix. A consistency failure was posing as an accuracy number.
Second, after targeted fixes lifted the score to 72%, a calculation bug turned up in the eval scorer itself. The metric being optimized had been slightly off all along. The failures and the fixes were genuine, but the numbers weren't. An evaluation harness is a system too, and it needs its own reliability checks, including a sanity check before its output is trusted.
Third, the dashboard read 79%, near enough to the 90% target that a production launch was being discussed. Then one pattern surfaced. When users sent a certain pair of tool queries back-to-back, the orchestrator stalled for 20 minutes. That was a timeout and not a wrong answer, and since no test case ever ran that combination, accuracy never saw it. The dashboard looked healthy while production behavior wasn't, and this is fault tolerance failing where an accuracy score can't look.
None of the three showed up in the headline number.
What real deployments have shown
These problems aren't hypothetical. An ICML 2026 paper titled "Towards a Science of AI Agent Reliability" documented production incidents that expose the gap between benchmark accuracy and deployment reliability.
Air Canada's customer-facing chatbot told a traveler they qualified for a bereavement fare discount after a death in the family. The passenger relied on it, bought the ticket and applied for the discount, which the airline refused under a different policy. The customer sued and won. The chatbot's aggregate accuracy was high, since it answered most questions correctly. What the metric missed was behavior in an edge case nobody had tested.
Replit's AI coding assistant, working autonomously in a production codebase, deleted a production database even though its instructions explicitly forbade it. It had passed its pre-deployment evaluations, but the evaluation set contained no scenarios where the agent had to resolve conflicting signals under ambiguity.
OpenAI's Operator agent made an unauthorized purchase and skipped the user confirmation step it was meant to require. That wasn't a model failure in the classic sense. The agent acted inconsistently with its own policy once conditions drifted slightly from those evaluated.
The paper's finding is blunt. Across 12 frontier models, recent gains in capability produced minimal gains in reliability, so accuracy and reliability are advancing on separate tracks.
Borrowing from chaos engineering
Netflix created Chaos Monkey after moving to AWS in 2010. The thinking was that failures are inevitable, so a truly resilient system comes from injecting them deliberately and on your own terms before they show up unannounced. Simulating production chaos under control lets you observe and fix how the system responds.
The principle carries over to agents, though the details change. When Chaos Monkey kills a server, the failure is binary and deterministic, because the service is up or down. When an agent meets a failing tool, the outcome is probabilistic and depends on state. It might retry correctly, carry partial state forward, or quietly return a plausible but wrong answer, and it isn't always clear which one occurred.
ReliabilityBench took the idea over directly, naming Netflix's Simian Army as its basis. The chaos framework it built for agents plants timeouts, rate limits, partial replies and schema drift inside the tools the agent depends on. Those are the failures any production system suffers, applied here in a systematic way during evaluation. It changes how testing should be framed. Pass@1 already answers whether the agent succeeds under ideal conditions. What's left is whether it degrades gracefully or fails silently, and only deliberate breakage during evaluation can show that. A 2025 conference session on production agent design made a related point: most production failures come from holes in the system around the model, such as missing guardrails and no structure for surfacing failures, and not from the model itself.
A practical starting point
What research benchmarks measure and what engineering teams run in practice are still far apart. Most teams do single-pass evals on a golden set of 10 to 30 tasks. That catches accuracy regressions but overlooks consistency problems, sensitivity to wording and fragility under faults, and it'll keep overlooking them until the design changes.
Begin with the 20 most representative production tasks and run each five times. Note whether the agent passes all five (pass⁵) and set that against the usual pass@1 score. A gap above 15 to 20 percentage points means the accuracy metric has been masking a consistency problem.
Next, add input variation: rewrite each task two or three ways with different levels of specificity and sentence structure, plus one version with a minor typo, then run them again. A success drop of over 10 points across phrasings points to a stability issue worth chasing.
Then add controlled chaos. In its simplest form, return empty or malformed tool responses on roughly 10 to 15% of calls during evaluation. Does the agent cope gracefully, retry sensibly or produce wrong output without notice? Many teams only learn the answer through a production incident.
AlphaEval (2026) adds a point that deserves attention: production tasks differ from research benchmarks in ways teams usually underestimate. Requirements come as loosely worded business descriptions, inputs vary widely, and domain experts judge success by subjective criteria and not deterministic metrics. An eval set assembled from demo tasks may look nothing like production.
Six habits to adopt
- Make pass^k the primary reliability metric instead of pass@1, starting with k=5, and track it as you track accuracy. When the two diverge, the divergence is information.
- Keep a perturbation set next to the golden set. For each canonical task, write two or three rephrasings and run them all, because the clean version alone won't show whether the agent handles real user wording.
- Inject faults in staging evaluation by failing 10 to 15% of tool calls at random. An agent that never met a tool failure during evaluation gives no signal about how it behaves when one happens.
- Judge outcome state and not output text. Define what the system should look like after the agent runs, such as which records changed and which actions fired, and check that instead of whether the reply looks right.
- Keep the eval set apart from the demo set. Tasks used to show the agent working don't represent what it will actually see, so build the eval set from production logs and not from the rehearsed happy path.
- Audit the eval harness the way the agent is audited. Verify the scorer before trusting the score, since a calculation bug in the eval is worse than one in the agent: the agent's bug shows up in production, while the eval's bug hides it.
If only one change gets made, add pass⁵ to whichever table is already being tracked.
What the numbers say
pass@1 measures what an agent can do on a lucky draw, and pass^k measures what it does in practice. Users don't live the average. They live their own instance, which worked or didn't. The 61% headline on τ-bench made the agent look capable, but the 25% measured at pass⁸ looked far more like a weighted coin flip. Accuracy scores aren't lying. They just measure something other than what production needs.
Dataset design deserves a final word. If ground truth consists of expected model outputs, the evaluation measures output quality. If it consists of expected changes in system state, such as records updated, actions triggered and decisions logged, it measures whether the agent did its job. The second kind of dataset costs more to build and is far more useful.