Testing AI Agents by Outcome: Verifying State Instead of Trusting Replies

When an AI agent replies "Done", that sentence is a claim, not a receipt. An evaluation that only reads the final message will happily score a hallucinated success as a pass. This guide shows how to test tool-using agents against what actually changed in the world: how to define completion, which evidence to inspect, how to handle timeouts and cases where the agent should do nothing, and how to measure reliability rather than a lucky demo.

A confident reply is not evidence

Take a simple request to a scheduling assistant: move a meeting to Thursday afternoon. The assistant answers, politely and without hesitation, that the meeting is now at 3 p.m. Then you open the calendar and the event sits exactly where it was.

Variations are just as plausible: the event shifted to 3 p.m. in some other time zone, or a brand-new event appeared while the original stayed put, leaving the user with two. The example is invented, but the testing problem is very real: if you grade only the text the agent produced, all three failures look identical to success.

The useful mental shift is this: for an agent, the answer is a report about the work, and a report can be wrong. The model's surrounding system, its harness, determines whether it can act reliably at all (we cover that layer in where reliable agent behavior actually comes from). Evaluation is the step that confirms whether that system actually delivered.

Write the acceptance contract first

"Move my meeting" sounds trivial until you try to write acceptance criteria for it.

For the running example, assume the user has identified a single event, given a date and a time zone, and authorized an update. The required outcome is not merely that some meeting exists at 3 p.m. It is that the same event now starts at the requested time, keeps its original duration and attendee list, and that no other event was touched. If the request is ambiguous or the assistant lacks permission, the correct behavior is to ask, not to guess.

Those rules belong to this hypothetical product. A different product might send notifications, require explicit approval, or treat attendees differently. What matters is that the contract is written down. Without it, one reviewer praises an assistant for taking initiative while another marks the same run as an unauthorized change, and your scores become noise.

Every evaluation case should therefore specify:

• the starting state of the environment;
• the user request;
• the actions the agent is allowed to take;
• the conditions that must be true afterwards; and
• the things that must remain unchanged.

That last item is easy to forget. A good test defines both the goal and the boundary around it.

Three views of a single run

Each agent run can be examined from three angles, and each catches different failures.

• Output: the message the user saw. Is it accurate and helpful, and does it admit unfinished work instead of papering over it?
• Trajectory: the observable sequence of tool calls, tool results, approvals, retries and checks. Did the agent pick the correct event, stay within its permissions and react sensibly when a tool failed?
• Outcome: the actual state of the calendar once the attempt is over.

Anthropic's guide to agent evaluations makes the same separation between the transcript and the final state of the environment, and it cautions against grading every valid solution against one fixed sequence of steps. That balance is worth adopting. Look at how the agent behaved, but do not insist on your preferred path unless the order of operations is genuinely part of the contract.

Concretely, two agents might read the calendar through different operations and still arrive at the same correct result; both should pass. Skipping a mandatory approval, on the other hand, is a failure even if the calendar ends up looking right. The outcome decides whether the task succeeded, the boundaries decide whether it was allowed, and the trace explains the gap between the two.

Check the world through an independent channel

To verify the calendar test, read the stored event through an authorized interface that has nothing to do with the agent's own reply. The agent is not allowed to grade its own homework.

Compare the fields that actually define correctness against the expected state:

• the stable event identifier;
• start and end times, and the time zone;
• the attendee list;
• neighboring events that should have stayed the same.

Avoid matching on the event title. Two meetings can share a name, and a title match will happily confirm the wrong one.

This idea predates today's agent frameworks. τ-bench scores tool-using conversations by comparing the final database state with an annotated goal state, and WebArena measures whether tasks are functionally completed inside realistic web environments. Both are research examples of grading what happened rather than what was said.

Know what a success signal really promises

A successful API response is useful evidence, but only once you know what it guarantees. "Accepted" may mean the request was queued, not applied. A success banner in a browser may appear before the underlying record is stored.

For asynchronous systems, decide in advance how long the verifier waits and which state counts as settled. If you cannot observe the state at all, record the result as unverified. Do not quietly count it as a pass, and do not assume the opposite either. That third category becomes essential as soon as requests start timing out.

The hardest moment: right after a timeout

A timeout tells you nothing about whether the update landed. The request might have died before reaching the calendar service, or it might have succeeded and only the response got lost on the way back.

An agent that reacts by creating a fresh replacement meeting can leave the user with two events. That makes recovery behavior a first-class test case. A sound recovery sequence looks like this:

1. Read the target event again.
2. Check whether the requested change is already in place.
3. Retry only if the evidence and the tool's contract make a retry safe.
4. Where the service supports it, use an idempotency mechanism so that a repeated request is recognized as the same operation rather than a new one.

If the outcome is still unknown after that, the right move may be to stop and explain the uncertainty to the user. A clear handoff beats a manufactured success. The same pattern shows up in ordinary backend design; see idempotency keys for Node.js POST endpoints for the server side of the story.

Grade both the state and the wording. "I couldn't confirm the update" is truthful even if the service applies the change a moment later. "Nothing was changed" is an unsupported claim in exactly the same situation. Uncertainty is a legitimate result that the agent must handle, not a gap it should fill with confidence.

Match the grader to the question

Different questions deserve different judges.

• Deterministic code for exact facts: identifiers, timestamps, the set of permitted changes, the count of duplicate events.
• A model-based grader with a clear rubric for communication quality: did the assistant distinguish completed work from attempted work, and did it describe next steps without inventing details?
• Human review for ambiguous cases, and for periodically checking that automated grades still agree with what the product team expects.

The LangSmith evaluation documentation treats code rules, human review and model-based judging as distinct options. You do not need that platform to benefit from keeping them separate.

One anti-pattern deserves special mention: handing a second model the agent's confident reply and asking whether the task looks successful. The judge will be persuaded by the same confidence that fooled you. Instead, give the judge the evidence the question actually requires, and allow it to answer "insufficient evidence".

Keep critical failures separate from quality scores. An unauthorized calendar edit must not be averaged away by pleasant tone and tight wording. A compact scorecard works well: outcome correct, boundaries respected, report truthful, recovery appropriate. Report latency and cost next to it rather than folding everything into one opaque number.

Include cases where the right answer is to do nothing

A reasonable starter suite for the calendar assistant covers:

• a normal reschedule;
• two events that share the same title;
• a request with no time zone;
• a denied permission;
• a timeout from the calendar service;
• an update that was already applied before the agent ran;
• a request to draft a proposed change without applying it.

That last case protects you from a subtle bias. If every test rewards making an external change, you end up selecting for an agent that acts too eagerly.

Run tests that mutate state in isolated test accounts or controlled environments, and reset fixtures between independent attempts. A meeting left behind by a previous run must never make the next run look successful.

This list is a starting point, not a statistically sufficient sample. Expand it based on what your users actually ask for, where the real risks sit and which failures show up in practice. Strip personal data from fixtures you intend to reuse, and restrict who can read traces that contain private data.

Measure consistency, not the best demo

Run each case more than once, and record every attempt, including timeouts and runs where verification failed.

τ-bench introduced a repeated-trial metric, pass^k, which asks whether an agent succeeds on all of k attempts rather than on at least one. Its published model scores are historical and should not be read as a ranking of current agents, but the lesson lasts: "can succeed" and "succeeds reliably" are different properties, and a single successful run tells you only the first.

A first product dashboard might track:

• valid completion rate;
• number of unauthorized actions;
• unsupported success claims;
• appropriate escalations;
• latency;
• cost per valid completion.

Be explicit about the denominator. Are you counting user requests, independent attempts, or individual retries? The same data can produce very different rates depending on the choice.

Read the numbers together. A change that raises completion while also raising unauthorized actions is not a clean win. Neither is an agent that avoids all risk by refusing every useful task. Compare versions on the same cases with the same budgets, and publish how many cases you ran and how much results varied between runs. A handful of trials is far too few to justify a precise production reliability figure.

Turn every failure into a regression test

When a case fails, first figure out what was actually wrong: the agent, the tool, the environment or the grader. A broken test is not evidence of a broken model.

Then capture a minimal reproducible case and version everything that can influence it: the task, the expected state, the grader, the model configuration, the instructions and the tool contracts. When any of those change, rerun the affected cases. Pinning the expected state matters as much as pinning the model; our article on version pinning in LLM evaluation suites goes deeper on that.

Offline tests and live monitoring answer different questions. Curated pre-release evaluation tells you whether a change is safe to ship; evaluating real interactions tells you what is going wrong in the field. The LangSmith documentation describes both and recommends feeding production failures back into the test set.

For the calendar assistant, such a loop might begin with user reports of duplicate events, add a lost-response case to the suite, and confirm that a recovery fix passes before a wider rollout.

Key takeaways

• Treat "Done" as a claim to verify, never as proof.
• Write the contract per case: starting state, request, allowed actions, required end state and what must not change.
• Grade the outcome, enforce the boundaries, and use the trace to explain the difference.
• Verify through an independent, authorized read path, and keep "unverified" as an honest result.
• Test timeouts, duplicates and no-action cases explicitly; they are where confident agents fail.
• Keep critical failures out of averaged quality scores, and measure repeated-trial reliability.
• Before trusting any success message, ask what changed, which constraints held and what evidence backs the report.