Scoring Voice Agents on Outcomes: A Three-Layer Test Harness

Picture three phone calls that look perfect in a review. In the first, a customer fixes an order number halfway through a sentence, and the agent reads the corrected number back. In the second, the customer cuts in while the agent is talking, and the agent stops cleanly with no talking over anyone. In the third, the agent says the order has been updated.

Now look at the backend. The first call updated the wrong order, namely the one the customer had just moved away from. In the second, the tool call that was in progress when the interruption hit kept running anyway. In the third, the update never landed, and the system still holds the old value.

No transcript review would flag any of these. A WER check or a conversation-quality rubric wouldn't either. Yet all three missed the thing that counted. The rest of this piece lays out how to build a test harness that catches that whole class of failure before a customer does.

Why transcript metrics aren't enough

The usual voice-agent measurements are word error rate, latency, how fast it yields to interruptions, pronunciation and natural flow. They're all worth tracking, and none of them is wrong. They answer a narrower question than production asks, though. They report whether the agent sounded right. They say nothing about whether anything actually happened.

A harness worth running has three layers:

• Conversation: did the agent hear and answer correctly?
• Execution: were the right tools called with the right arguments?
• State: did the intended change take effect in the outside system?

Apply that to the three calls. The first passed the conversation layer and failed execution. The second also failed execution, but for a different reason: a stale call was never cancelled. The third cleared both conversation and execution and still failed on state, because the backend write didn't stick. Three failures sat in different layers, and all of them shared one symptom, a transcript that looked fine.

Map what can go wrong first

Nothing should be scored until there's a matrix of failure modes. Four categories cover most of the ground.

Interruptions include a cut-in before the agent finishes talking, one while a tool call is being put together, one right after a tool call starts executing, and accidental ones triggered by background noise that should never have caused a stop.

Corrections cover a changed order number, a new delivery date, an altered quantity, and negation, as in "ship it" followed by "actually, don't."

Tool failures range from timeouts and empty results to a tool fetching the wrong record, backend 500s, a success reply that shows up late, and duplicate requests caused by a retry that isn't idempotent.

State conflicts occur when three versions of the truth coexist: the transcript says one thing, the live state another, and the payload that was actually sent to the tool a third.

These overlap in practice. A correction arriving during a tool-failure retry is a scenario of its own. Still, grouping them like this lets each category be tested alone first.

What a test case looks like

Every case requires an input scenario, a statement of what a correct system should do at all four layers, and a structure rigid enough for automatic scoring so nobody has to read and judge it by hand. A minimal schema looks like this:

{
  "id": "corr-02",
  "category": "correction",
  "setup": {
    "field": "order_number",
    "initial_value": "5590",
    "corrected_value": "5599",
    "interrupt_stage": "during_tool_prep"
  },
  "expected": {
    "captured": true,
    "active_state_correct": true,
    "tool_args_correct": true,
    "stale_write_prevented": true,
    "backend_correct": true,
    "confirmation_matches_backend": true,
    "should_execute": true
  }
}

The setup block is deliberately loose, because interruptions and tool failures need different fields than corrections do. The expected block is strict. Every case is scored on the same six axes, and that's what lets 30 very different scenarios roll up into one scorecard.

The six axes

1. Event captured. Did the system notice the correction or interruption at all?
2. Correct active state. After the event, is the live working state updated to match?
3. Correct tool args. Were the values in the tool call current and consistent?
4. Stale write prevented. Was every action based on out-of-date state blocked?
5. Backend correct. Does the stored result fit what the customer really intended?
6. Confirmation matches backend. Is what the agent told the customer what's actually stored?

One rule matters above the rest: these never get averaged into one number. A system that is right on five axes and wrong on backend_correct produced the wrong outcome, and five green checks beside it don't dilute that. A case counts as a genuine pass, called End-to-End Task Integrity, only when all six are true at once.

What a passing trace looks like

Now take a scenario where the fix arrives early: the correction lands before any tool call exists, so there's nothing stale to cancel.

Customer says 4821                    → state_version = 4
Correction: 4281                      → state_version = 5
Tool call built using version 5
Backend verified = 4281
Agent confirms 4281

Each state transition carries a version tag, and the tool call is only built once a stable version exists. Nothing is ever constructed against version 4, so nothing has to be thrown away later. That's the shape a passing case takes. It isn't "no correction happened." It's "the correction fully propagated before anything downstream committed."

Running it on a flawed mock

To make this concrete, take a runnable harness (fixtures, schema and scorer) and point it at a rule-based mock pipeline. The mock transcribes accurately and handles turn-taking well, but it never invalidates a tool call once built or sent, and it never revisits a tool result unless it recognizes the result as malformed. That's exactly the sort of system behind the failures above, made explicit so the harness has a target.

The run used 30 synthetic calls, split evenly into correction, interruption and tool-failure scenarios.

• Event captured: 24 of 30
• Correct active state: 24 of 30
• Correct tool args: 11 of 30
• Stale write prevented: 8 of 30
• Backend correct: 8 of 30
• Confirmation matched backend: 7 of 30
• End-to-end task integrity: 7 of 30

The conversation layer looks solid, with 24 calls out of 30 capturing the event and updating state. Execution and state collapse after that. Only 8 calls prevented a stale write, and just 7 produced a spoken confirmation that agreed with what ended up in the backend. Judged by transcript quality alone, this system would seem ready for production. It isn't.

Turning failures into a fix list

A red result is only useful when it names the layer to repair. Sorting the 23 cases that didn't pass by what caused them gives this breakdown:

• 10 cases: arguments for the tool were built from stale values
• 4 cases: a faulty tool result was accepted without any check
• 3 cases: calls already underway were never cancelled
• 2 cases: background noise wrongly counted as the customer interrupting
• 2 cases: a backend error combined with a retry that wasn't safe to repeat
• 1 case: the confirmation was spoken before an async result came back
• 1 case: no idempotency key on the action

Ten of those 23 misses come from a single cause. A tool call had already been assembled or dispatched when a correction or interruption showed up, and no mechanism downstream could cancel it. That's one architectural hole appearing across ten scenarios, not ten bugs, and spotting it is the whole point of running a harness. A bare "the call failed" gives a team nothing to act on, whereas "argument construction broke in 10 of 23 cases" sets the priorities for the next sprint.

A release gate for consequential actions

Before shipping a change to a voice agent that takes consequential actions, "conversation quality improved" isn't a high enough bar. The gate should read:

• No stale-write failures across the correction and interruption suites
• No mismatch between confirmation and backend
• Idempotent tool retries
• Critical actions verified against the backend after execution, rather than assumed from the tool's return value

Only when the conversation, the tool trace and the backend state agree on what happened is a voice agent ready for production. Most systems today can't claim that, and the way to find out is to quit grading the transcript and grade the result.