Testing a Rubric Gate That Retries an Agent's Wrong Final Action

Anthropic's Claude Outcomes feature packages up a loop that plenty of teams wire together themselves: a worker agent drafts an answer, a second component grades it against a rubric, and a miss earns the worker another go. The idea underneath is small enough to try without the managed product, so a stripped-down version was rebuilt in a controlled setup with agent output, a rubric judge and exactly one retry.

The question

The test asks a single thing. When an agent must pick a structured decision, does a rubric-gated retry loop lower the number of wrong final actions?

Final actions matter here because many real agent failures have nothing to do with prose quality. A reply can read well, the JSON can parse, and the chosen action can still be the wrong one. In a support workflow that turns into an operational error. Say an agent returns DENY where NO_ACTION was correct: the cautious-sounding rationale doesn't rescue it.

The setup

Thirty synthetic support cases and one model, gpt-5.4-mini, made up the run. A sample like that can't pass for a benchmark. The value lies in how failures looked: which ones the gate repaired, which it missed and where the pattern earns its cost.

Each case bundles a short customer scenario, a policy and a closed list of permitted final actions:

APPROVE
DENY
ASK_CLARIFYING_QUESTION
ESCALATE
NO_ACTION

Two loops competed. In the baseline, the case goes directly to the agent and whatever action comes back is recorded. In the gated version a rubric judge reviews the first answer, and a failure sends the agent back for one more attempt, this time carrying the criteria it missed.

The diagram below is a compact reference for outcome-gated agent workflows. It shows rubric gates testing a proposed action against a policy contract before any final decision is accepted.

The contract was kept tight on purpose, with a closed action set, explicit policy constraints and a single retry once a rubric check fails.

Baseline:
case -> agent -> final action

Gated:
case -> agent -> rubric judge
if failed -> retry once
retry answer -> final action

Cases were synthetic but built around common production-style constraints:

A real LLM provider was used and not a deterministic simulator, because the interesting part is how a model actually behaves once its first answer is judged and revised.

A judge that doesn't hold the answer key

A judge that already knows the answer would defeat the point. Every case holds an expected_action, yet that value feeds offline metrics after the run only, and the judge never gets it. What the judge sees is the action the agent picked, and it checks that action against objective policy constraints.

Picture a refund request that has an open chargeback in the payments workflow. NO_ACTION could be the right outcome, because payments owns that resolution and support shouldn't rule on it. A weak judge would simply test whether the model chose NO_ACTION. That amounts to an answer-key check. It suits offline scoring but isn't a rubric gate.

A genuine gate poses a different question. Given DENY, has the agent wrongly cast support as the decision owner while a different workflow holds the case?

At runtime, a hidden answer key is usually not what's wanted. What's wanted is a check on whether the proposed action breaks a clear contract. Here the contract was simple: choose one final action and don't pick one that conflicts with the case's policy constraints.

What happened

In a single pass over the 30 cases, the baseline agent got 6 final actions wrong. Once the judge and one retry were added, only 2 stayed wrong.

The gain is real, but the numbers alone mislead. The loop didn't give the agent any broad new skill. It worked only where the judge could name a specific clash between the chosen action and the policy contract, which the failure examples bring out.

What the gate fixed

The loop repaired 4 of the 6 baseline failures. Most fixes came from cases where the baseline was too quick to deny. The policy didn't support immediate approval, but it didn't demand denial either. The better move was to ask for clarification or wait on another workflow.

Take a request to change an account email. It arrived from a new device, MFA hadn't been completed, and the requester did know the billing ZIP code. The policy required completed MFA or some other approved ownership check before support could proceed. The baseline answered DENY, although ASK_CLARIFYING_QUESTION was expected. Closing the door was premature when a route to an approved check remained, so asking was the proper step. The judge rejected the denial as too early, and on the retry the model changed to ASK_CLARIFYING_QUESTION.

Another repaired case concerned a replacement device. A customer asked support to resend one, but an RMA had been created that morning and was waiting for a warehouse scan. The baseline picked DENY and the expected action was NO_ACTION. Denial misdescribed the situation, because the customer wasn't ineligible. The replacement process was already running and support shouldn't open a duplicate RMA. The retry corrected it to NO_ACTION.

Both examples show the pattern's practical value. The gate didn't make the model broadly smarter. It forced a repair of one specific mismatch between the selected action and the policy contract.

What it couldn't fix

The two remaining failures say more than the aggregate score. Both were existing-workflow cases where the expected action was NO_ACTION, and the final gated answer stayed DENY even after the retry.

In the first, a refund request had an open chargeback in the payments workflow. Policy barred support from issuing refunds while a chargeback was active, because payments owns the resolution. In the second, a buyer wanted a manual shipment release while a fraud review was open, and policy forbade support from releasing shipments in that state. In both cases the judge spotted the problem and returned failed criteria, and the retry kept the wrong action anyway.

That shifts how the pattern ought to be judged for production use. A gate can notice that an action breaks a rule, but noticing isn't mending. Suppose the model can't tell "refuse the request" from "stay out of it because another workflow decides". Then even a flagged answer can come back unchanged. Sounding safe doesn't make an action right, and DENY reads as conservative while still being wrong when the agent lacks authority over the case.

Where to trust it

Success here rested on a checkable outcome. The agent picked from a closed set, the policy limits were spelled out and the judge could compare the chosen action with one concrete rule. Outcome-gated retries earn more trust in situations like these:

More caution applies when the outcome is subjective:

It shouldn't be treated as a general reliability layer. Reserve it for places where a wrong action costs enough to pay for an additional model call and where the desired result can be put into a contract.

Production changes worth making

One adjustment matters above the rest: read repeated non-repairs as feedback on the design. When the agent keeps mixing up DENY and NO_ACTION, tighten the action definitions, add examples or rework the workflow contract. Yet another retry rarely fixes a gap in understanding.

Keep runtime gating apart from offline scoring too. At runtime the judge enforces constraints, while offline evaluation measures final actions against labeled expectations. Merging them flatters the system. It also pays to log each retry, covering the first action, the failed criteria, the revised action and the final verdict, since the mix of repairs and non-repairs frequently says more than overall accuracy.

The takeaway

Outcome gates pay off when "done" can be written as a contract. Here the contract was simple: pick one final action that doesn't violate the policy constraints. Under that condition the gated loop cut wrong final actions from 6 of 30 to 2 of 30.

Those leftover failures are where production attention belongs. The judge caught the problem and the retry still produced the wrong action. So the open question isn't just whether a system can grade output, but whether the whole loop can translate that grade into the right decision. That's the evidence to ask for before any outcome-gated loop touches a live workflow.

Further reading

Anthropic's API documentation on defining outcomes and its cookbook entry on agents that verify their own work cover the managed feature this experiment mirrors.