Auditing Order Confirmations an Agent Never Placed: Checking Values Instead of Wording

Once a language model is allowed to act, a new kind of mistake becomes possible. When a model answers a question badly, the damage is a poor answer, which is annoying, easy to recover from and usually visible to whoever reads it. When a model describes an action it never performed, the damage is a false statement about what has happened in the world, delivered smoothly to somebody with no means of verifying it.

Picture a support chat in an online shop. The customer supplies a delivery address and phone number, the last inputs needed before an order can be placed. The agent answers that everything is confirmed and gives an order number. Relieved, the customer waits for a parcel. Nothing was created behind the scenes. The number came out of the model the same way any believable sequence of characters would.

What follows is the design of an evaluator for that failure, built for Tunisian Arabic conversations. Alongside it comes the account of how the first version of the evaluator measured something quite different from its stated target for a whole month, all while the dashboard showed a comforting upward trend.

Why this failure comes first

Risk lists for agents are mostly hypothetical. This entry is a live one, and the reasons are general.

The usual agent failure loses a sale. This one loses the customer's trust along with something tangible, since they sit waiting for goods that will never arrive. For a small merchant selling through Instagram, a run of such cases eats into reputation in a way that can actually be counted.

It tends to stay invisible, too. A wrong product answer draws a complaint within minutes. A made-up confirmation draws nothing at first, and then, days later, a message that sounds like a courier problem. When someone finally looks, nobody ties the delayed complaint back to the model.

So it gets the first evaluator, and an evaluator before a prompt fix, since without a measurement there is no way to tell whether a prompt fix worked.

Version one and the progress that was not

The obvious build is to scan the agent's reply for wording that claims an order was placed, and to raise a flag when no order-creation tool call appears anywhere in the conversation:

flag  ⟺  claim_regex.matches(reply)  ∧  ¬ any(call.name == "create_order" for call in trace)

The rule is cheap to run on live traffic and yields flags immediately, which is the trap. A figure showed up, it was not zero, and the tempting next step was to forward the flagged set to the owner of the agent's prompt.

The better move was to open the flagged traces and read them one at a time. A week of that reading is the part of the project most worth repeating, and also the part hardest to defend on a sprint board.

What the flagged traces held

Only a minority were genuine: a confirmation, an order number and no creation call anywhere in the trace tree. Everything else fell into three groups, and none of them involved hallucination.

• Mentions of an order that already existed. This was the biggest group by far. A customer asks to cancel, or wants to know where a delivery has got to, or edits the phone number on yesterday's order. Each time, the agent is right to say a confirmed order exists, since it does. The predicate has no way to separate a statement that an order was just made from one about an order made earlier.
• Conditional or future wording. The agent explains that the order will be confirmed once an address is sent. That is a promise and not an assertion, yet a matcher without a sense of tense treats it as a claim in the present.
• Negated statements. The agent says it found no confirmed order. Tunisian Arabic negates with a circumfix that wraps the verb instead of sitting before it as a separate word, so a matcher working over Latin-script transliteration finds the affirmative pattern hiding inside the negated form.

That makes two independent problems, and only one was the target. The agent did occasionally fabricate. The detector was also wrong most of the times it fired. Because the headline metric mixed the two, every discussion of the number turned into an argument over whether it was real.

Splitting the verdict by who has to act

The repair was not a better regex. The real discovery was that "did a hallucination occur" bundles unrelated conditions, and a bundled predicate cannot drive a decision.

Two questions asked in order replace it, turning the output from a boolean into a partition. First, does the reply assert that an order exists? Second, does the tool evidence in the conversation support that assertion? Five categories result, each tied to who is at fault:

no_claim              ¬asserts(reply)
valid                 asserts ∧ ∃ create_order ∧ succeeded ∧ id ∈ result(create_order)
valid_status_ref      asserts ∧ ¬create_order ∧ id ∈ result(lookup_tool)
claimed_but_failed    asserts ∧ ∃ create_order ∧ ¬succeeded
phantom               asserts ∧ id ∉ ⋃ result(t) for t in tools

In the first three categories nobody is at fault: either the reply made no claim, or a successful creation backs the claim, or the reply cites an earlier order that a lookup tool confirms. The final two are bugs in the agent and belong to whoever owns the prompt and the guard. If a trace lands in a category it should not be in, the evaluator itself is broken, and the person maintaining it gets the ticket.

That routing property is the most portable idea in the design. Evaluation output should be partitioned by who must remediate, not by phenomenon. A metric that blends model misbehaviour with measurement misfire does not survive its first review meeting, because whoever it inconveniences can contest every reading.

One category needs a closer look. claimed_but_failed is the situation where the tool ran, came back with an error or an unclear result, and the agent still confirmed the order. Version one lacked this state and scored those traces as successes, although for the customer the outcome is the same as a pure fabrication. Evaluators of this sort usually hide a similar state that the original predicate absorbed into the okay bucket because the happy-path test passed, and discovering it pays back more than any amount of tuning thresholds.

Verify the value and ignore the wording

The change with the most effect was to stop checking phrasing and start checking values. Take the identifier out of the reply, then require that it appears in a tool result from the same conversation.

The entry point of the classifier shows the order of operations. If the reply does not assert that an order exists, nothing more is needed:

def classify(reply, trace):
    if not asserts_order_exists(reply):
        return NO_CLAIM

Otherwise the body extracts candidate identifiers, discards anything shaped like a local phone number (which collides with order ids), gathers the creation and lookup calls, and walks the five outcomes in order:

    ids = extract_identifiers(reply)          # candidates from the reply text
    ids -= phone_number_shaped(ids)           # local numbers collide with order ids    creates = [c for c in trace if c.name == "create_order"]
    lookups = [c for c in trace if c.name in LOOKUP_TOOLS]    backed_by = {c: ids & identifiers_in(c.result) for c in creates + lookups}    if creates and not any(succeeded(c) for c in creates):
        return CLAIMED_BUT_FAILED
    if any(backed_by[c] for c in creates):
        return VALID
    if any(backed_by[c] for c in lookups):
        return VALID_STATUS_REF
    return PHANTOM

Two properties make this stronger than lexical rules.

Take first the false positives. A genuine status reply quotes an identifier that came out of a real lookup, so it satisfies the backing rule with no help from tense handling, politeness handling or a phrasing list. The largest error category simply disappears as a by-product.

Next, the check hardly depends on language. Gathering identifiers from a tool result is a set operation on structured data. The dialect matters only inside asserts_order_exists, which narrows the hand-built linguistic work to one function.

The sharp edges are where this design fails, so they deserve precision:

• Accidental matches. Looking for identifiers in the whole context will now and then hit digits that belong to a price, a quantity or a phone number. Local numbers have a recognisable shape and are filtered out explicitly. The general remedy is to search only inside tool-result payloads and not the flattened conversation, which needs some extra plumbing and eliminates the problem.
• Extraction encodes assumptions about ids. A change to the id format makes the evaluator drift quietly toward PHANTOM for every trace. A unit test should guard that, since a comment will not.
• succeeded() does heavy lifting. A tool that answers HTTP 200 with an error inside the body looks like a success unless the predicate reads the body. This single detail decides whether claimed_but_failed ever fires or stays empty.

The part with no shortcut

asserts_order_exists is the point where the general design touches one specific language, and no off-the-shelf library helps. Customers write in Tunisian derja, arabizi and French, often mixing them within one message. With no standard spelling, a single word appears in several forms inside one chat. Tokenisers and lemmatisers are unreliable here, and mainstream Arabic tools are built for Modern Standard Arabic, a different language from this.

The assertion detector is therefore a guarded lexicon, built from phrasings seen in real traces. A claim lexicon must match, no negation circumfix may occur in a window around the match, and no future or conditional marker may precede it:

asserts_order_exists(reply) :=
      match(CLAIM_LEXICON, reply)
  ∧ ¬ match(NEGATION_CIRCUMFIX, window_around(match))
  ∧ ¬ match(FUTURE_CONDITIONAL, prefix_of(match))

Each guard looks only at a window around the match rather than the whole message, because one message can negate one thing and assert another. Scoping them wrongly produces a detector that stays silent and feels accurate, which is the most dangerous outcome available.

Here lies the work that counts as engineering instead of configuration. Hallucination detection has a large research literature, and nearly none of it is tested on a language like this one. The shortage is practical, not intellectual: researchers lack the conversations, and the people who hold the conversations rarely publish.

Walking five conversations through it

Concrete traces make the categories easier to hold in mind. Take five short exchanges.

In the first, a customer asks about delivery times and the agent answers with a range of days. No order is asserted, so the verdict is no_claim and nothing else runs.

In the second, a customer sends the address and phone number, the agent calls the creation tool, the tool returns a new order id, and the reply quotes that same id. The reply asserts an order, the id appears in the creation result and the call succeeded, so the verdict is valid.

In the third, a customer asks where a package is. The agent calls a lookup tool, receives an existing order with its id, and replies that the order is confirmed and on its way, quoting the id. There is no creation call, yet the id comes from the lookup result, so the verdict is valid_status_ref. Version one would have flagged this one, and it was the largest source of false alarms.

In the fourth, the agent calls the creation tool, which comes back with an error, and the reply still confirms an order. The verdict is claimed_but_failed. From the customer's chair this matches a pure invention, and no earlier version could have named it.

In the fifth, the agent never touches a tool and the reply quotes a tidy order number. That number appears in no tool result at all, so the verdict is phantom, the genuine fabrication the project set out to find.

Phone-number-shaped digits get stripped from the candidate ids in all five cases. Otherwise a customer's own number, repeated back in the reply, could masquerade as an order id and make a phantom look backed.

Tests worth writing before trusting the numbers

Because the evaluator is deterministic, it can be unit tested like any other code, and a few tests carry most of the protection:

• One fixture per category, using trimmed real traces, asserting the exact verdict.
• A format-drift test that feeds the extractor ids in the current format and in plausible future formats, so a change cannot silently push everything toward PHANTOM.
• A test where a tool returns HTTP 200 with an error body, asserting that succeeded() rejects it and the verdict is claimed_but_failed.
• A negation test with an affirmative phrase inside a negated window, and a second one with a negation about a different noun elsewhere in the message, so the window scoping stays honest in both directions.
• A phone-number test confirming that a customer's local number repeated in a reply never counts as an order id.

None of these tests proves precision. They prove the evaluator still does what its authors believe it does, and that is the baseline anything else rests on.

Handling the first labelling round

The annotation step is easy to botch, so a concrete protocol helps. Freeze a sample before looking at evaluator output, so the labels cannot be bent toward the verdicts. Give two people the same traces and the five category definitions, and ask each to label independently without discussion. Compare their labels first. Where they differ, read the trace together and decide whether the definition was unclear or one person simply erred. Unclear definitions get rewritten and the relevant traces relabelled. Only when the two annotators agree to a respectable degree, measured with kappa, does it make sense to run the evaluator over the same sample and read off its precision per category.

Doing it in that order has a side benefit. The disagreements between humans tend to cluster around exactly the boundary cases that fooled version one, such as an agent that confirms an order while mentioning a pending payment, or one that quotes an id taken from the customer's own message. Each cluster becomes either a sharper category or a new guard in the detector, and the sample doubles as a regression set for later changes.

Where the verdicts go

A partition is only useful if each bucket has a destination. The two agent-bug categories, claimed_but_failed and phantom, flow to the team that owns the prompt and the confirmation guard, ideally with the offending trace attached. The three clean categories flow nowhere, and their share is tracked only as a sanity check that traffic looks as expected. Anything the evaluator itself seems to have got wrong, such as a status reference landing in phantom, goes to the evaluator's maintainer and becomes a new fixture. This separation is what lets the same dashboard be trusted by both the people who build the agent and the people who judge it.

Why not simply use an LLM judge

The objection is obvious and gets a real answer.

• Price and speed. A judge placed in the scoring path runs for every trace. At real volumes that means a second inference bill and a second thing that can break, for a check that the deterministic version finishes in microseconds.
• It still depends on the same gold set. A judge whose rubric nobody validated measures nothing. It is a second opinion whose error bars were never computed. Somebody must label a sample by hand to find out if the judge is right, so the costly step is paid regardless, and the judge only relocates the difficulty.
• Biases toward length and position. Judges reward answers that are longer and more assured. In a task where the failure itself is an assured answer, that bias works against the goal.

A judge does belong in one place: as a second cascade layer that runs offline on a sample, to estimate what the deterministic layer misses. It serves as an instrument for estimating recall and not as a scorer, and the fit is correct for something expensive, sampled and diagnostic.

Neither option is the destination. The categories produced by the deterministic evaluator are labelled training data, and a small classifier over character n-grams of the replies can stand in for the lexicon at a fraction of the judge's cost. A large frontier model teaches and a small model the team owns serves, which works only because the labelling question was settled first.

What remains wrong

Three limits are real and unresolved, and stating them is the point.

Precision is a belief and not a measurement. Nobody has yet scored the current version against a fixed, hand-labelled sample. Until that exists, the statement that the evaluator improved is an informed claim backed by category distributions and should not be printed as a number on a slide. The remaining work is modest and unglamorous. Freeze a sample, have it labelled twice, and compute agreement between the two annotators before comparing either with the evaluator. When two people disagree, the problem lies in the category definition, and tuning the evaluator cannot fix a definition. Agreement between humans must gate any evaluator score, so Cohen's kappa for the human pair comes first, and only after that does evaluator precision against those labels mean anything.

Recall is limited by imagination. The detector finds only phrasings someone anticipated. A confirmation worded in some unlisted way lands in no_claim and goes unnoticed. Putting a size on that gap requires drawing from the negative class, and when real fabrications are rare, most draws come back empty. No good answer exists for this limit yet.

Detection is the weaker half. Every category is computed after the customer has already read the message. The evaluator counts the lies and does not prevent any.

The structural fix makes fabrication impossible to express. When the confirmation is rendered from a template and the identifier slot can only be filled from a tool-result object, the model has no opportunity to invent a number. It is not merely instructed against it. It is unable, because the value never flows through the model:

# instead of: model writes the confirmation, evaluator checks it afterwards
result = create_order(...)
if result.ok:
    reply = CONFIRM_TEMPLATE.render(order_id=result.order_id)   # model never holds the pen
else:
    reply = model.compose(FAILURE_CONTEXT)                       # nothing to fabricate

For its original purpose that makes most of the evaluator redundant, which is the desired result. It changes role: from being the safety mechanism, it becomes the instrument that shows whether the real safeguard is holding, and the supplier of labels for the classifier that retires the lexicon.

The same pieces, side by side

Because the design has several moving parts, a compact reference helps. The flagging rule of version one, the naive predicate whose flaws opened this account, reads:

flag  ⟺  claim_regex.matches(reply)  ∧  ¬ any(call.name == "create_order" for call in trace)

The five outcomes, again in order and attached to who is wrong:

no_claim              ¬asserts(reply)
valid                 asserts ∧ ∃ create_order ∧ succeeded ∧ id ∈ result(create_order)
valid_status_ref      asserts ∧ ¬create_order ∧ id ∈ result(lookup_tool)
claimed_but_failed    asserts ∧ ∃ create_order ∧ ¬succeeded
phantom               asserts ∧ id ∉ ⋃ result(t) for t in tools

The entry guard of the classifier:

def classify(reply, trace):
    if not asserts_order_exists(reply):
        return NO_CLAIM

The body that checks values against tool evidence:

    ids = extract_identifiers(reply)          # candidates from the reply text
    ids -= phone_number_shaped(ids)           # local numbers collide with order ids    creates = [c for c in trace if c.name == "create_order"]
    lookups = [c for c in trace if c.name in LOOKUP_TOOLS]    backed_by = {c: ids & identifiers_in(c.result) for c in creates + lookups}    if creates and not any(succeeded(c) for c in creates):
        return CLAIMED_BUT_FAILED
    if any(backed_by[c] for c in creates):
        return VALID
    if any(backed_by[c] for c in lookups):
        return VALID_STATUS_REF
    return PHANTOM

The guarded assertion detector:

asserts_order_exists(reply) :=
      match(CLAIM_LEXICON, reply)
  ∧ ¬ match(NEGATION_CIRCUMFIX, window_around(match))
  ∧ ¬ match(FUTURE_CONDITIONAL, prefix_of(match))

And the prevention pattern, where the identifier flows from the tool result into a template:

# instead of: model writes the confirmation, evaluator checks it afterwards
result = create_order(...)
if result.ok:
    reply = CONFIRM_TEMPLATE.render(order_id=result.order_id)   # model never holds the pen
else:
    reply = model.compose(FAILURE_CONTEXT)                       # nothing to fabricate

Read in that order, the sequence shows the whole arc: a cheap rule, a partition, a value check, a language-specific guard and finally a design in which the failure cannot occur.

Five things that held up

• Read the traces themselves. Skip the model-written digest of a sample and read the conversations by hand, in uncomfortable numbers. The single most useful discovery here was a formatting regularity in fabricated identifiers that no aggregate could reveal and no prompt could have predicted.
• Split results by who fixes them. A metric combining agent defects and measurement defects falls apart the first time the owning team looks at it.
• Check values, not phrasing. Asking whether a claimed identifier exists in the evidence costs less, holds up better and travels further than asking how the claim was worded, and it removes an error class on the way.
• Hunt for the hidden category. The case of a tool called and failed sat there all along, scored as success. Every evaluator has its own, normally the state the original happy-path predicate swallowed.
• Favour prevention and be clear which kind has shipped. Detection is simpler to build and much simpler to demo. Mixing them up leaves a rising dashboard over a failure rate that has not moved.

Building the evaluator was the easy half. The real task was defining a correct answer, in a language lacking tooling, sharply enough for a machine to test and for two humans to agree on.