An Architecture Against Hallucination Compounding in Agent Graphs

The instinctive patch is to bolt on another critic agent. That is exactly how you scale up the deception instead of fixing it.

If you have ever put a single-agent RAG chatbot into production, you already recognize the failure: the model answers something the retrieved documents never said, and it delivers that fabrication with the same tone of certainty it uses when the citation is genuine.

In a multi-agent pipeline, this same failure gets amplified, because you are no longer dealing with one model's output — you are dealing with an entire graph of them. A planner dreams up a subgoal. A researcher invents a source to support it. An analyst pulls a number out of that invented source. A writer turns the number into a recommendation. A tool-calling agent then executes that recommendation against your billing system. At every handoff, there's an opportunity for a guess to be laundered into something that looks like a verified fact.

Teams building LangGraph-style multi-agent systems for legal, financial, and operations use cases run into a recurring pattern. The issue isn't that the underlying model is unintelligent. It's that the workflow itself has no epistemic contract. Nothing in the graph forces any agent to disclose where a claim originated, what evidence would disprove it, or what should happen if that evidence simply isn't there. So the system does what language models always do when there's a gap: it generates something plausible to fill it.

What follows is the architecture worth using when the output can move money, affect a legal filing, or land in a client's inbox. It's not a quick tutorial, and if you're hoping for a single package install that makes your agents trustworthy, you won't find it here.

The compounding problem nobody budgets for

Any individual LLM call has some baseline rate of unsupported claims — call it p. Chain five agent hops together, and if each hop has the potential to introduce or magnify an unsupported claim, your effective error rate is no longer p. It becomes a product of conditional probabilities, compounded by a second issue unique to multi-agent graphs: the transfer of authority between agents.

Agent B treats Agent A's output as established fact simply because it arrived packaged as a structured JSON object labeled something like research_findings. The schema validates. The field types are correct. The actual content is invented. Passing schema validation is not the same as being true, yet most teams only bother enforcing the former.

Three specific failure patterns show up in real deployments more often than most write-ups acknowledge:

1. Fabricated tool calls. The model emits a function invocation that looks syntactically correct — something like search_matters(client_id=…) — but targets a tool that doesn't exist, or supplies arguments that would fail if the tool's schema were actually enforced. Orchestration layers that quietly coerce mismatched types, or that let the model retry with slightly reworded arguments, effectively train the system to push through gaps in data rather than surface them.
2. Laundering across agents. The researcher hedges with "according to the matter file." The analyst downstream never actually opens that file. The writer then cites the analyst as the source. By the time a human reviews the final summary, the original qualifier has vanished entirely. This is precisely how a tentative maybe turns into a billable, confidently stated fact.
3. Verifier theater. Teams often respond by adding a "critic" or "verifier" agent whose sole job is catching hallucinations. The problem is that this verifier is itself an LLM, conditioned on the same context — frequently from the same model family — and implicitly rewarded, through your prompt design, for being agreeable and helpful. A verifier optimized to be helpful will tend to confirm rather than challenge. A genuinely independent verifier needs a separate source of information, a distinct objective function, and a refusal pathway that is less costly than simply agreeing. Very few architectures actually provide that.

Retrieval-augmented generation doesn't rescue you from this. Retrieval only supplies a prior. If the planning agent never formulated the right query, or if the chunking process split the one paragraph that would have disproven the claim across two separate embeddings, the researcher agent will still retrieve something fluent-sounding and topically adjacent. Being adjacent to the right answer is not the same as being logically entailed by it.

What “grounded” has to mean, or it means nothing

"Grounded" shouldn't be a loose descriptor of vibe or tone. A claim inside a multi-agent system only counts as grounded if every one of these conditions holds:

1. The claim carries an explicit type. It's tagged as an Assertion, a Conjecture, a Quote, or an ActionIntent. Blending these categories into one untyped text field is exactly how audit trails fall apart.
2. Each Assertion links back to a provenance record — a retrieved passage, a tool's output, or a fact a human supplied directly. That link is a content hash, never a URL the model made up on its own.
3. A deterministic, non-LLM check has verified that the referenced provenance record genuinely exists somewhere in the run's ledger. Confirming existence is inexpensive; confirming entailment is not. Both checks are necessary, and in that specific order.
4. When a check fails, the system abstains or asks for clarification. It does not quietly rewrite the claim until the wording sounds convincing enough to pass.

An architecture incapable of refusing an answer is incapable of being reliably truthful. Generation is the path of least resistance by default; abstention has to be built in as an explicit, first-class state in the graph — and it needs to be cheaper to reach than simply retrying with a different prompt.

That's the entire premise. Everything else is the engineering required to make these four rules hold up once you introduce LangGraph, real tool calling, and a product manager insisting the demo always produce an answer.

The control plane, not the prompt

Treat your agents as an untrusted runtime, and wrap them in a control plane. That plane has six components. Leave one out and the rest quietly degrade.

1. Typed tool bus

Every tool gets a versioned JSON Schema for both input and output, stored in a registry that the model has no permission to modify. The model can propose a call, but a deterministic validator has to accept or reject it before anything happens as a result. No type coercion, no "good enough" mapping between an enum value and whatever the model produced. If a call is invalid, it goes back to the planner as a structured error — never as a conversational scolding.

This looks like an obvious requirement, yet it's the part most CrewAI or AutoGen demos leave out, because a demo never runs into a tool that actually writes to a permanent ledger.

For any tool with irreversible consequences — sending something, writing a file, deploying, updating a system of record — the bus should demand dual control: a ClaimSet that already passed verification, plus either a human-in-the-loop approval or an explicit policy grant. Under this design, the operator agent never gets a direct line to those tools.

2. Append-only citation ledger

Every retrieval result, every tool output, every fact a human supplies gets written into an append-only store, indexed by run_id and a content hash. Agents aren't allowed to just "remember" a source in their own scratchpad — they have to cite ledger ids instead.

If a writer agent produces a sentence with no ledger id attached, that sentence isn't an assertion at all. It's unlabeled text, and it should never be allowed through a schema gate. This is arguably the highest-leverage fix you can apply to an existing agent graph: stop letting free-form prose exit the system disguised as data.

Hashing matters here too. A model can cite doc:matter-4421#p3 and still misquote what's actually written on page 3. The ledger needs to store the literal span text — or at minimum a pointer into an immutable object store — so the verifier checks against those exact bytes, not against whatever the model recalls about them.

3. Schema gate (non-LLM)

Before any artifact moves downstream to the next agent — a research summary, a computed figure, a suggested action — it has to pass validation against a schema that includes:

• a claims[] array, each with type, text, ledger_ids[], and a confidence value that gets calibrated later rather than simply asserted by the model as "high"
• an open_questions[] array that the planner must either resolve or explicitly escalate
• an action_intents[] array that references a named tool, never a vague paragraph suggesting "we should probably..."

This gate has to be ordinary code — Pydantic, JSON Schema, a CEL or Rego policy, take your pick. It must not be an LLM. Put a language model here and you've simply reinstalled critic-agent theater one level down.

4. Claim verifier with a different information diet

This is where the real cost lives. Take each Assertion and split it into its smallest checkable pieces, so that a single piece names one subject, one relationship or property, and one point or window in time. For each of those pieces, pull the supporting evidence only from the ledger, never from a live web search and never from the model's own parameters.

Then run an entailment check: does the retrieved span support the proposition, contradict it, or say nothing about it at all? You can implement this with a small NLI model, a constrained decoder limited to copying from the span, or a human reviewer. What you cannot do is hand this job to the same large chat model, running the same system prompt, and ask it whether the claim "looks right."

When a claim gets rejected, nobody quietly patches the wording to make it pass on the next attempt. It comes back as Unsupported{proposition, missing_evidence}, and the planner's job from there is to go find more evidence, not to rephrase the problem away.

This step also catches a subtler failure: swapping a vague, unfalsifiable statement for one that only sounds rigorous. Saying an invoice looks overstated isn't a checkable claim by itself. Pinning it to a specific figure — naming the line item, the contractual cap it exceeds, the dollar amount over that cap, and the ledger entries backing all three — is what turns it into something a verifier can actually confirm or reject. A schema that accepts the first, vaguer form will end up validating tone rather than fact.

5. Eval harness on gold traces, not vibes

You need a frozen benchmark set: inputs, ledger snapshots, the claims you'd expect, and the abstentions you'd expect. Every change to the graph — a prompt edit, a model swap, a different chunker, a new tool — gets scored against:

• faithfulness: what fraction of emitted assertions are actually entailed by the ledger
• coverage: what fraction of the expected assertions the graph actually produced, since staying silent can itself be a failure
• abstention precision: when the system refused to answer, was that refusal actually warranted
• side-effect hygiene: zero irreversible tool calls executed without a valid grant

If a two-point drop in faithfulness can't block a deployment, you don't have a control plane — you have a dashboard that looks reassuring.

Whatever you do, don't build this evaluation set out of the model's own past outputs. That just encodes the current hallucination pattern as ground truth. Gold traces should come from the underlying source documents and the actual system of record, with the graph then run against them independently. It's slower to build this way, but it's the only version of the metric that actually means something.

6. HITL on the irreversible edge

A human reviewer isn't there to make the agents smarter. Their job is to stand at the exact point where the system is about to send an email, file a document, or write to a record. The review interface should surface the claim set, the relevant ledger spans, and the pending tool call — not a wall of chat history. If someone has to reverse-engineer why the agent believed a given fact, the design has already failed at its job.

For high-throughput processes — invoice review is a good example — human review should be sampled and triggered by exceptions rather than applied to everything. Let the system auto-approve when every assertion is fully entailed and the financial variance falls within an acceptable policy range. Everything else gets flagged, and only the specific unsupported claims are surfaced, not the entire document.

A sketch of the contract, not the implementation

Here's what the object passed between agents should roughly look like. Orchestration logic, the NLI verification stack, the ledger's storage layer, and the component that issues grants are deliberately left out — those are implementation-specific and belong in production code, not in a blog post.

{
  "run_id": "run_7f3c",
  "from_agent": "analyst",
  "claims": [
    {
      "id": "c_19",
      "type": "Assertion",
      "text": "Line item 14 exceeds the engagement-letter hourly cap.",
      "propositions": [
        {
          "id": "p_19a",
          "pred": "exceeds_cap",
          "args": {"line_id": "14", "cap_source": "engagement_letter"},
          "ledger_ids": ["led_aa12", "led_bb90"],
          "entailment": null
        }
      ]
    }
  ],
  "open_questions": [],
  "action_intents": [
    {
      "tool": "flag_invoice_line",
      "args": {"invoice_id": "INV-4421", "line_id": "14"},
      "requires_grant": true,
      "depends_on": ["c_19"]
    }
  ]
}

Notice what's absent: there's no summary field for the next agent to quote directly. Summaries are exactly where qualifiers and hedges get lost. If a downstream agent needs prose, it has to build that narrative purely from verified claims, and until a human signs off on it as customer-facing text, it's labeled Conjecture.

Also notice that entailment is set to null. The agent that authored the claim is not permitted to populate that field itself — only the verifier can. Letting an agent grade its own claims is like letting a business run its own compliance audit and calling it independent oversight.

Why "just add citations" still fails

The most common fake safeguard is output that merely looks cited. The model attaches a [1] or [2], sometimes even pointing at a real retrieved chunk. Then you open that chunk and discover it doesn't actually support the claim.

A citation by itself proves nothing. Real proof looks like: this specific span of text, this exact set of bytes, mapped to a specific proposition, carrying an entailment label, governed by an explicit policy for what happens when that label is neutral. Without that full chain, a "citation" is just a formatting choice dressed up as evidence.

The second fake safeguard is setting temperature to 0. That doesn't make output more truthful — it just makes a wrong answer consistent. A stable hallucination will sail through snapshot testing every time. It will not survive a properly maintained ledger.

What this costs, honestly

Getting a demo pipeline running in something like LangGraph takes a matter of days. Building the actual control plane underneath it — a tool registry, a citation ledger, schema gates, an NLI-based verifier, gold-trace evaluations, human review on every write-capable path, and logs detailed enough for a lawyer to follow — is what separates a working prototype from a system you can trust with real business decisions.

There's no single number that applies across the board. The real cost depends on how many of your tools trigger side effects, whether your source of truth is already structured and queryable, and how expensive an unsupported claim actually is in your specific domain. Legal billing and financial reporting sit at the high-cost, high-stakes end of that spectrum. A brainstorming tool built from a swarm of agents sits at the opposite end, and forcing it to carry this much infrastructure would be overkill.

The question worth asking first isn't which framework to build on — it's figuring out where your own workflow falls on that spectrum.

If this is the problem you actually have

This kind of control plane matters most for teams that can't tolerate an answer that sounds confident but is wrong: legal work, finance, and document-heavy operations generally. That means agent graphs built with tools like LangGraph, retrieval pipelines that are actually evaluated rather than assumed to work, and extraction systems that need to hold up under an audit.

If your own graph performs convincingly in a demo but starts producing false claims once it's live, the gap is almost always traceable to one of the six components covered here — and figuring out which one, and whether it's worth the engineering investment to fix, is the real starting point for scoping the work.