Tracing Which Hop Introduced a Hallucination With Hadith-Style Provenance

Who told the model that?

Say five agents handled a claim before it reached you, and the claim turns out to be wrong. Which of the five introduced it? Which one reworded it into something a little different? Who's accountable? In most production pipelines the honest answer is that nobody knows.

An LLM hands you an answer, and the question is where it came from. Not which document, but which transform. Every model output is an assertion with no provenance. Logs exist, but a log tells you what happened, not whether the transmission deserves trust. Multi-agent chains make it worse. A claim that survives five hand-offs isn't more reliable, it's just better hidden.

Where errors actually start

The common assumption is that hallucination compounds with depth: longer chain, more drift. A leaderboard built to test that idea (issue #71 in the project repo, with everything committed) found something else. On well-known facts the hallucination rate was 0%. On facts from after the model's training cutoff it landed between 44% and 56%. And it stayed flat as the chain got deeper.

So the error doesn't grow in transit. It's born at hop 1, at the moment of memory generation, when the model reaches for something it doesn't have. After that, the relay copies it faithfully and adds nothing. That shifts where instrumentation belongs. A drift detector isn't what you need. You need to know at which hop the claim originated and whether that narrator had any business making it.

Borrowing a method that's 1,400 years old

Islamic scholars faced the same problem with human claims fourteen centuries ago and built a whole discipline around it, hadith criticism. It runs three loops that stay deliberately apart:

• isnad is the chain: who passed the report to whom, in order.
• rijal covers the narrators: a registry that grades each transmitter's reliability by role, with the grade decaying as it ages.
• matn is the content: does the claim contradict what's already established?

The separation matters because a flawless chain can carry a false claim, and a true claim can arrive over a broken chain. The system won't squash those into one number, since they answer different questions.

The ISNAD library maps each loop onto LLM infrastructure:

• Chain (isnad): every transform in order, with input and output hashes.
• Registry (rijal): narrator reliability per role, decayed by freshness.
• Grading: chain quality is taken from the weakest narrator.
• Content criticism (matn): claim against evidence, designed to plug into your existing critic.
• Decision matrix: the outcome is serve, caveat, review or quarantine.
• Corroboration: independent witnesses (tawatur) and detection of errors shared between them.
• Grounding: scoped to the chain, meaning what this chain actually retrieved.
• Content-madar: fingerprints errors shared across sources that look independent.

Each claim goes through a 4×3 table, chain quality crossed with the content verdict, and leaves with exactly one action: serve, serve with a caveat, send to review, or quarantine. That's an action, not a score, and you can drop an action into a pipeline.

It ships as a library, a REST API and a CLI. It works alongside LangChain, LlamaIndex, CrewAI, LangGraph and OpenTelemetry instead of replacing them, so adopting it doesn't mean rewriting your stack.

What it won't do

Most reliability tooling skips this part, so it's worth stating plainly.

• It grades who, not whether. A sahih chain says the transmission is well attested. It never says the claim is true, and ISNAD doesn't pretend otherwise.
• Grades are ordinal: sahih ranks above hasan, then da'if, then mawdu'. There's no "92.3% confidence", because no honest method produces such a figure. Two decimal places on a trust value is decoration.
• It composes with critics. ISNAD is the transmission layer and deliberately isn't the judge of content.
• Everything is measured, not asserted. Each claim in the README traces to a committed, reproducible result, with negative controls printed beside it.
• Limits come first. Cold start is worse per role, integrity only holds within a domain, independence can be discounted but never proven, and the critic sets the ceiling on coverage. These sit in the README above the fold rather than buried in a footnote.

What the measurements show

Fingerprinting by content-madar cut the false-positive rate from 0.750 to 0.375, with token recall of 1.0 and near-miss detection at 4 of 4. In the chain-grounding evaluation, the separation between grounded and ungrounded claims held under an NLI critic, and a degeneracy guard stops the critic from gaming it.

The least comfortable result is that the critic is the ceiling. A stronger cross-model setup, deepseek-v4-pro judging deepseek-flash with the evidence right there in context, still missed up to 2 of 4 to 5 hallucinations. A critic doesn't replace adjudication, and ISNAD's docs say so openly instead of hoping you won't check.

Who should look at it

• Teams shipping multi-agent pipelines that want accountability per hop rather than a stack trace.
• Medical, legal and financial work, anywhere an audit trail matters more than a log file.
• Researchers after a preregistered, reproducible way to measure hallucination and provenance.
• Anyone tired of "trust us, it's fine" who'd rather read "here's who vouched for this, and what their vouching is worth."

Install it with:

pip install isnad          # Python ≥ 3.11, Apache-2.0
npm i isnad                # JS verifier — verifies audit records, grading stays in Python

The repo is at github.com/alizahidraja/isnad. It contains a one-command demo, the full benchmark harness and the ARCHITECTURE doc, which lays out the reasoning, the limits and the problems still open. Chain independence remains the biggest one.

ISNAD grades the transmission, not the truth, which is how reliability worked before machines existed, now applied to them.