Diagnosing RAG Failures: A Stage Map and a Claim-Level Groundedness Judge

When a RAG bot starts answering wrong

Deploying a RAG app doesn't finish the job, least of all in production. Eventually it begins answering wrongly, and the first challenge is finding the cause. The system has at least seven spots where silent breakage can happen: chunking, embeddings, retrieval, reranking, context assembly, generation, and the prompt that binds them. A bad final answer reveals almost nothing about which one failed. Some engineers have burned a week refining a generation prompt when a chunking bug was behind it.

There's a fairly mechanical way to localize a failure. After that comes a second problem, measuring groundedness at scale, since reading every transcript stops being feasible past the proof-of-concept stage. An LLM acting as a judge can do that job, but judges have failure modes of their own, which are as easy to miss as the RAG faults they're meant to expose. Both halves are covered below: tracing failures to their source, and building a groundedness judge that deserves trust.

Find the broken link first

A RAG pipeline is a chain, and the first task is to identify which link failed before anyone edits a prompt. The map below is a framework for localizing the problem. It isn't exhaustive, and it can be extended to match a particular system and use case.

The ordering is deliberate: retrieval gets fixed before generation. Prompt engineering can't repair a context window that never held the right fact. Teams often work backwards, rewriting the system prompt five times while the reranker demoted the one passage that counted. Inexpensive fixes with big payoffs, such as better chunking or hybrid search, belong ahead of costly ones like fine-tuning a reranker or retraining anything.

Three realistic symptoms show how the map gets used:

1. Answers come out confident even though the documents never touch the topic. This isn't retrieval, since nothing relevant exists to fetch. It's a generation or abstention failure: the model won't treat "I don't have this" as an acceptable reply.
2. The bot gives plausible but outdated information. Chunking is almost never the culprit. The index is stale, because the documents changed and nobody re-embedded them.
3. Single-fact questions pass, yet anything requiring two facts from two documents collapses. This is a retrieval failure particular to multi-hop queries, because one dense retrieval pass can't fetch two pieces that each look unlike the query.

The move that follows is to pull the actually retrieved context for a failing example and read it before changing anything. If the fact isn't there, retrieval is the problem. If it is there and the model still gets it wrong, generation is the problem. That one check can save days.

What groundedness means

Reading transcripts is fine for a handful of cases and hopeless at thousands. An LLM judge steps in here, specifically one for groundedness, also known as faithfulness. Its question is whether each claim in the answer can be traced to something in the retrieved context.

One subtle point trips people at once: groundedness and correctness are separate. A true answer can be ungrounded, say when the model knew the fact from training and not from the documents. A grounded answer can be false, if the retrieved passage was itself stale or wrong. When evaluating RAG, groundedness is the measure to want, since it reveals whether retrieval did its work whatever the quality of the underlying documents.

A claim-level judge

A single prompt that asks for a groundedness rating from 1 to 5 sounds reasonable and tends to be close to useless. It yields a number with no explanation, and whole-answer scoring is exactly where LLMs are inconsistent. Breaking the answer into atomic claims and checking each one works far better.

The starting prompt has a system role that casts the model as a groundedness evaluator, supplies a question, the retrieved context and the generated answer, and says up front that the task isn't to decide whether the answer is true in general. It then asks for claims to be extracted and checked one by one against the context, with a verdict and a supporting span for each:

SYSTEM:
You are a groundedness evaluator. You will be given a QUESTION, a CONTEXT
(retrieved passages), and an ANSWER generated by an AI assistant. Your job
is NOT to judge whether the answer is true in general, or well-written, or
relevant — only whether it is fully supported by the CONTEXT.

Do the following:

1. Decompose the ANSWER into a list of atomic claims (each claim = one
   discrete factual statement, not a compound sentence).
2. For each claim, decide:
   - SUPPORTED: directly stated or reasonably entailed by CONTEXT
   - CONTRADICTED: conflicts with CONTEXT
   - UNSUPPORTED: not addressed by CONTEXT at all (NOT the same as "false")
3. Quote the exact span of CONTEXT used as evidence for each claim
   (or "none" if UNSUPPORTED/CONTRADICTED).
4. Give a one-sentence rationale per claim.
5. groundedness_score = (# SUPPORTED) / (# total claims).

Be strict: a claim that adds specificity not in CONTEXT (context says "a
discount applies," answer says "a 20% discount applies") is UNSUPPORTED.

Return valid JSON only:
{
  "claims": [{"claim": "...", "verdict": "...", "evidence_span": "...", "rationale": "..."}],
  "groundedness_score": 0.0,
  "num_claims": 0
}

QUESTION: {question}
CONTEXT: {context}
ANSWER: {answer}

Some design choices need to be explicit, because they aren't obvious at first:

• Claim-level scoring. One overall score is hard to debug. Per-claim verdicts show exactly which statement failed, which matters when someone has to fix the underlying cause.
• A mandatory evidence span. Requiring the judge to cite supporting text pushes it to base verdicts on evidence and not on plausibility, and it lets a human reviewer check a judgment quickly without rereading the whole context.
• UNSUPPORTED versus CONTRADICTED. Contradicting the evidence and merely lacking support are two distinct failures with different remedies. Folding both into a single "bad" label throws diagnostic detail away.
• An explicit instruction to be strict. LLM judges tend toward leniency on borderline cases, especially when an answer sounds plausible, and asking for strictness reduces that.

Validate the judge against people

This is the step that's always skipped, and it decides whether any of the rest has value: a judge must be validated against humans before it's trusted.

The process is simple. Take 100 to 200 examples, have a person rate groundedness independently, and compute agreement with Cohen's kappa. Raw accuracy misleads when most answers are grounded, because a judge that always says "grounded" scores well and is worthless. Also check for the usual judge biases:

• Position bias: does the verdict change if the context passages are reordered?
• Verbosity bias: does the judge favor longer answers whether or not they're actually better grounded?
• Self-preference bias: if the judge belongs to the same model family as the generator, it may rate that family's outputs more kindly.
• Leniency drift: does agreement erode quietly over weeks as the judge model or the prompt version changes?

Weak kappa in one category is a sign to fix the rubric for that category. It doesn't show that the RAG system is broken there. People confuse the two constantly, and the next section turns on that distinction.

Reading a dashboard like an operator

Imagine a dashboard broken out by query type. The right posture is that of an operator, not a scoreboard watcher.

The overall average shows a mild dip, which may be misleading. Row by row, the reading goes like this.

1. Simple factual lookup holds steady and high for groundedness and agreement alike. System and judge both find it easy. Alone it's boring, but a fall would signal that something basic broke: indexing, base retrieval or the judge prompt, and not reasoning or ambiguity.
2. Multi-hop and comparison dropped sharply. With κ=0.71, agreement is merely moderate, so judge noise on long answers that resist decomposition may explain part of it. Before declaring a regression, pull five failing transcripts and see whether both passages the answer needs sit in the context. When only one arrived, that's a retrieval-recall issue, as single-shot dense retrieval often misses two facts that don't each resemble the query, and the cure lies in query decomposition or multi-query retrieval and not in prompt tweaks. When both arrived and the comparison still fails, synthesis is to blame, with the lost-in-the-middle effect a prime suspect: the model leans on the first passage and gives the second too little weight.
3. Numeric and financial figures show κ=0.52. That's weak agreement and the loudest signal in the table, because it means the judge can't yet be trusted here. Numeric equivalence is truly hard for an LLM judge: "20%," "roughly a fifth" and "$20 off a $100 item" are one claim in different clothes. The sensible instinct is to leave the RAG system alone until the rubric is fixed, since chasing an untrusted number wastes engineering time.
4. Policy and procedural is flat and healthy like the first row, for a different reason: answers usually come from one well-structured document and need no synthesis across several. If it dips, reasoning isn't the first suspect. A numbered list split across chunk boundaries is likelier, leaving the model with steps 1 to 3 and inventing 4 and 5.
5. Out-of-scope deserves the first fix, even though n=20 makes it the smallest segment. With κ=0.90 the judge is very reliable here, because spotting an invented answer that has no supporting context is far easier than judging subtle numeric or multi-hop claims. That reliability makes the decline credible, and it's the most dangerous failure in the table. The system moved from correctly admitting "I don't know" to making things up with confidence, which is worse than a wrong figure since it's indistinguishable in tone from a correct answer. Two likely causes: retrieval is returning near-miss passages that the model takes as permission to answer, or a recent prompt or model change silently removed the abstention instruction. Start with the retrieval similarity scores for this segment. If they hover near the noise floor and the model answers anyway, a relevance-threshold gate before generation is missing.

The method in five moves

The whole exercise reduces to this:

• don't trust the aggregate
• check judge reliability before trusting the score
• segment by category
• hypothesize from actual evidence which stage broke
• prioritize by severity times confidence, and not by which bar moved most

Practical notes for new evaluators

1. Each judged response costs an extra LLM call. At real volume, judge a sample instead of everything, and stratify it by query category so rare but critical segments like out-of-scope aren't swamped by plentiful easy queries.
2. Re-validate kappa whenever the judge model or the judge prompt changes. Either change creates a new judge.
3. A judge tuned to one domain's numeric or jargon quirks doesn't automatically transfer to another, so calibrate per domain and not once globally.

All examples and figures here are illustrative and not taken from a live production system. The framework and prompt structure are a reasonable starting point.

Where this stops

The material above spans a map for localizing failures, the gap between groundedness and correctness, a claim-level judge, and the calibration step many skip. One limit remains: it all presumes one clean block of retrieved context. That presumption fails once a system calls tools, chains retrieval steps or reasons over its own intermediate outputs, as agentic systems do. Defining groundedness without a single context block to check against, and reshaping the judge accordingly, is a separate problem.