Home / Articles

Seven Ways a RAG Answer Goes Wrong and How to Find the Failing Stage

A practical way to diagnose wrong RAG answers: seven failure points grouped into three stages, four fields to log, and one test that names the culprit.

Publisher: REACTAPP.TOP7 min read1670 words

This article is published in English.

Sources: The RAG Pipeline as Building Blocks: Seven Ways an Answer Goes Wrong

When a RAG system gives a wrong answer, the break happened at exactly one of seven places, and which one decides what to change. Tune a stage that was working and the answer stays just as wrong.

A request travels through parsing and chunking, embedding and indexing, retrieval, context assembly and generation. Any of them can fail, and the output reads the same whichever did. Without knowing the failing stage, a fix is a guess: a new chunk size, another embedding model or a reranker changes every query at once.

What follows is a diagnosis method. It names seven places an answer can break, groups them into the three stages that own the repair, and lists four things to log that tell them apart.

A bad answer hides its cause

Every one of the seven arrives as fluent, confident, wrong or incomplete prose. Nothing in the output points to a stage.

  • All seven look alike on arrival. A paragraph no one wrote, a chunk buried at rank eight and a model that read the wrong spreadsheet column each end up as one convincing reply.
  • Yet the remedies don't transfer. Ingestion decides what exists, retrieval what is found, assembly what is sent along, and the prompt what happens to it. Effort spent on the wrong layer changes nothing for the failing one.
  • Guessing costs more than the error did. Re-embedding a whole corpus to fix a formatting complaint takes days and leaves the complaint in place.

The better question to ask is where the answer text was last intact, and which stage let go of it.

Seven failures in three stages

The seven come from a published study of RAG systems in operation (arXiv 2401.05856), renamed here for what went wrong. They're grouped by where the repair lives, so locating a failure in a group already narrows the options.

Ingestion: missing content

The needed fact never turned into text the index could search. Maybe no source document contains it, or it vanished during parsing, cleaning or chunking.

  • Missing content. An internal wiki is asked about parental leave in Ireland. Only the US policy was ever written, and the system answers anyway.

Both causes produce the same symptom but need different repairs. If nobody ever recorded the fact, the system should be able to decline. If ingestion mangled it, look at the parser, the cleaning rules or a sync job that has quietly stopped.

Retrieval and context: not retrieved, cut before the prompt, wrong value read

The fact is sitting in a chunk somewhere, yet the model received something else.

  • Not retrieved. The chunk is in the index and search simply never surfaced it. Picture a torque value inside a spec table in a manual, outranked by a generic troubleshooting page.
  • Cut before the prompt. Search returned it, but it never made the context. Twelve support tickets match a query about failed imports and the token budget keeps five.
  • Wrong value read. The chunk was in the prompt beside a duplicate, an outdated version or heavy noise, and the model used the wrong source. A contract and its superseded amendment both appear, and the renewal date comes from the old one.

Order matters as well as fit. Research on long contexts (arXiv 2307.03172) shows models use material at the start and end better than the middle, so a chunk can reach the prompt and still be overlooked.

Why does wrong value read belong here? The model produced the error, so it looks like a generation problem. But the repair is nearly always upstream: remove the duplicate, drop the stale version, trim noise, reorder the context. Rewriting the prompt seldom helps.

Generation: wrong format, wrong scope, incomplete

The model saw all it needed and still produced a bad reply.

  • Wrong format. Correct facts in the wrong shape: a checklist was requested and a paragraph arrived.
  • Wrong scope. The reply is pitched too wide or too narrow. A question about a spending cap gets the staff-wide policy even though the asker is a freelancer with different rules.
  • Incomplete. One part of a multi-part question is skipped. Someone asks about four ticket queues and gets numbers for three.

Incomplete is the one people miss when reading, since nothing written is false. A citation audit catches it: if a source made it into the prompt but is never cited, something was left out. Because the context was sound, the repair lives in the prompt, the answer format or the interface.

How to diagnose

Log four fields per request

Diagnosis means rebuilding a request after the fact, from a complaint that shows up hours later. Re-asking by hand doesn't substitute, since the index has likely changed and the failure may not reproduce. Four fields are enough:

  • The raw question, before any rewrite, because rewriting is itself a failure point.
  • What search returned: every candidate chunk with its chunk_id, doc_id, version and score.
  • What the model was given: the chunk_id values that remained after the token budget was applied.
  • The final reply, character for character.
# The retrieved list and the prompt list stay separate. Collapsed into one,
# a chunk lost at ranking and a chunk lost at the token budget leave the
# same record, and no later reading of the log can separate them.
log_request({
    "question": question,
    "retrieved": [
        {"chunk_id": c.chunk_id, "doc_id": c.doc_id,
         "version": c.version, "score": c.score}
        for c in candidates
    ],
    "in_prompt": [c.chunk_id for c in context],
    "answer": answer,
})

Keep the retrieved and prompt lists separate. A log with only the rendered prompt and the answer can't distinguish a ranking failure from a budget failure.

One test, three outcomes

Pick the passage that ought to have shaped the reply. Look for it in the logged record and note how deep into the pipeline it got. There are only three stopping points, and each maps to one stage.

  • Found in no chunk: this is ingestion's problem. The corpus lacks it, or it was lost before indexing.
  • The text is in a chunk that never reached the prompt. Retrieval and context own it: ranking didn't return the chunk, or the budget cut it.
  • Found in the prompt, answer still wrong: the model had everything. If it picked a wrong value, look upstream for duplicates or noise. If the form, scope or completeness is off, look at generation.

What the seven don't cover

These seven judge whether one answer was right. Several things that make a system unshippable are invisible to them.

  • Permissions. An answer quoting a document the asker can't read is correct and unacceptable. It needs acl_principals stored beside each chunk and a filter at query time.
  • Poisoned documents. A file containing instructions aimed at the model can pass every one of the seven checks, since the reply remains faithful to what it was given.
  • Tenancy. Customer A's chunks showing up in customer B's reply is fixed by a tenant_id on stored chunks and on every query, never by prompt wording.
  • Scale and cost. Latency and spend per query at large corpus size don't appear anywhere in the seven.

All seven can pass while data still leaks. This framework explains wrong answers; it is silent on safety.

Pitfalls

  • Widening retrieval for an answer whose chunk already reached the prompt. More results mean more noise, which makes a wrong value read likelier.
  • Logging only the rendered prompt. Not retrieved and cut before the prompt look identical, so you tune search when the budget was at fault, or the reverse.
  • Using refusal as the cure. If the answer lives in a table the parser flattened, declining to answer only hides an ingestion bug behind good manners.
  • Adding chunk metadata after launch. Introducing a version or section path later rewrites every chunk, and re-embedding is the costly half.
  • Arguing about scope by taste. Whether a reply is too general stays a debate until a test question carries the exact answer it should have produced.

Putting it to work

Think of the pipeline as a chain: a wrong answer means a link failed, and diagnosis is locating the last link where the answer text survived.

  • Start logging the four fields before touching any setting. One record per request ends most debates before they begin.
  • Run the single test first: is the passage in no chunk, in a chunk that missed the prompt, or in the prompt itself? That answer picks the stage and so the type of fix likely to work.
  • Give the system a way to say it doesn't know. A test set where every question has an answer will never reveal missing content.
  • Store doc_id, version, section_path and updated_at on each chunk from the first ingest, adding tenant_id and acl_principals if access matters.
  • Defer the rest until a specific failure demands it. Graph, multimodal and agentic retrieval each target one flavour of not-retrieved, and none should be the default.

The stages differ with your retrieval architecture, but the four logged fields stay the same in all of them.