Debug the Pipeline, Not the LLM: A Production Mental Model for RAG Failures

Imagine a support bot at a mid-size SaaS firm that one day answers two customers differently when both ask about the refund window.

An incident is opened, and the on-call engineer's first thought is that the LLM is hallucinating and a different model is needed. That's the junior diagnosis. It treats RAG as a black box, swaps the model and hopes the symptom disappears.

An experienced engineer begins elsewhere and asks which stage of the pipeline has failed. Saying the model hallucinates names a symptom and explains nothing.

Two days later the real bug turns up. The refund policy had been revised, yet nobody deleted the outdated PDF from the retrieval corpus, so both versions kept coming back with similar relevance scores, and the model was faithfully summarising conflicting context. It wasn't hallucinating at all. (This is a composite scenario for illustration, but variants of it are common.)

Here is the distance between the textbook version of RAG and the way production RAG breaks. Tutorials typically compress it to a single line:

Embed → Search → Generate

It's enough for a demo and not enough for production. A real system has several failure boundaries. The important engineering question isn't only whether the LLM can write a good answer, but whether every layer feeding that answer is working properly.

Ingestion: quality is settled before retrieval starts

Most RAG debugging begins at the wrong end. People tune prompts, switch embedding models and adjust top-k, when the actual fault was introduced weeks earlier during ingestion.

Parse by structure

Tables, code blocks, headings and lists carry meaning through their structure. Flatten a pricing table into plain prose and the link between a row and its column header can vanish. An ingestion pipeline should keep that structure and pick a representation suited to retrieval, and not treat every document as one undifferentiated block of text.

The right approach depends on the content. A table might call for structured extraction, a text serialisation, metadata, or a mix of lexical and semantic retrieval.

Chunk on meaning

The simple method cuts a chunk every 512 tokens. Yet token counts and meaning don't line up, so one chunk may start with a setup and stop partway through the point it was building toward.

Semantic chunking aims to split at topic shifts instead of counting tokens. It's not a universal upgrade and it costs more during ingestion. On lengthy material like policies, contracts and technical documentation, keeping ideas whole often improves what retrieval returns.

Give each chunk some context before embedding

A chunk lifted from page 40 of a 60-page document may be perfectly valid text and still be ambiguous outside its home. Take a sentence saying that a limit applies only to Enterprise customers. Without the surrounding section, nothing tells the retriever what "this limit" refers to.

Contextual retrieval handles that by prepending document-level context before the chunk is embedded: the title, section path, product, version, or a short generated note on where the chunk sits. The aim is for each chunk to keep the context that chunking would otherwise strip away.

Attach metadata

Treat metadata as functional data. Later steps need source, date, document type, tenant, permissions and version for filtering, access control, freshness checks and debugging.

That brings up one of the most important production questions in ingestion: what happens when a source document changes tomorrow? If the pipeline can only reindex everything, a small content edit becomes an operational burden. Ingestion should ideally be incremental and idempotent, so an update is detected, processed and propagated without rebuilding the whole corpus.

In the opening story, the old refund policy was never an LLM fault. It came down to how ingestion and the corpus were managed.

Retrieval is a pipeline and not a lookup

The word suggests a single step. In practice retrieval is a chain of choices, and the sequence has consequences.

1. Rewrite

People don't naturally write search queries. They ask something like whether the refund thing shipped. An LLM can turn that into something more useful for retrieval, perhaps folding in conversation history or splitting a complex question into narrower ones. Rewriting isn't free, though: a bad rewrite can drop key details or add assumptions. Treat it as one more component to evaluate and not as magic.

2. Scope before searching

This is among the most important production details. If the system is multi-tenant or permission-aware, the search space must respect those boundaries. Tenant, permissions, visibility, date and source should be applied inside the retrieval operation wherever the underlying system allows.

Filtering after a global top-k causes two problems. First, useful documents may already have been pushed out of the top-k by documents the user isn't permitted to see. Second, authorisation logic ends up dangerously detached from retrieval. Put simply, don't search everything and then ask what the user may see. Narrow the search first.

3. Search in two ways

Dense vector search handles semantic closeness well. Someone asking how to cancel can be matched to text saying customers may end their plan whenever they like, despite zero shared wording.

Embeddings are weaker with exact identifiers, though. Think of strings like these:

ERR_CONNECTION_RESET
SKU-48392
API-v3-beta

For such queries the literal token is the signal, and that's where lexical retrieval such as BM25 earns its place. Production retrieval often pairs dense search for meaning with lexical search for exact strings.

4. Fuse

Now there are two ranked lists. Dense search has its ordering, BM25 has its own, and their raw scores aren't necessarily comparable. Reciprocal Rank Fusion (RRF) offers a simple way to merge them using rank position:

RRF_score(d) = Σ 1 / (k + rank_i(d))

Each list contributes according to where a document sits in it, which avoids having to calibrate the score distributions of two unrelated systems against each other.

5. Rerank

Fusion gives a candidate set, not necessarily the final context. A cross-encoder reranker reads the query and each candidate together and produces a relevance score for that specific pair. That differs fundamentally from embedding query and document independently and comparing vectors.

The cost is compute, so you wouldn't run a cross-encoder across the whole corpus. Retrieval might return the top 20 or 50 candidates, and the reranker picks the strongest context from those.

The production mental model is therefore dense retrieval, then lexical retrieval, then fusion, then reranking. It isn't "vector database, done."

Generation: good context still doesn't ensure a good answer

Finding the right documents is necessary but not sufficient. The LLM still has to use them correctly.

Managing the context window

More context doesn't automatically yield a better answer. Models don't use long contexts evenly. Research has found that information in the middle of a long context can be recalled less reliably than information near the start or end. Pushing the top 20 chunks into the prompt can therefore make the answer worse. The target is maximum useful context and not maximum context.

Prompt boundaries and attribution

Spell out in the prompt how the retrieved passages should be used. The model has to treat them as evidence to cite, never as instructions to obey.

Citations help with trust and debugging too. When an answer states that Enterprise customers have 30 days to cancel, you need a way to trace that statement to a retrieved source.

Abstention

Abstention marks a major divide between demo and dependable system. If retrieval comes back empty-handed, a model will still try to help unless declining is an allowed outcome. A production system ought to be able to reply that the evidence isn't sufficient. That's often the right call and not a defect.

Without abstention, every retrieval failure can become a fluent hallucination. The LLM takes the blame for producing the wrong answer, though the real failure may have happened several layers earlier.

Evaluation comes before any fix

RAG has an awkward property: quality can fall while the application code stays untouched. A new embedding model, a different chunking method, documents added or removed, adjusted retrieval settings or an edited prompt can each change answers, even when yesterday's deployed code is byte-for-byte the same. Hence production RAG should be evaluated at multiple levels.

Retrieval quality. Context precision asks how much of the retrieved context was relevant. Context recall asks how much of the available relevant information retrieval actually found. Together they separate "we never found it" from "we found it but buried it in noise."

Generation quality. Faithfulness (also called groundedness) checks that the answer follows from the retrieved evidence, and citation accuracy checks that the named sources back up each claim. A well-written answer may still be ungrounded.

System health. Operational metrics count too: p95 latency, cost per query and abstention rate. They aren't merely infrastructure concerns. If latency doubles because an agent starts running several retrieval loops, that's part of the system's quality. A sudden rise in abstentions can mean retrieval has degraded. A jump in cost per query can mean a harmless-looking change introduced an expensive path.

Separating the measurements pays off in diagnosis. Healthy recall with falling groundedness points toward generation. Falling precision points toward ingestion, filters, retrieval or ranking. If you only track overall answer quality, every regression looks identical and gives you nothing to act on.

From pipeline to feedback loop

The core pipeline stays straightforward:

Ingestion → Retrieval → Generation → Evaluation

Plenty of systems need nothing more. One well-indexed knowledge base and clear questions rarely need an agent choosing the next move. Harder questions do exist, however.

Take a request to compare second-quarter revenue figures with what was reported in the board deck, then explain the difference. The answer may need the Q2 numbers, the board presentation, the relevant sections of each, a comparison, and possibly a query to another data source. One retrieval pass may not be enough, and that's where agentic RAG becomes useful. In place of a plain retrieve-then-generate step, you can run a loop:

Retrieve
   ↓
Evaluate
   ↓
Enough evidence?
   ↓ no
Retrieve again
   ↓
Generate

Such a system can conclude that the evidence so far is thin, reword the query, search again, or hand a sub-question to a SQL database, knowledge graph or live API. Retrieval turns iterative and stops being a single shot.

Agentic RAG isn't automatically better, though. Every extra loop adds latency, token cost and failure surface, and it can trigger needless searches when the agent has no clear reason to continue. It pays for its complexity when questions are ambiguous, multi-hop or need several sources. A question about the refund policy should probably be answered in one pass by a well-designed static pipeline. The board-deck comparison may justify iteration.

The useful question is whether the system has a measurable reason to look again, and not whether it should have an agent at all. Looping with no sound exit rule only adds delay and cost.

A mental model for production

Starting with the LLM is the commonest debugging error. A bad answer is merely the last symptom to show, and the cause may sit far upstream. Stale documents resemble hallucination, poor chunking resembles a retrieval miss, weak retrieval resembles a generation fault, and absent evaluation hides every one of them until customers complain.

So don't ask which model to use. Ask which layer is breaking.

INGESTION
     ↓
RETRIEVAL
     ↓
GENERATION
     ↓
EVALUATION
     ↖
      └── feedback loop / Agentic RAG

Stale data means fixing ingestion. A missing document means fixing retrieval. A document found but ranked low means fixing ranking. Correct context with a wrong answer means looking at generation. If you can't tell which one applies, build better evaluation first. And when a single pass can never suffice, consider an agentic loop.

Start with the pipeline and leave the model for last.

Where the ideas come from

For further reading, the original retrieval-augmented generation paper from Lewis and colleagues (NeurIPS 2020) defines the basic pattern. Anthropic's 2024 write-up on contextual retrieval reports large drops in retrieval failure from contextual embeddings, contextual BM25 and reranking, though those are the vendor's own measurements. Cormack, Clarke and Büttcher (SIGIR 2009) introduced Reciprocal Rank Fusion. The "Lost in the Middle" paper by Liu and colleagues studies how models use long contexts, the RAGAs paper describes automated RAG evaluation, and the Self-RAG paper by Asai and colleagues covers self-reflective, agentic retrieval.