A Field Map of RAG: From First Principles to Production Failures

Retrieval-augmented generation is simple to show off and hard to get right. A working prototype fits in forty lines and an afternoon. Then comes the long stretch of working out why it answers wrongly three times in ten, or why a chatbot confidently quotes a contract clause that was never written. This roadmap covers both ends: it starts with no jargon at all and ends at the techniques that separate a weekend demo from a system people trust with money.

The idea in plain terms

Picture a language model as a well-read graduate who has never seen your files. It has absorbed much of the public internet, yet nothing from your rental agreements, HR policy, product catalogue or last quarter's board deck.

RAG converts a closed-book exam into an open-book one. Instead of trusting the model to have memorized your data, the system first looks up the relevant pages, hands them over and instructs the model to answer from those alone. The name lists three steps:

• Retrieve: search your documents and pull the few passages that matter.
• Augment: place those passages into the prompt next to the question.
• Generate: have the model write an answer grounded in what it was handed.

That's the whole idea. Every later refinement, from rerankers to graphs, only exists to make one of those three steps less bad.

For non-technical readers, the payoff is being able to explain what an engineering team is building, why it fails the way it does and which questions to ask during a vendor demo. For engineers, it's the mechanics, trade-offs and failure modes, which means asking why cosine similarity is an odd stand-in for relevance, why 512-token chunks wreck tables and why an evaluation set counts for more than the choice of model. Nobody gets "just use a vector database" as a complete answer.

Two pipelines, not one

A structural point trips up nearly everyone. A RAG system isn't a single pipeline. It's two pipelines that never run at the same moment.

Lane A is indexing. It's done ahead of time, typically as a batch job. Documents are split into pieces, each piece is converted into a list of numbers capturing its meaning, and the numbers go into a searchable index. It's the dull lane, and it's where roughly 80% of final answer quality is decided.

Lane B is querying. It runs live whenever a user asks something. The question is converted to numbers by exactly the same method, the closest stored pieces are found, they're stuffed into a prompt and an answer comes back.

If the two lanes disagree on encoding, nothing works, rather like filing papers alphabetically and then searching by color. People tend to fixate on Lane B, meaning the prompt, the model and the temperature, while most of the trouble sits in Lane A. Indexing runs once, querying runs every time, and most bugs hide in the gap between them.

The stages, from basics to production

Foundations

The first topic is where hallucination actually comes from, which isn't the model being dumb. Related ideas include what a context window is, why an entire database can't just be pasted into one and what a knowledge cutoff means. Then there's the question people get wrong: RAG, fine-tuning and a bigger context window are three separate tools for three separate problems, yet they're constantly recommended for one another's jobs.

An insurer whose policy wording changes every quarter shows the difference. Fine-tuning would mean retraining each quarter. RAG means updating a folder.

Ingestion and chunking

Getting documents in is its own challenge: PDFs that are really scanned images, tables, headers, footers and a 40-page appendix nobody needs. Then comes chunking, the decision of how to slice a document. It sounds trivial and isn't. Slice too small and a chunk reads "the deposit shall be refundable" with no hint of the conditions. Slice too large and the model drowns in noise. The main methods are fixed-size, sentence-aware and recursive splitting, plus an overlap trick that stops sentences being guillotined.

Take a 60-page lease whose penalty clause depends on a definition given 40 pages earlier. Chunk boundaries decide whether that link survives.

Embeddings and vector search

This stage feels like magic and isn't. Text gets turned into a list of numbers, arranged so that a phrase about cancelling an order lands close to one about returning a purchase and far from one about ordering more stock. Related ideas include dimensions, cosine similarity, and the fact that closeness in vector space doesn't guarantee an answer.

Storage comes next: what a vector database really does, why brute-force search breaks down at scale and what approximate nearest-neighbor search sacrifices for speed.

The retrieval loop

Here the pieces are assembled. Choosing top-k, the number of chunks to fetch, is part of it, and more isn't better. Metadata filtering lets a search cover only 2024 documents, or only the Mumbai region. Prompt assembly matters too, since how retrieved context is arranged, and in what order, changes the answer.

Grounding and citations follow. The model should point to something like Section 4.2 and not merely sound sure of itself. Just as vital is getting it to state that the answer isn't in the documents, probably the most underrated feature of any RAG system.

Evaluation

Almost everyone skips this stage, which explains why almost everyone ships broken RAG. A problem that isn't measured can't be fixed. Quality divides into two questions: was the right material retrieved, and did the model stay faithful to it? Each fails on its own and calls for its own repair. The toolkit includes a golden set of test questions taken from actual users, retrieval scored with precision and recall, generation scored on faithfulness and relevance to the question, and an LLM acting as judge, with an eye on where that approach goes wrong.

One system scored 90% on retrieval and 55% on faithfulness. Its search was fine and its answers were still wrong.

Upgrades, each tied to a failure

Better retrieval

Hybrid search combines meaning-based lookup with old-fashioned keyword matching, because embeddings handle exact identifiers such as model numbers and invoice IDs terribly. Reranking fetches 50 candidates cheaply, then lets a slower, smarter model choose the best 5. Query transformation rewrites a vague question into a searchable one, splits a compound question into parts, or uses HyDE, which generates a fake ideal answer and searches with that in place of the question.

A request to compare two Bangalore projects on possession date and price is a single question needing three separate lookups.

Better indexing

Fixing Lane A properly involves several techniques. Parent-child retrieval searches small, precise chunks but feeds the model the larger section around them. Semantic chunking splits where the topic shifts and not at an arbitrary character count. Summary and hierarchical indexes search document summaries first and then drill down. Graph RAG stores relationships between entities and not only text, so it can answer questions like which vendors are also customers.

A compliance query is a good case: it needs a rule, its 2023 amendment and an exemption note, three documents for one answer.

Agentic behavior

Past this point RAG is no longer a fixed pipeline and begins to make choices. Routing directs a question to the proper index, to a SQL database or to no retrieval at all. Multi-hop retrieval reads, spots a gap and fetches again. Self-RAG and Corrective RAG let the system score its own retrieved context and throw it out if it's irrelevant, then fall back on a web search or an honest admission that it has no answer.

Take a question about why a claim was rejected. Answering it takes the policy, the claim file and the rejection code, retrieved one after another.

Messy documents

Real business knowledge seldom shows up as tidy paragraphs. It arrives as spreadsheets, scans, floor plans, screenshots and diagrams. Skills in play include layout-aware parsing, table extraction that copes with merged cells, image embeddings, and retrieval over documents where a chart holds the answer. The textbook case is a pricing question whose number lives only in a scanned rate card.

Production concerns

Everything breaks after the demo. Latency budgets show where the milliseconds go. Cost per query matters, along with caching that genuinely helps. The index has to stay fresh when documents change hourly. Access control must guarantee that user A can't retrieve user B's chunks, a common and real leak. Poisoned documents can inject prompts. Monitoring and logging reveal when quality has quietly degraded. A system that ran beautifully on 200 documents can fall over at 200,000.

Learn the failure first

If a single point sticks, make it this one: advanced RAG isn't a more sophisticated edition of basic RAG. It's a set of patches, each for a specific failure with a name. Once the failure is understood, the matching technique is the obvious step.

Three principles keep the approach honest:

• No framework worship. Concepts come before tools, and anyone who understands chunking can implement it anywhere.
• No unexplained copy-paste code. Code shows up only after the reason it looks that way is clear.
• No pretending it always works. Some cases make RAG the wrong answer, and something simpler wins.

Newcomers should take the topics in order, since each assumes the one before. Anyone with a running system that disappoints can jump straight to the evaluation stage.