Why RAG Exists: Parametric Memory, Dense Retrieval and the 2020 Paper
Why RAG exists: the limits of parametric memory, how BM25 and embeddings led to dense retrieval, and what RAG-Sequence and RAG-Token trade off in the paper.
This article is published in English.
Sources: What Is RAG? The Story Behind Retrieval-Augmented Generation and Why It Changed AI Forever
There isn't one right way to build a RAG system. Each business problem needs its own architecture to scale, and every retrieval strategy has trade-offs, so a design that suits a support bot may fail in legal research. The field also moves quickly. In 2020, when Retrieval-Augmented Generation first appeared, language models were much weaker than now. Since then reasoning, instruction following, multimodal understanding, context windows and tool use have all improved a great deal. RAG still stays a core technique for dependable AI applications, because even the strongest model can't know everything or always see the latest information. Retrieving relevant knowledge from outside sources reduces hallucination and keeps answers anchored in trustworthy material.
Today the technique sits behind support platforms and enterprise chatbots, coding assistants, legal research tools, clinical applications, finance systems and internal knowledge helpers. Even so, many tutorials leap to vector databases, embeddings or LangChain code and never ask why RAG was invented in the first place. Answering that starts with the original research and moves on to the engineering ideas behind modern production systems, including why design choices were made, what they cost and how they changed.
The limit of what a model knows
When GPT-style models first showed striking language understanding, many people assumed they could answer nearly anything. To a degree they could. Training on huge datasets taught them facts about history, science, programming, mathematics and literature. But everything the model "knew" was compressed into its parameters during training, and when training ended that knowledge froze.
People work somewhat similarly. Experts don't rely on memory alone. A cardiologist who meets a rare disease or a newly approved treatment will check current clinical guidelines or the medical literature first. A seasoned Python developer asked to move a large application to Java doesn't automatically know every Java framework or best practice for doing it without hurting performance or user experience. They read the Java documentation, pick a framework such as Spring based on the scope of the app, and keep consulting migration guides and project documentation throughout. The skill is already there, and outside material supplies the context for solving the problem accurately.
Language models face the same issue. Their general knowledge is strong, but ask about something after their training cutoff, an internal company document or an organization's newest policy, and they have nothing to draw on. Worse, where a person would admit uncertainty, a model often produces a fluent, confident and factually wrong reply, now widely called hallucination.
Retraining whenever new information appeared wasn't realistic either, since frontier models demand enormous compute, money and time. Researchers needed a way for a model to read relevant outside information at the moment of answering, with no retraining. That need produced Retrieval-Augmented Generation and laid groundwork for many of today's context engineering techniques, which aim to give a model the right information at the right time.
Two kinds of memory
To see why RAG mattered, it helps to separate two memory types. Parametric memory is knowledge held in a model's learned parameters: patterns, concepts, language ability and facts picked up in training. It has real limits:
- It can't be updated easily.
- It may be stale.
- Rare facts can fade.
- No model can memorize every document ever written.
RAG added non-parametric memory: knowledge kept outside the model, in documents, databases or an indexed corpus that gets searched while the model is answering. Instead of leaning only on what was memorized months earlier, the model fetches relevant material on demand before writing a response. That plain idea changed how AI systems could handle knowledge that keeps changing.
Retrieval before language models
Retrieval predates large language models. Search engines long faced a similar problem: given millions of documents, which are most relevant to a user's query? Early systems matched words, counting how often query terms appeared in a document. TF-IDF (Term Frequency-Inverse Document Frequency) improved on that by weighting informative words more and common ones less. BM25 later became the industry standard for ranking, with a more refined scoring function that accounts for both term frequency and document length, and it's still a key part of many production search systems.
These approaches do well when the exact keywords appear in a document. They struggle when two passages express the same idea in different words, and that gap paved the way for semantic retrieval.
As neural networks improved, researchers began representing text as dense vectors, known as embeddings. An embedding is a numeric representation of a word, sentence or document in a high-dimensional space, often called the embedding or latent space. Individual numbers mean little alone. Together they capture meaning, so texts with similar meaning sit close together and unrelated concepts lie far apart. Because of that clustering, retrieval can work on meaning and not on exact keyword matches.
Take a user asking how to improve a neural network's performance. With embeddings, that query can still find a document titled "Techniques for optimizing deep learning models," although neither "neural network" nor "performance" appears in it. After both queries and documents are embedded, retrieval becomes a search for the document vectors closest to the query vector. That idea underpinned Dense Passage Retrieval (DPR), introduced in 2020. DPR showed dense semantic retrieval beating traditional keyword retrieval on many open-domain question answering benchmarks, and it became one of the building blocks that made RAG possible.
From finding documents to answering questions
Semantic retrieval covered only half of the task. People don't want a list of documents, they want answers, and that brings in open-domain question answering. A closed-domain system draws answers from a fixed knowledge base or dataset. Open-domain QA must search millions of documents for useful material before answering. Given a question such as who won the 2024 Nobel Prize in Physics, the system first scans a huge collection for the best evidence and then writes its response from that.
That yields a two-stage pipeline:
- Retrieve the most relevant documents.
- Generate an answer from those documents.
This retrieve-then-generate pattern is the conceptual root of modern RAG. The 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks formalized it by joining a dense retriever and a sequence generation model in one end-to-end architecture. Its central idea was neat: don't ask a language model to answer purely from its internal parameters. First fetch the most relevant external documents, then condition generation on those passages. Retrieval didn't replace language models. It strengthened them, producing better factual accuracy, stronger results on knowledge-intensive tasks, and knowledge that can be refreshed by swapping the document collection and not retraining the model.
RAG-Sequence versus RAG-Token
The paper proposed two retrieval strategies. RAG-Sequence fetches a fixed set of relevant documents once, before generation begins, and they stay the same for the whole response. It's computationally efficient and simple to implement, and it still underlies most production RAG systems.
RAG-Token is more dynamic. The model can retrieve different documents as it generates different parts of the response, which in theory lets it gather fresh information whenever it needs more context and could yield better-informed answers.
Picture writing a research report. With RAG-Sequence all the reference material is gathered before the first sentence. With RAG-Token the writer stops mid-report to look something up whenever a fresh question arises. The second approach is more flexible, but it demands many more retrieval operations, costs much more compute and adds latency. That's why most production systems choose the simpler RAG-Sequence design and spend their effort on the retrieval pipeline itself, using hybrid search, reranking, query rewriting and metadata filtering.
Where the original design fell short
Revolutionary as it was, the original architecture had flaws that showed up once practitioners deployed it in real settings:
- Generation quality was capped by retrieval quality.
- Bad chunking hurt retrieval accuracy.
- Context windows limited how much could be supplied.
- Dense retrieval sometimes missed important exact-term matches.
- Retrieval ran before generation, which added latency.
- Support for structured, tabular and multimodal data was limited.
Those problems fed years of follow-up research, and the last five years have reshaped RAG. Language models are far more capable, embedding models have improved and vector databases have matured. Hybrid search blends lexical and semantic retrieval, and cross-encoder rerankers sharpen document choice. Newer systems add metadata filtering, query rewriting, context compression, adaptive retrieval and agentic workflows, while researchers have proposed approaches such as GraphRAG, Self-RAG, Corrective RAG (CRAG), RAPTOR and multimodal retrieval. These additions repair much of what the early pipeline got wrong, but none replaces the retrieve-first step at its center.
Today's production architectures look very unlike the simple 2020 pipeline, yet they share its founding principle: retrieve relevant knowledge before generating an answer.
The shift in thinking
What stands out most is that RAG didn't mainly make language models smarter. It changed how people use them. Instead of expecting a model to memorize an ever-growing pile of knowledge, RAG treats outside information as something to fetch whenever it's needed. That shift made it possible to build systems that are more accurate, more reliable and able to cope with constantly changing information.
Knowing RAG's origins is more than a history lesson. It explains why modern systems look the way they do, and why particular architectural decisions matter once they're in production. A team that understands the original motivation can tell which new techniques address a real weakness and which only add complexity.