A Basic RAG Pipeline Explained: From Raw Documents to Grounded Answers

Language models have an awkward habit: they sound sure of themselves even when they don't have the facts. Ask about an internal company policy, the manual for a product released last month, an obscure legal document or yesterday's support ticket, and the model may simply not know. Worse, it may produce something plausible-sounding to cover the hole.

Retrieval-augmented generation, usually shortened to RAG, is the standard remedy. A RAG pipeline lets a model consult outside knowledge at the moment it's needed. Not everything has to live in the model's memory, because the system searches a trusted knowledge base, pulls out the most relevant passages and writes an answer grounded in them. Put simply, a closed-book assistant becomes an open-book one.

The approach gained attention through a 2020 research paper on retrieval-augmented generation for knowledge-intensive NLP tasks, which paired a generative model with a retriever searching an external document collection. The central observation was that a model's built-in knowledge and external information can complement each other. The model supplies the language ability, and the retriever supplies current, specific details.

A basic pipeline runs in two phases. Indexing gets documents ready for search. Answering takes a user's question, fetches relevant material and hands it to the model as context. That reads as simple, yet every stage hides decisions that shape the result. A weak system rarely fails because the model itself is poor. Far more often it retrieves the wrong documents, the chunks are messy, the prompt is vague, or the system is unable to admit it doesn't have the answer.

The sections below build the idea from the ground up.

What RAG does in practice

The basic loop has four moves. A user asks something. The system searches the knowledge base. The best-matching passages go into the model's prompt. The model answers using that retrieved context.

Picture a customer who wants to know whether a refurbished laptop can still go back after 20 days. A plain chatbot might guess from general retail habits. A RAG chatbot can look up the real returns policy, find the section on refurbished products and answer from it. If the policy sets a 14-day window for refurbished laptops, the bot can confirm that. If the policy is silent about refurbished laptops, the bot shouldn't invent a rule.

That's the heart of it: retrieval first, generation second. Retrieval narrows the world to a few useful pieces of evidence, and generation turns them into a helpful reply. Done well, RAG improves factual accuracy, makes answers easier to check, and lets you update what the system knows without retraining a model.

Choosing the knowledge source

Every pipeline starts with a body of material to retrieve from. Typical candidates are company documentation, product manuals, help-centre articles, PDFs, contracts, research papers, meeting notes, database records, code docs and support tickets.

The biggest source isn't always the best one. A focused, tidy and current set of documents usually beats a large heap of stale files. RAG can't fix bad source material, so outdated, unclear or contradictory documents produce outdated, unclear or contradictory answers.

Before indexing anything, it helps to ask some practical questions:

• Is the information accurate and up to date?
• Who owns it?
• Is every user allowed to see it?
• Are there duplicate or conflicting versions?
• Do tables, images, footnotes or legal clauses need special handling?

A small, well-kept knowledge base is a good start. For a first project, choose one clear use case, such as questions about an employee handbook or a product FAQ, and expand once that works.

Cleaning and preparing documents

Raw files are seldom ready for retrieval. PDFs may have broken line breaks. Web pages bring along menus, cookie banners, adverts and footers that repeat on every page. Internal documents may have headings with too little context, and spreadsheets may hide key details in columns that read poorly as plain text.

Cleaning turns that mess into searchable text. It can involve stripping boilerplate, keeping headings, extracting tables, converting everything to a consistent format and attaching metadata. Metadata is especially useful, because a chunk gains a lot when it carries fields like its document's title, author, date, owning department, product version, link, access level or section heading.

Compare a bare chunk such as "Returns must be requested within 14 days" with one that comes labelled: the title is the refurbished laptop return policy, the section is the return window, it was last updated in March 2026, and the body says a refurbished laptop return has to be requested inside 14 days of arrival. That second form gives both the retriever and the model more to work with, and it makes citations and filtering simpler later.

Splitting documents into chunks

Language models and retrieval systems generally don't handle whole books, manuals or policy libraries in one piece. They cope better with smaller passages called chunks.

Chunking is among the most consequential parts of the pipeline, because retrieval works at chunk level. If chunks are too big, the answer can be buried among irrelevant text. If they're too small, the system may retrieve fragments without enough context to answer properly. A good chunk is large enough to hold a complete idea and small enough to stay focused.

For text-heavy systems, developers often begin with chunks of a few hundred words and adjust according to results. Some use overlapping chunks so context isn't lost at a boundary, for example where a heading closes one chunk and the policy text it introduces opens the next.

Chunking should follow document structure where possible. Splitting by section, heading, paragraph or question-and-answer pair usually beats cutting at a fixed character count, and different document types, such as refund policies, troubleshooting guides and scientific papers, may each need their own approach.

This is where many basic systems quietly succeed or fail. The model can answer only from what it receives, so a chopped-up fragment with no context can trip up even a strong model.

Creating embeddings

After cleaning and chunking, each chunk is converted into an embedding, a numerical representation of meaning. Technically it's a vector, a list of numbers capturing semantic information about the text. Texts with similar meanings tend to land close together in vector space, which lets a system search by meaning and not only by exact keyword.

Now suppose the question is how long someone has to send back a refurbished computer, while the document states that refurbished laptops can be returned if requested within 14 days after delivery. A traditional keyword search might miss the link between "send back" and "returns," or between "computer" and "laptop." Embedding-based search is more likely to see the relationship.

That kind of semantic matching explains why dense retrieval became important in open-domain question answering. The Dense Passage Retrieval work from 2020 showed that learned dense representations could beat a strong BM25 baseline on several retrieval benchmarks for open-domain QA.

In a basic pipeline you embed every chunk in the knowledge base. When a user asks a question, you embed it too, compare it with the stored chunk embeddings and fetch the closest matches.

Storing embeddings in a vector database

The embeddings need a home that supports searching, which is the role of a vector database, or vector store. It keeps each chunk's embedding, its original text and any useful metadata. At question time, it finds the chunks whose embeddings sit nearest to the question's.

For a prototype, a local or in-memory store is enough. Production might call for a managed vector database, a search engine that supports vectors, or a database extension that adds vector search. Scale, speed, cost, filtering needs, security and how often documents change should drive the choice.

Treat it as part of the retrieval engine and not merely a storage box. Good metadata filtering can make a large difference. A question about the European refund policy shouldn't pull a policy from another region. If a user may view only their own department's HR documents, the retriever must enforce that before anything reaches the model.

That matters because RAG can leak information when access control is loose. The model shouldn't be trusted to hide restricted content after retrieval, and it's safer to keep restricted material from being retrieved at all.

Retrieving the relevant chunks

Answering starts when a user asks something. The system embeds the question, searches the vector store and returns the best-matching chunks. Practitioners call this top-k retrieval, with k standing for how many chunks come back; top-5 gives the five nearest.

Choosing k is a balancing act. Retrieve too little and the answer may miss key details. Retrieve too much and the prompt becomes noisy, costly or confusing. More context isn't always better, but better context is.

Many basic systems begin with vector similarity search, which handles questions about meaning well but isn't perfect. Some queries depend on exact words, IDs, names, dates, error codes or legal phrases, and keyword search can beat vector search alone there.

Hybrid search is popular for that reason. It combines vector search with full-text keyword search and merges the results. Azure AI Search, for instance, describes hybrid search as a single query that runs both kinds together and merges the ranked results using Reciprocal Rank Fusion. In plain terms, the system gets two routes to the answer: by meaning and by exact wording.

Vector search is a sensible opening move. If the audience frequently types product codes, policy titles, legal wording or error messages, plan on adding the hybrid approach.

Building the prompt

Once chunks are retrieved, they go into the model's prompt, which should clearly state the context, the user's question and what the model is expected to do. A simple RAG prompt can tell the model to answer using the supplied context, to rely only on information found there, to say it doesn't know when the context lacks the answer, and to cite the source title or section where possible. The retrieved chunks and the question follow.

These instructions matter because the model carries general knowledge of its own. With no firm guidance it may mix retrieved facts and its own assumptions. That is sometimes acceptable, but for a support bot, legal assistant, medical tool or policy helper the model ought to stay close to the evidence it was given.

OpenAI's prompt engineering documentation frames RAG as adding relevant context, drawn from a vector database or file search, to the generation request. Phrasing counts: it positions the retrieved text as the thing the model should lean on.

A well-written prompt also deals with uncertainty, by allowing the model to say that the provided documents don't answer the question. That reply is less flashy than a confident guess and far more useful.

Generating the answer

Now the language model writes the final answer, the part users actually see. It should be clear, grounded and suitably specific. It shouldn't dump raw retrieved text unless asked. It should explain in natural language, mention important limits and cite sources where it can.

A good example: "Refurbished laptops have to come back within 14 days of delivery. The policy mentions no exception for 20-day returns, so on the basis of the provided policy, a return requested after 20 days would not qualify." That works because it states the rule, applies it to the user's situation, and doesn't pretend an exception exists when the source offers none.

Citations add value too, since they let users verify an answer and not just trust the model. Making AI answers traceable is one of RAG's biggest practical benefits.

Evaluating the pipeline

A RAG system should be tested before anyone relies on it. A common mistake is to inspect only the final answer. It's better to evaluate retrieval and generation separately: first ask whether the retriever found the right chunks, then whether the model used them properly.

A simple evaluation set pairs real user questions with the correct source passages and the expected answers. For each question, check that the right document appears among the retrieved results. When the right passage never reaches the model, dependable answers are unlikely.

After that, review the generated answer. Does it stay true to the context? Has it added claims nobody supports? Did it address the real question and cite the right source? When the documents fell short, did it admit that?

The Claude RAG cookbook recommends assessing the retrieval pipeline and the end-to-end system separately and not going by vague impressions of answer quality, which is advice worth following even on a small project.

A basic evaluation loop can track:

• Retrieval accuracy, meaning whether the right passage was found.
• Faithfulness, meaning whether the answer is supported by the retrieved context.
• Completeness, meaning whether the important details are present.
• Citation quality, meaning whether cited sources are truly relevant.
• Latency, the time an answer takes.
• Cost, the tokens and retrieval calls used.
• User satisfaction, meaning whether the problem got solved.

RAG is seldom perfect the first time. Evaluation shows where to improve, whether that's chunking, metadata, retrieval, prompts, reranking or the source documents themselves.

Where basic pipelines go wrong

The first version of a pipeline usually works well enough to be thrilling and badly enough to be humbling. Typical trouble spots:

• Poor chunking. A relevant chunk comes back, but the whole answer isn't in it.
• Retrieval noise. Passages that sound related but don't answer the question are returned, often when a query is broad, ambiguous or full of terms that appear across many documents.
• Stale information. Updates are easier than retraining, but only if the knowledge base is kept current. Old documents left in the index can still be retrieved.
• Metadata mistakes. Missing dates, versions, regions or permissions let through information that is technically relevant but not actually useful.
• Overtrust. RAG reduces hallucination without eliminating it. The model can still misread context, overgeneralise or combine snippets misleadingly. Grounding isn't a guarantee of truth.
• Security gaps. Prompt injection is a genuine risk in LLM applications. OWASP describes it as manipulating model behaviour through crafted inputs, including ones that make the system ignore instructions or reveal information. In RAG, malicious instructions can sit inside retrieved documents and not just user messages. A web page, support ticket or shared document might contain text telling the model to disregard earlier instructions and expose the system prompt. Treat anything retrieved as data, never as trusted instructions.

A simple architecture

A beginner-friendly design runs in two lines. On the indexing side, documents are collected, cleaned, split into chunks, embedded, and stored in a vector store together with the original text and metadata. On the answering side, the user's question is embedded, the most relevant chunks are retrieved and optionally filtered or reranked, then placed into a prompt for the language model to answer from that context.

In shorthand: documents become chunks, then embeddings, then a vector store; a question becomes a query embedding, passes through the retriever to produce context, and the LLM turns that into an answer.

Frameworks such as LangChain and LlamaIndex supply tools for loading documents, building indexes and retrievers, and connecting those retrievers to language models. LangChain's docs present retrieval as using your own knowledge bases via loaders and vector stores. LlamaIndex frames RAG as indexing your data up front so each query can pick up relevant context ahead of the LLM call.

No complex system is needed to learn this. A small prototype needs just five parts: some documents in a folder, an embedding model, somewhere to store vectors, a retriever and a chat model. What counts is understanding each step and not treating the pipeline as a black box.

What separates a good pipeline

A good pipeline does more than retrieve text. It finds the correct text, presents it to the model in usable form and yields an answer that stays faithful to the evidence. The best systems share a few habits: clean, current sources, careful chunking, kept metadata, retrieval by both meaning and precision where needed, clear instructions, citations, separate evaluation of retrieval and generation, protection of sensitive data before it reaches the model, and the ability to admit ignorance.

Good RAG is therefore more than an AI feature. It combines knowledge management, search, product design and safety.

Where to go next

Once a simple pipeline works, several upgrades are available:

• Add hybrid search to retrieve by meaning and exact terms.
• Add a reranking step in which a second model reviews and orders the retrieved chunks.
• Chunk by document structure and not fixed lengths.
• Rewrite vague queries into clearer ones.
• Sharpen citations by recording source titles, sections and links.
• Build dashboards to watch quality over time.

Advanced setups bring in agents, retrieval over several steps or over graphs, queries against structured databases, retrieval across images and text, or rerankers tuned to a domain. Recent surveys suggest the field is shifting from simple RAG toward modular systems, in which retrieval, generation and augmentation are separate, flexible components and not one fixed process. Even so, the basic pipeline stays the foundation. Locate relevant evidence, pass it to the model, and hold the answer to account against that evidence.

Why it's worth learning

RAG has caught on because it addresses a practical need. Businesses, researchers, developers and teams already store valuable knowledge in documents, databases, manuals, tickets and notes, and a language model alone can't reliably know any of it. RAG gives the model a way to look things up.

The recipe itself is short: prepare documents, split them up, embed and store the pieces, retrieve the right ones, then have the model answer from them. The craft is in the details, where source quality, chunking, retrieval, prompt design, evaluation and security all matter. A basic pipeline isn't the final form of an intelligent assistant, but it remains among the plainest routes to AI that is more useful, up to date and better grounded. Rather than asking the model to be an all-knowing oracle, RAG hands it a library card and teaches it to use the library well.