RAG From Scratch: Chunk, Embed, Store, Retrieve, Answer With Context

Ask a chat model what a particular company's refund policy says and it will rarely admit ignorance. More often it produces a smooth, believable paragraph that has nothing to do with the real policy. The cause is not a defect. A language model is trained once, then frozen. It has never read a private PDF, an internal wiki or a Notion workspace, and it knows nothing written after its training cutoff.

Retrieval-augmented generation, usually shortened to RAG, is the standard remedy, and the idea is almost embarrassingly small. Rather than expecting the model to remember an answer, the system finds the relevant text and places it directly in the prompt, then asks the model to read it and reply. Everything else is engineering to make that lookup reliable. The running example below is a support bot that answers questions from a company policy PDF.

Two students, one exam

Picture two people sitting the same test. The first memorized a textbook months ago and writes from memory. If the book was out of date, or a chapter was half forgotten, the answers come out confident and wrong. The second person is allowed an open book. Before writing, they turn to the right page, read it, and combine that page with their general reasoning.

The second student usually wins, and not through extra intelligence. Looking things up before speaking is simply a better way to answer. In a RAG system the language model plays the second student, the company data is the open book, and the vector database is the index that says which page to turn to.

Two pipelines, not one

Any RAG system, however elaborate, is two separate pipelines joined at the middle.

• Indexing runs offline, once, and again whenever the documents change. Raw files become a searchable index.
• Retrieval and generation run on every question. The index is searched, the best pieces are pulled out and given to the model.

Newcomers often try to picture a single long pipeline, which makes it confusing. Splitting it in two helps, since the halves run at different moments and solve different problems.

Building the index

Load the documents

Start by getting the raw content into plain text. PDFs, Word files, web pages and exported notes all need a loader that returns text.

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("company_policy.pdf")
documents = loader.load()

print(f"Loaded {len(documents)} pages")

Chunk the text

Chunking deserves more attention than it normally gets. A very common first mistake is pushing a whole document into the vector store as one enormous unit. An embedding model squeezes a passage into one vector of fixed size. Feed it twenty pages and the result is a blurred average of everything, which can express that the file is generally about company policy but cannot express one particular paragraph about a thirty-day refund window.

The fix is cutting documents into small pieces that overlap. The overlap matters because a sentence sliced exactly at a boundary loses its context on both sides, and a margin of roughly ten to twenty percent of the chunk size works as a safety net.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,      # characters per chunk
    chunk_overlap=80,    # overlap between consecutive chunks
    separators=["\n\n", "\n", ". ", " "]
)

chunks = splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks")

As a guide, oversized chunks make retrieval vague, while undersized ones strip away context. A starting range of about 300 to 800 characters is sensible, then adjust after looking at real answers.

Embed each chunk

An embedding model converts text into a list of numbers that reflects its meaning and not merely its wording. Two sentences that say the same thing end up as vectors sitting near each other, even when they share no words at all.

from langchain_openai import OpenAIEmbeddings

embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")

# under the hood, this is what happens per chunk:
vector = embedding_model.embed_query("30-day refund window for annual plans")
print(len(vector))   # e.g. 1536 numbers representing this sentence's meaning

One way to picture it is a flat map where related ideas gather in clusters.

        "refund policy" •
                          \
                           • "money-back guarantee"
   "vacation days" •
                     \
                      • "paid time off"

A phrase like refund policy lands beside money-back guarantee although the words differ, and that is the whole trick behind semantic search. It explains why RAG outperforms the old Ctrl+F style of keyword matching.

Store the vectors

Each vector goes into a vector database together with the original text. That store is the index the open-book student leafs through.

from langchain_community.vectorstores import FAISS

vectorstore = FAISS.from_documents(chunks, embedding_model)
vectorstore.save_local("faiss_index")

FAISS is a comfortable place to begin because it is free and runs on a local machine, and Chroma is about as simple. Managed services such as Pinecone or Weaviate make sense when production scale arrives. The concept does not change from one to the next, and only storage and infrastructure differ.

Indexing is now finished. The result is a searchable memory built from the documents, and it only needs rebuilding when those documents change, not for each user question.

Answering a question

This half runs every time somebody types a query.

Embed the question with the same model

query = "Can I get a refund on my annual subscription?"

The question has to be embedded with exactly the model that was used while indexing. People trip on this constantly. If the index was built with one embedding model and queried with another, the vectors sit in different meaning spaces, and similarity search quietly stops working. No error appears, only poor results, so the rule is one model throughout.

Fetch the closest chunks

retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
relevant_chunks = retriever.invoke(query)

for chunk in relevant_chunks:
    print(chunk.page_content[:100], "...")

Internally, FAISS runs a similarity search. It measures the distance, typically cosine similarity, between the vector for the question and the vector of every stored chunk, then returns the nearest ones. It is the clusters-on-a-map picture again, applied in reverse to find the cluster the question belongs to.

Paste the chunks into a prompt

This is the move that makes RAG work: the retrieved text is literally inserted into the prompt.

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_template("""
Answer the question using ONLY the context below.
If the answer isn't in the context, say "I don't have that information."

Context:
{context}

Question: {question}
""")

The instruction to answer using only the supplied context is what keeps the model grounded and stops it from inventing things. It separates an open-book exam from a closed-book one where improvising is allowed.

Wire it into a chain

Now the pieces are assembled into the RAG chain, the thing people mean when they talk about building one. In LangChain it is normally written with the pipe operator, which passes the output on the left into the step on the right.

from langchain_openai import ChatOpenAI
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

llm = ChatOpenAI(model="gpt-4o-mini")

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

answer = rag_chain.invoke("Can I get a refund on my annual subscription?")
print(answer)

Read left to right, the chain narrates itself. The question goes in, chunks are retrieved and formatted as context while the question passes through unchanged, the prompt template gets filled, the model is called, and the plain text is parsed from its reply.

Run it, and in place of a hallucinated guess comes something like: yes, annual plans qualify for a full refund within thirty days of purchase, as the refund policy states. The answer is grounded and specific, and it can be traced to a real paragraph in the PDF.

Why not paste everything in?

A fair objection is that context windows are now huge. Why not place the entire document in every prompt and skip vector search?

• Cost and latency. Each token sent costs money and takes time. With a 500-page knowledge base, sending everything on every question is wasteful when the answer sits in a single paragraph.
• Degraded recall. Even with large windows, models measurably use information buried mid-prompt worse than information that is handed over directly and briefly.
• Scale. Pasting everything fails outright once the knowledge base outgrows any window, as with thousands of PDFs, a full company wiki or years of support tickets.

RAG lets a system grow from one document to a whole knowledge base while the prompt stays the same size. Only the retrieval step touches the full dataset, and the prompt sees a handful of relevant chunks.

Mistakes that show up early

• Poor chunking ruins retrieval regardless of model quality. If a table is cut in half or a sentence breaks mid-thought, nothing downstream repairs it. Check retrieval quality before blaming the model.
• Mismatched embedding models make similarity scores meaningless. An index built with text-embedding-3-small cannot be queried with some other model.
• Retrieving too few or too many chunks both hurt. With k=1 the real answer may be missed, and with k=20 the model drowns in noise and cost. Begin at three to five and tune from the results.
• Without a fallback for not knowing, the model will still make up an answer even when relevant context is in front of it. Spell out what to do when the context lacks the answer.

Where to go next

When the basic loop works, three upgrades are the natural next moves.

• Hybrid search blends vector similarity with keyword matching, because some queries, such as exact product codes or names, are better served by keywords.
• Re-ranking fetches a wider set of candidates first, then lets a second and more precise model reorder them before the language model sees any.
• Metadata filtering attaches details like date, source or department to each chunk, so retrieval can be narrowed before similarity search runs at all.

None of these matter until the basic loop of chunk, embed, store, retrieve and generate holds up. The quickest way to really understand it is to load one PDF that matters, break something on purpose, such as mismatched embedding models or chunks with no overlap, and watch the answers degrade. With RAG the failure modes teach more than the happy path.