RAG with LangChain from Scratch: Loaders, Chunks, Embeddings, Evaluation

A system that reads your documents and answers questions about them is one of the most useful things you can build with an LLM today. The pattern is called RAG, retrieval-augmented generation. Here's every moving part, what it's for, and how to tell whether your build is any good.

What LangChain is

LangChain isn't a model and it isn't a database. It's a framework, closer to a universal adapter. There are dozens of LLM providers with their own APIs, dozens of vector databases, dozens of document loaders. Without a layer in the middle, using GPT-4 today and Claude tomorrow means rewriting half your code. LangChain gives one consistent interface to all of them.

The name comes from its second job. A real LLM app is rarely a single model call. You might load a document, summarize it, run another prompt over the summary and format the result. A chain wires those steps into a pipeline. For RAG the chain starts with the user's question. That gets embedded, the embedding drives a database search, and the top hits are dropped into a prompt template. The LLM is called with that prompt, its reply is parsed, and the parsed answer goes back to the user. LangChain supplies the building blocks for each step, so you aren't gluing everything by hand.

Loading documents

First the documents have to get in, and not as raw bytes. Whether the data is a PDF, a webpage, a CSV, a Word file or a folder of markdown, a loader turns it into a standard Document object with two fields: page_content for the text and metadata for where it came from (page, URL, anything useful). The metadata matters later, because it lets you show users which source an answer came from.

from langchain.document_loaders import PyPDFLoader

loader = PyPDFLoader("handbook.pdf")
documents = loader.load()

That yields one Document per page of the PDF, each carrying its text and its page number. This is the habit worth building early: always keep the origin of a piece of text attached to it, because every later step, from chunking to citing a source in the final answer, depends on that trail. Swap the loader for another file type and the output keeps the same shape.

Chunking

A document might be a 200-page PDF. Even with a huge context window, stuffing whole books into every prompt is wasteful, expensive and often worse than giving the model focused snippets. So long documents get split into pieces of a few hundred tokens, and only the most relevant pieces go to the LLM when someone asks a question.

Two settings matter most. Chunk size is the length of each piece: too small and you lose context, too large and you drag in irrelevant text. Around 500 tokens is a fair starting point. Overlap is how much neighboring chunks share. With no overlap, a sentence cut across two chunks loses its meaning, while with overlap both chunks see the bridge.

RecursiveCharacterTextSplitter is the usual choice. It prefers natural boundaries, trying paragraphs first, then sentences, then words, and only chops mid-word as a last resort. There are specialized options too: MarkdownHeaderTextSplitter for documents with headings and TokenTextSplitter when you want to count real tokens and not characters.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = splitter.split_documents(documents)

Chunking sounds trivial and strongly affects quality. It's also the setting people tune last, even though a bad split quietly caps everything downstream. If the chunks are wrong, no retrieval trick will rescue the system.

Embeddings

Computers can't ask "which chunk resembles my question?" of raw text. An embedding model, also called an encoder, converts text into a vector, a list of numbers, and the key property is that text with similar meaning gets similar vectors.

Picture every text being given an address in a space with hundreds or thousands of dimensions. A barking dog and a yelping puppy get neighboring addresses, while a stock market crash lives somewhere far off. Comparing meaning turns into geometry: how close are two points?

Two clarifications help. First, the encoder creates the vectors, not the vector database. Chroma, Pinecone, Weaviate and Qdrant only store vectors and search them quickly, and none of them understands language. People mix this up constantly. Second, vector size depends on the encoder. OpenAI's text-embedding-3-small outputs 1536 dimensions, while an open-source model such as all-MiniLM-L6-v2 outputs 384. More dimensions capture more nuance, at the price of more storage and slower search.

Real embeddings are impossible to picture, so people project them down to 2D or 3D. Embed a pile of words and project them, and clusters appear: animals together, fruit elsewhere, vehicles in another corner. Nobody told the model about those categories. It learned from huge amounts of text that "dog" and "cat" show up in similar contexts.

That's what makes RAG work. If a user asks what the company says about parental leave, the question's vector lands near chunks about "family leave benefits" or "time off for new parents", although those words differ.

The pipeline

There are two phases. Indexing happens once at setup: load, chunk, embed each chunk, store the vectors. New documents later just get embedded and added. Retrieval runs on every question: embed the question with the same encoder, find the nearest chunks in the vector database, put them in a prompt with the question and send it to an LLM, which grounds its answer in your documents and not in guesses. One vector database serves both phases, written to during indexing and read from during retrieval. That's the whole architecture. Because indexing is a one-time cost, the slow part (embedding thousands of chunks) doesn't touch the user's wait time. Only the single query embedding, the search and the LLM call do.

At runtime four actors take part: the user, your application code, the retriever and the LLM. The user sends a question to your code. Your code asks the retriever for relevant chunks, and the retriever embeds the question and searches the database. The top-K most similar chunks come back. Your code builds a prompt from those chunks plus the original question and sends it to the LLM, which generates an answer from the provided context. Your code returns that answer.

LangChain covers most of this with two abstractions. The LLM (or ChatModel) wraps your model and the Retriever wraps the vector store lookup. Chain them and the framework does the rest:

from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(),
    retriever=vectorstore.as_retriever(),
    return_source_documents=True
)
response = qa_chain.invoke({"query": "What is our parental leave policy?"})

That's a working RAG system in a handful of lines, and it's the point where it's tempting to stop. Don't. A system that answers is not yet a system you can trust.

Why evaluate

The trap is asking one question, getting a plausible answer and calling it done. RAG fails in quiet ways. The retriever may pull the wrong chunks, the LLM may hallucinate even with good context, and chunking may slice an answer in half. Evaluation tells you which part is broken: retrieval, generation or both, since each has a different fix. It also gives you a yardstick, because you'll keep trying other chunk sizes, encoders and prompts, and without measurement every change is a guess.

Retrieval metrics

Four standard metrics, each telling you something different:

• Recall@K: of the documents that should have appeared, how many landed in the top K? With 4 relevant documents in the corpus and 3 of them in your top 5, Recall@5 is 0.75. High recall means little is missed.
• Precision@K: of the K results returned, how many were relevant? Three good ones out of five gives 0.60. High precision means little noise.
• MRR (Mean Reciprocal Rank): only the position of the first correct result counts. First hit at position 2 scores 0.50, at position 1 scores 1.0. It rewards surfacing a correct answer quickly.
• nDCG (Normalized Discounted Cumulative Gain): the most refined one. It weighs how relevant each document is and where it sits, so higher-ranked relevant items count more, and it compares against the best possible ranking.

In practice, track two: one for completeness (Recall) and one for ordering quality (MRR or nDCG).

Answer quality

Perfect retrieval can still produce a bad answer, so score answers separately. You start from the same curated test set, where each entry pairs a question with the chunks that ought to come back and a reference answer. The four retrieval metrics above cover the first half. For the answers themselves, score along other dimensions:

• Faithfulness: is it grounded in the retrieved context or invented?
• Relevance: does it address the question?
• Correctness: how does it compare with the ground truth?
• Completeness: does it cover everything it should?

The usual method is a second LLM acting as judge. Give it the question, the answer and the ground truth, and ask for a rating on each dimension. It isn't perfect, but it scales.

The test set

Every evaluation step above leans on one thing: a collection of questions with known good answers.

This is the most underrated piece. Without it nothing can be evaluated. Aim for 50 to a few hundred examples, each with a realistic question, the chunks containing the answer and an ideal response. Sources are real user queries if the system is live, or synthetic ones: feed your documents to an LLM, have it write question and answer pairs, then review and clean them up.

Guard this set. Once it exists, every change gets scored against it, and that's how you learn whether a change helped.

Where it gets harder

Boiled down, RAG is: chunk the docs, vectorize each chunk, store them, embed user queries, find the nearest vectors and hand those chunks to an LLM. LangChain supplies standard parts for each step and evaluation shows how well it works. A real product raises harder questions: the right chunk size for your documents, which encoder is best, how to keep the index fresh, when to rerank, whether to use hybrid search. Those decisions separate a toy demo from something useful, but they rest on the foundation above.