Improving RAG Answers One Measured Change at a Time

A first RAG pipeline is easy to assemble: load documents, embed them, store the vectors, fetch a few chunks and pass them to an LLM. It runs, yet the answers are often wrong even when the document plainly contains the correct information. In most of those cases the model is not the culprit; the retrieval step never handed it the right context. This guide walks through the retrieval levers that usually matter most and, more importantly, a disciplined way to prove which of them actually helps your system.

Treat bad answers as retrieval problems first

Before changing prompts or models, check what was retrieved for a failing question. If the relevant passage is missing from the context, no amount of prompt tuning will fix the answer.

Chunk along meaningful boundaries

Chunk size has a surprisingly large effect on retrieval quality. Chunks that are too large mix several topics, so their embeddings become vague and they pull in unrelated text. Chunks that are too small cut sentences away from the context that gives them meaning. Rather than splitting blindly every 500 characters, keep related paragraphs or whole sections together, and use headings and paragraph breaks as natural split points.

Tune top_k instead of guessing it

Many pipelines retrieve the five most similar chunks simply because five is a common default. The correct passage, however, may sit at position six or seven. Raising top_k can improve recall, but every extra chunk also adds noise and tokens to the prompt. Treat top_k as a parameter to test against your own questions, not a constant to copy. Relevance thresholds are another option, covered in going beyond top-k with thresholds, hybrid search and reranking.

Add a reranking stage

Vector search is fast but coarse: it is good at finding plausible candidates and weaker at deciding which one is truly best. A reranker adds a second pass. First, vector search returns a wider set, perhaps ten chunks. Then a reranking model, typically a cross-encoder that reads the query and each chunk together, scores them and keeps the best three or four. The model receives cleaner context, at the price of extra latency and cost per query.

Combine semantic and keyword search

Embeddings capture meaning well, but exact tokens sometimes matter more than meaning. A query such as ERROR_CODE_4291 has little semantic content, so similarity search may miss the one document that mentions it. Keyword ranking such as BM25 handles this case well. Many systems therefore run both vector and keyword search and merge the results, an approach known as hybrid search.

Bridge the vocabulary gap in queries

Users rarely phrase questions the way documents are written. Someone may ask why a payment is failing, while the relevant page talks about card authorization failure. Query rewriting, MultiQuery (generating several phrasings and retrieving for each) and HyDE (generating a hypothetical answer and searching with its embedding) can all close that gap. They add LLM calls and complexity, though, so reach for them only after the simpler levers above.

Measure every change against a baseline

The most important habit is refusing to assume. "We added reranking, so retrieval is better" is a hypothesis until it is measured. Build a small set of real questions with known relevant passages, then track metrics such as:

• Recall@K: whether the correct information appears in the top K retrieved chunks.
• Context precision: how much of the retrieved context was actually useful.
• Faithfulness: whether the answer stays grounded in the retrieved context.
• Answer relevancy: whether the answer addresses the question that was asked.

Record a baseline first. In an illustrative run, it might look like this:

Baseline Recall@5: 68%

Then apply one change at a time, rerun the same questions and record each result. A sequence of improvements could look like this:

Better chunking: 74%
Hybrid search: 82%
Reranking: 89%

These figures are an example, not a benchmark; your own data will behave differently. The point is that a per-change log tells you which step earned its complexity and which did not. For a deeper treatment of locating failures, see evaluating RAG by failure stage.

Wrapping up

Start from the simplest pipeline, query to retrieval to context to LLM, and improve one stage at a time: make a change, measure it, weigh its latency and cost, and repeat. A modest RAG system that you understand and can measure is usually worth more than an elaborate one full of techniques nobody can justify with numbers.