Debugging RAG Retrieval: Recall@K, BM25, Hybrid Fusion and Reranking

A question the retriever got wrong

Take a small RAG pipeline and ask it something that ought to be trivial: how long is a CSV export link usable? The documentation has the answer. A finished CSV download link stays active for seven days. Yet the retriever ranks that passage fourth, and with only the top three chunks going to the language model, it never reaches the prompt.

What makes the miss irritating is that the top three results don't look wrong. One describes how CSV exports are processed, one covers large exports and one explains canceling an export. A glance suggests the retriever is doing fine. It isn't. It found passages on the right topic, and none of them held the fact the question needed.

The usual reflex with a bad RAG answer is to head downstream: rewrite the prompt, add an instruction, blame the model. Prompt tweaking is satisfying because three edited lines change the output right away. Here it would've achieved nothing. The model didn't pick those documents, the retriever did, and no prompt can bring back a passage that never entered the context window.

This matters more as a system matures. Past the demo stage, poor retrieval becomes a failure that's hard to spot from the final answer. The model can sound sure of itself, the context can seem vaguely relevant, and the real fault sits several steps upstream. A better habit is separating the failure causes: missing evidence, weak chunking, poor ranking or the generation step. The debugging order then becomes:

Bad answer, then inspect retrieval, then measure it, then fix it, and only afterward look at generation.

A small experiment shows how much the earlier stages matter.

A corpus small enough to read

The experiment used a made-up documentation set about rate limits, backups, CSV exports, API errors, authentication, monitoring and audit logs. It wasn't impressive, and that was intentional: small enough for a person to read every retrieved passage, yet tangled enough that several chunks could seem relevant to one question. Every chunk got a stable ID:

chunks = [
    {
        "id": "audit_03",
        "source": "audit-logs",
        "text": (
            "Enterprise audit logs are retained for 365 days "
            "before automatic deletion."
        ),
    },
    {
        "id": "errors_07",
        "source": "api-errors",
        "text": (
            "NX-204 means the requested resource exists but is not "
            "available in the caller's current region."
        ),
    },
    {
        "id": "exports_01",
        "source": "csv-exports",
        "text": (
            "CSV exports run asynchronously and appear in the exports "
            "panel when processing completes."
        ),
    },
    {
        "id": "exports_04",
        "source": "csv-exports",
        "text": (
            "A completed CSV download link remains active for seven days."
        ),
    },
]

The ID looks like bookkeeping, but it avoids a nasty shortcut later. The set also included 12 evaluation queries, each labeled with the chunks that were genuinely relevant:

eval_cases = [
    {
        "query": "How long are enterprise audit logs kept?",
        "relevant": {"audit_03"},
    },
    {
        "query": "What does error NX-204 mean?",
        "relevant": {"errors_07"},
    },
    {
        "query": "How long is a CSV export link usable?",
        "relevant": {"exports_04"},
    },
]

Twelve queries would be a laughable production benchmark, but that wasn't the goal. The aim was to have enough cases to reveal when fixing one query quietly broke another. Even a tiny evaluation set punctures the confidence that comes from retesting the same three questions by hand.

A deliberately plain baseline

The baseline used TF-IDF, truncated SVD and cosine similarity. A real project would likely choose a modern embedding model, but adding one here would introduce another variable without helping the question at hand. The retriever didn't need to be fancy. It needed to be something that could be broken, inspected and measured.

import numpy as np

from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.preprocessing import normalize


class LsaRetriever:
    def __init__(self, chunks, dims=16):
        self.chunks = chunks

        self.tfidf = TfidfVectorizer(
            stop_words="english",
            ngram_range=(1, 2),
            sublinear_tf=True,
        )

        term_matrix = self.tfidf.fit_transform(
            chunk["text"] for chunk in chunks
        )

        # The corpus is tiny. SVD doesn't need dimensions it cannot use.
        dims = min(
            dims,
            term_matrix.shape[0] - 1,
            term_matrix.shape[1] - 1,
        )

        if dims < 1:
            raise ValueError("Need more text to build the LSA index.")

        self.svd = TruncatedSVD(
            n_components=dims,
            random_state=0,
        )

        self.index = normalize(
            self.svd.fit_transform(term_matrix)
        )

    def search(self, query, limit=None):
        query_vec = self.tfidf.transform([query])
        query_vec = normalize(self.svd.transform(query_vec))

        similarity = cosine_similarity(
            query_vec,
            self.index,
        )[0]

        ranked = np.argsort(similarity)[::-1]

        if limit is not None:
            ranked = ranked[:limit]

        return [
            (self.chunks[i], float(similarity[i]))
            for i in ranked
        ]

The first chunking pass split the documentation sentence by sentence. Asking the CSV question then produced this:

1. 0.879  exports_01
   CSV exports run asynchronously and appear in the exports panel...

2. 0.843  exports_02
   Large exports are split into multiple compressed files.

3. 0.830  exports_03
   Users can cancel an export while it is still queued...

4. 0.772  exports_04
   A completed CSV download link remains active for seven days.

The top score of 0.879 is almost funny in hindsight, since it looks like the system knows exactly what it's doing. Then you read the chunk, and it doesn't answer anything. That's the moment to stop treating a similarity score as reassurance.

The failure happens before generation. The retriever picks plausible but unhelpful context, and the LLM sees only what survives the top-k cutoff.

Print what came back

Before touching models, chunk sizes or retrieval strategies, a tiny helper that prints what the retriever returns earns its keep.

def show_hits(retriever, query, limit=5):
    print(f"\n{query}\n")

    for position, (chunk, score) in enumerate(
        retriever.search(query, limit),
        start=1,
    ):
        print(
            f"{position:>2}. {score:.3f}  "
            f"{chunk['id']} ({chunk['source']})"
        )
        print(f"    {chunk['text']}\n")

Of everything in the experiment, this helper gets used the most. Once the chunks are on screen, failures sort into types. The right document may be present while the wrong section wins, or two or three chunks may be near-copies. Elsewhere the query and passage overlap in vocabulary enough to score well while the passage is useless as evidence. Plain numbers conceal each of these cases unless somebody reads the text.

It also underlines something easy to gloss over: the retriever makes a consequential decision. Whatever survives this stage becomes the model's version of reality, and that deserves more scrutiny than it usually gets.

Evaluating by ID, not by string match

An early evaluation draft had a shortcut that was later dropped. It checked for a hint string in the chunk text:

if answer_hint in chunk["text"]:
    ...

Convenient, but unreliable. A small shift in chunk boundaries, a punctuation change or a paraphrase can make a correct retrieval look wrong. Worse, it ties the evaluation to the wording of a synthetic answer and not to the relevance judgment that matters. That's why the chunks got IDs up front, and the evaluator can check them directly:

def evaluate_retriever(retriever, cases, k=3):
    recall_scores = []
    reciprocal_ranks = []

    for case in cases:
        hits = retriever.search(case["query"])
        relevant = case["relevant"]

        relevant_positions = [
            position
            for position, (chunk, _) in enumerate(hits, start=1)
            if chunk["id"] in relevant
        ]

        found_in_top_k = sum(
            position <= k
            for position in relevant_positions
        )

        recall_scores.append(
            found_in_top_k / len(relevant)
        )

        reciprocal_ranks.append(
            1 / relevant_positions[0]
            if relevant_positions
            else 0.0
        )

    return {
        f"recall@{k}": float(np.mean(recall_scores)),
        "mrr": float(np.mean(reciprocal_ranks)),
    }

Two measures were tracked. Recall@K answers whether the needed evidence gets into the context window, and Mean Reciprocal Rank distinguishes a correct chunk at position one from a buried one at position eight. Baseline results: Recall@1 of 0.750, Recall@3 of 0.917, MRR of 0.854.

Recall@3 is the troubling number. With a planned context of three chunks, some questions already lost the information needed for a correct answer before generation even began. Adjusting the system prompt at that point would be theater.

Chunk size: a partial culprit

Sentence-level chunks were the prime suspect, and they were partly to blame. With two-sentence chunks, Recall@1 rose from 0.750 to 0.917, Recall@3 hit 1.000 and MRR climbed to 0.958. Three-sentence chunks came out nearly identical. Whole pages produced perfect scores, which feels great for about five seconds.

The catch is that the pages were tiny. Returning a whole page made it nearly impossible to miss the answer, so retrieval improved partly because each unit held more of the corpus. That has a price. Bigger chunks use up context, carry unrelated sentences and leave less room for separate evidence, and a retriever that always returned an entire manual would score wonderfully on recall too.

So the lesson wasn't a magic chunk size. It was that the original sentence-level split was too aggressive, which is a duller conclusion but a usable one. It also raises suspicion about settings like these:

chunk_size = 500
chunk_overlap = 50

Nothing is wrong with those numbers as such. They just shouldn't reach production because a tutorial happened to use them. Chunking is part of retrieval, and it needs evidence like everything else.

Exact terms carry the meaning

The CSV example held another clue. Every high-ranking passage was semantically close to exports, so the retriever hadn't misread the domain. It simply wasn't discriminating enough among passages inside it. Technical documentation sharpens this, because exact strings carry an enormous amount of meaning. An error code like NX-204 isn't merely semantically similar to a neighboring code, and the same goes for customer_id, an endpoint, a version number or a function name.

BM25 scored a flat 1.000 on Recall@1, Recall@3 and MRR. It beat the semantic baseline, did better than anyone would have guessed and, irritatingly, was flawless on this tiny set. A headline like "embeddings are dead" would be tempting, but the data doesn't support it. What it shows is that this evaluation set paid off heavily for exact terminology. A collection full of reworded questions might pick a different winner. The takeaway is how much lexical matching counted for this corpus, which is the reason to build an evaluation set at all.

Hybrid retrieval with rank fusion

Dropping semantic retrieval still seemed premature. The two methods fail differently, so their rankings were combined with Reciprocal Rank Fusion. One implementation detail counts: each retriever contributes only a limited number of candidates. Ranking a huge corpus twice and merging everything in Python would be a strange design to copy.

from collections import defaultdict


def fuse_rankings(vector_hits, bm25_hits, rrf_k=60):
    fused = defaultdict(float)
    chunks_by_id = {}

    for hits in (vector_hits, bm25_hits):
        for rank, (chunk, _) in enumerate(hits, start=1):
            chunk_id = chunk["id"]
            chunks_by_id[chunk_id] = chunk
            fused[chunk_id] += 1 / (rrf_k + rank)

    ranked_ids = sorted(
        fused,
        key=fused.get,
        reverse=True,
    )

    return [
        (chunks_by_id[chunk_id], fused[chunk_id])
        for chunk_id in ranked_ids
    ]

The wrapper around it is plain:

def hybrid_search(query, lsa, bm25, candidate_k=20):
    vector_hits = lsa.search(query, limit=candidate_k)
    bm25_hits = bm25.search(query, limit=candidate_k)

    return fuse_rankings(vector_hits, bm25_hits)

Hybrid retrieval lifted MRR to 0.958 and repaired the original Recall@3 miss. It still lost to BM25. The experiment wasn't massaged until the more elaborate architecture won, since that would defeat the purpose. More components don't automatically mean more relevance.

A cheap reranker on top

Retrieval has to search widely, so it makes sense for it to be cheap. Once there are 10 or 20 plausible candidates, a more focused relevance check becomes affordable. For this local experiment that check was a character n-gram scorer. It's a cheap experimental reranker and not a production recommendation. A real system would be the place to consider a learned cross-encoder.

def cheap_local_rerank(query, candidates, limit=5):
    """
    Good enough for this experiment.
    I'd use a learned reranker for a real deployment.
    """
    candidate_text = [
        chunk["text"]
        for chunk, _ in candidates
    ]

    tfidf = TfidfVectorizer(
        analyzer="char_wb",
        ngram_range=(3, 5),
        min_df=1,
    )

    matrix = tfidf.fit_transform(
        [query, *candidate_text]
    )

    relevance = cosine_similarity(
        matrix[0],
        matrix[1:],
    )[0]

    reranked = sorted(
        zip(candidates, relevance),
        key=lambda row: row[1],
        reverse=True,
    )

    return [
        (chunk, float(score))
        for ((chunk, _), score) in reranked[:limit]
    ]

The pipeline ends up with a simple two-stage shape.

The flow is to pull a broad candidate set first, then rerank it so only the strongest chunks reach the LLM. After reranking, the small test set scored a perfect 1.000 on Recall@1, Recall@3 and MRR. That figure doesn't belong on a slide announcing that retrieval is solved, since twelve queries help development but prove little about general reliability. Still, the experiment served its purpose. This is how every stage compares:

Had the story been drafted before the experiment, every row would improve in a tidy line. It didn't go that way. BM25, the plain option, tied with the final pipeline. Hybrid search improved on the starting semantic baseline without taking the top spot. Bigger chunks solved one problem and opened a different question about context budget. The untidiness makes the result more credible, and it revealed something concrete about this workload: exact terminology counted for a lot. That beats winning an architecture popularity contest.

The original question was:

How long is a CSV export link usable?

Before the changes, the ranking looked like this:

1. CSV exports run asynchronously...
2. Large exports are split...
3. Users can cancel an export...
4. A completed CSV download link remains active for seven days.

Afterward:

1. A completed CSV download link remains active for seven days.
2. Users can cancel an export while it is still queued...
3. CSV exports run asynchronously...

The source documents stayed the same and so did the question, and no smarter LLM was needed. The evidence changed.

A debugging order for bad RAG answers

That difference decides how to react when a RAG response goes wrong. When the needed evidence never reaches context, retrieval is where to look. When the evidence is plainly there and the model still talks nonsense, only then dig into generation, prompting, context order and model behavior. A third failure exists as well: the right answer does enter context but sits beneath a pile of loosely related text. That's a ranking or context-construction problem, and filing everything under "hallucination" hides it.

The practical flow checks the retrieved evidence first, then decides whether the fault belongs to retrieval or generation. The decision point is the important part. When Recall@3 is poor, spending an hour rewording "answer only using the provided context" in six increasingly stern ways makes no sense, because the context is already wrong.

Prompt changes still feel faster. You edit, rerun and see a new answer immediately. Retrieval work is slower and less satisfying: reading chunks, labeling relevance, checking rankings, arguing with chunk boundaries, and sometimes finding that an old-school lexical retriever beats the fancier setup. What it teaches is more trustworthy, though.

The question that helps most isn't why the LLM answered wrongly. It's whether the LLM ever got the information it needed. That sounds almost too obvious to bother with, yet it gets skipped often.