Why RAG Still Gives Wrong Answers: Hybrid Search, Reranking and Recall@K
Clean chunks don't guarantee good answers. See where vector retrieval fails and how fusion, a cross-encoder reranker, HyDE and Recall@K measurement fix it.
This article is published in English.
Sources: The AI Series: The Hardest Part of RAG, Part 2 — Why Perfect Chunks Still Give You Bad Retrieval
A common pattern in RAG projects: the parsing work is finished. Chunks are clean, tables come out as Markdown, and each chunk carries a page number and section title. Yet users in production still receive wrong answers to questions the system ought to handle with ease. The chunks aren't the problem. The retriever just keeps fetching the wrong ones.
Good parsing yields good data, but good data doesn't guarantee good retrieval. These are separate problems, and the second gets far less attention than it deserves.
What similarity search does underneath
Many teams treat similarity_search() as a black box, so a quick look inside helps. An embedding model converts text into a list of numbers, a vector, which is a point in a space with anywhere from 384 to 3072 dimensions depending on the model. Texts with similar meaning are supposed to land near each other, and unrelated texts far apart.
A toy version with only 4 dimensions keeps the numbers readable:
"How do I reset my password?" → [0.81, 0.12, -0.44, 0.09]
"Steps to change your password" → [0.79, 0.15, -0.41, 0.11]
"How do I cancel my subscription?"→ [0.22, 0.68, 0.05, -0.39]
Two vectors are compared with cosine similarity, which measures how closely they point the same way while ignoring length. A score of 1.0 means the same direction, 0 means unrelated and negative values mean opposite directions.
from numpy import dot
from numpy.linalg import norm
def cosine_similarity(a, b):
return dot(a, b) / (norm(a) * norm(b))
query = [0.80, 0.13, -0.42, 0.10] # "reset my password"
print(cosine_similarity(query, [0.81, 0.12, -0.44, 0.09])) # 0.998 - very close
print(cosine_similarity(query, [0.22, 0.68, 0.05, -0.39])) # 0.310 - far apart
Nothing more than that sits behind the phrase "AI understands the question". It's geometry rather than comprehension, and geometry explains the failures: it can't tell that "not allowed" flips the sense of "allowed", nor that "Error 4092" is a literal string a user needs matched.
Four ways retrieval breaks
Squeezing a paragraph into one point throws information away. Once a system leaves demo territory, the same four failures keep appearing.
- Lexical mismatch. Someone searches an exact error code or product ID, the embedding treats it as an ordinary token and the chunk holding that precise string misses the top 10.
- Negation blindness. A policy saying refunds stop after a month and a question about getting a refund after a month share most vocabulary, so they can land worryingly near each other although the meanings clash.
- Chunk isolation. The answer straddles two chunks, say a number in one and the label explaining it in the previous one, and the retriever returns only one of them.
- Recency confusion. A vector index has no notion that a document is outdated, so old and new versions of one policy often score as equally relevant.
A 20-document demo exposes none of this. It surfaces once the corpus contains real duplication, overlap and ambiguity, which describes every production system.
A pipeline instead of a single call
One call to a vector store isn't a retrieval system. A resilient one has several stages:
Each stage exists because the previous one falls short alone. That's the largest mental shift between a RAG demo and RAG in production: every stage patches a particular weakness upstream, and none is decoration.
Hybrid search with rank fusion
Vector search handles exact matches poorly, and keyword search (BM25) handles conceptual matches poorly. Rather than choosing, run both and merge the outputs.
Reciprocal Rank Fusion is the simplest way to combine two ranked lists when their scores aren't on a common scale. A typical implementation looks like this:
from collections import defaultdict
def reciprocal_rank_fusion(result_lists, k=60):
scores = defaultdict(float)
for result_list in result_lists:
for rank, doc_id in enumerate(result_list):
scores[doc_id] += 1 / (k + rank + 1)
return sorted(scores, key=scores.get, reverse=True)
vector_ids = [d.metadata["chunk_id"] for d in vector_store.similarity_search(query, k=20)]
bm25_ids = [d.metadata["chunk_id"] for d in bm25_index.search(query, k=20)]
final_ranking = reciprocal_rank_fusion([vector_ids, bm25_ids])
A frequent early mistake is to concatenate the two result lists without fusing them. Whichever list goes first silently dominates the final context, which defeats the purpose of running two strategies.
Reranking
Parsing's toughest stage is table extraction, and retrieval's is reranking. Embedding models encode a query and each document independently, comparing the results afterward. A reranker, normally a cross-encoder, reads query and candidate chunk jointly and outputs a relevance score directly. It costs more time, so it runs only on the 20 to 50 best candidates after fusion, never across the whole index.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-base")
def rerank(query, candidates, top_n=5):
pairs = [(query, c["text"]) for c in candidates]
scores = reranker.predict(pairs)
for c, s in zip(candidates, scores):
c["rerank_score"] = float(s)
return sorted(candidates, key=lambda c: c["rerank_score"], reverse=True)[:top_n]
In one internal support-search system, a reranker placed after fusion lifted Recall@5 from roughly 61% to 84%, with the embedding model and chunk size untouched. That single change outweighed the rest of the pipeline combined, which makes it the usual first suggestion when RAG answers feel close but not quite right.
Closing the vocabulary gap
At times the index is healthy and the real issue is wording. Users write casually ("why won't it let me log in") whereas documentation is formal ("Authentication failures may occur when session tokens expire").
Two techniques address it. Query rewriting has an LLM restate a vague question closer to the phrasing of the source material. HyDE (Hypothetical Document Embeddings) goes a step further: the LLM drafts a short imaginary answer, and that text, not the raw question, gets embedded and searched, because a plausible answer lands much nearer real answers in vector space than the question does.
def hyde_retrieve(query, llm, vector_store, k=5):
hypothetical = llm.invoke(f"Write a short passage answering: {query}")
return vector_store.similarity_search(hypothetical, k=k)
A support-ticket system that adopted this recovered a meaningful share of queries that had returned nothing useful, purely because users and documentation use different dialects.
Measure retrieval directly
A tempting shortcut is to judge retrieval by whether the final LLM answer sounds right. That reveals almost nothing about which stage failed.
What's needed is a small labeled set, even 50 to 100 pairs of query and correct chunk, so Recall@K can be measured before the LLM gets involved. Recall@K asks: of the chunks that really contain the answer, how many ended up in the top K?
def recall_at_k(retrieved_ids, relevant_ids, k):
top_k = set(retrieved_ids[:k])
return len(top_k & relevant_ids) / len(relevant_ids) if relevant_ids else 0.0
Without it, each change to chunk size, embedding model or reranker is a guess wearing the costume of an improvement.
Pitfalls and a sensible stack
Raising k seldom repairs weak retrieval, as it mainly feeds the LLM more distractions. Spending the first week on chunk-size tuning is also poor value next to the jump a reranker delivers. Finally, build the evaluation set before complaints arrive, because diagnosing retrieval blind afterward wastes an entire sprint.
A reasonable stack for building this layer fresh: Qdrant for vector search, BM25 via OpenSearch fused with RRF, a self-hosted BGE reranker with Cohere Rerank as a hosted fallback, and HyDE reserved for conversational query traffic.
Where answer quality really comes from
Debates tend to focus on which LLM writes the smoothest final answer, but the real determinant sits upstream of the model: were the right chunks retrieved at all? A brilliant model given the wrong context produces a brilliant, confident, wrong answer every time.
Parsing puts data into the system, while retrieval determines whether the right slice of it reaches the model. When either is wrong, downstream prompt engineering can't rescue the answer. With dependable retrieval in place, the follow-up question is how to check automatically, end to end, that the system reasons properly over what it fetched, rather than inspecting outputs one at a time.