A RAG System Can Work and Still Rank Wrong Evidence First
Dense and hybrid search both scored a perfect Recall@5, yet MRR caught RRF demoting one correct hit. Why a RAG eval needs per-query checks, not one average.
This article is published in English.
A retrieval pipeline can return green numbers and still put the wrong evidence first. On a 22-query bench, dense search and hybrid search fused with Reciprocal Rank Fusion (RRF) both posted Recall@5 of 1.0. Mean reciprocal rank told another story: 0.9545 for dense against 0.8977 for hybrid. Nearly all of that gap came from a single paraphrase query, q006. Fusion pushed a correct dense-only hit from first place down to fourth, and no value of the RRF constant k between 0 and 100 put it back. The practical lesson was not a magic setting. It was building a habit that catches that kind of regression instead of trusting one summary score.
Snapshot of the bench
Three retrievers ran on the same labeled set: BM25 for lexical match, dense embeddings from all-MiniLM-L6-v2, and hybrid fusion with RRF at k=60. Dense and hybrid both found every gold chunk somewhere in the top five. Their MRR scores did not match. q006 drove most of the drop. Sweeping k never restored the dense ranking.
Recall@5 said the two methods were tied. MRR said they were not.
Reading the scores
BM25 landed at Recall@5 0.9545 and MRR@5 0.8818. Dense hit 1.0 / 0.9545. Hybrid hit 1.0 / 0.8977. Side by side, dense and hybrid look equally successful if you only ask whether the labeled chunk appeared in five results.
That "somewhere in five" is the trap. Recall asks whether the gold item is present. MRR asks how early the first useful item appears. Burying the answer at rank five and placing it at rank one look identical on Recall and feel very different to a user or an agent that only reads the top of the list.
Short definitions make the contrast concrete:
def recall_at_k(gold_id: str, ranked_ids: list[str], k: int = 5) -> int:
"""1 if the gold chunk appears anywhere in the top k, else 0."""
return int(gold_id in ranked_ids[:k])
def reciprocal_rank(gold_id: str, ranked_ids: list[str]) -> float:
"""1/rank of the gold chunk's first appearance, 0 if absent."""
for rank, doc_id in enumerate(ranked_ids, start=1):
if doc_id == gold_id:
return 1.0 / rank
return 0.0
First takeaway: a green metric can answer the question it was built for and still hide the question you care about.
Compare query by query
Rather than tuning hybrid at once, line dense and hybrid up side by side. Of 22 answerable questions, eighteen kept the same gold rank. Hybrid helped one. It hurt three. The worst case was q006. Dense put the labeled passage first. Hybrid parked it fourth. That single case became the investigation.
Why fusion demoted a correct hit
Hybrid merges rank positions instead of averaging BM25 scores with cosine similarities, because those two number systems do not share a scale. RRF therefore adds contributions from rank alone. Sensible design, real trade-off. On q006, dense strongly preferred the labeled chunk, yet other candidates sat high on both lists. Those dual-list candidates collected credit from BM25 and from dense, and the dense winner slid to fourth.
A compact fusion function:
def reciprocal_rank_fusion(
rankings: dict[str, list[str]], k: int = 60
) -> list[str]:
"""Fuse multiple ranked lists using rank position only."""
scores: dict[str, float] = {}
for ranking in rankings.values():
for rank, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
Agreement across retrievers is a clue. It is not proof of correctness. That framing is more useful than "hybrid MRR fell." There is now a mechanism to inspect, not only a dashboard wobble.
Did knobs fix it?
The work habit is familiar: change one variable, measure again. Holding depth and top-k fixed and sweeping k through 0, 10, 20, 40, 60, 80 and 100 left Recall perfect. MRR shifted only a little, from about 0.91 at k=0 down toward 0.90. q006 moved from second to third to fourth and never returned to first. Widening the candidate pool from 5 to 30 with k fixed at 60 kept Recall perfect and hybrid MRR near 0.90, still short of dense. In this bench the obvious knobs did not erase the ranking gap.
From toy numbers to production habits
Businesses rarely care about the name of a retrieval architecture. They care whether answers are reliable, whether a change improves quality and whether someone can explain a regression. That pushes the loop away from "build, try a few prompts, ship" toward measure, slice, inspect, explain, change, measure again.
The useful artifact is not one score. It is the per-query order, the chunks that came back, where retrievers disagreed, slices by question type, concrete failure cases and the config that produced them. When a dashboard number moves, that package gives an engineer a place to start.
At larger scale, Uber has described a similar separation in Genie, its on-call copilot: retrieval quality and answer quality are measured apart against a golden set built with subject-matter experts. One detail sticks: bad PDF extraction can destroy table context and chunking before retrieval even runs. What looks like a ranking bug can begin as a document-quality bug.
Labels can be wrong too
The evaluator only knows the relevance labels it was given. A chunk can be genuinely useful and still count as a miss because it is not the labeled gold item. Overlapping chunks make that messier. When a query regresses, inspect both the retriever and the bench. Is the relevance set complete? Could two chunks answer the question? Did a chunk boundary cut a claim away from its explanation? Evaluation tools do not replace judgment. They give judgment a better place to work.
Five keepers
- Perfect Recall@5 is not a perfect ranking. Add MRR, or you only know the answer exists somewhere.
- Dig into the query, not only the average. One bad case can move the whole score.
- Retriever agreement is a clue, not a verdict. RRF rewards dual-list popularity; that is a heuristic.
- Twiddling a knob is not understanding a mechanism. The k sweep proved the regression was real; only the fusion math explained it.
- Scrutinize labels the way you scrutinize the model. A "wrong" hit may be a mislabeled gold chunk.
Closing
The win from a small bench is rarely crowning dense the champion. It is a failure like q006 that forces a path from a metric to a query to the chunks to the fusion math to a controlled experiment. Evaluation earns its keep when the number opens an investigation instead of ending one. A demo proves the pipeline runs. A good loop proves you can trust the next change.
Links
- RAG Systems Evaluation Bench on GitHub
- Live demo on Hugging Face Spaces
- Cormack, Clarke and Buettcher on Reciprocal Rank Fusion
- Uber also published an engineering post on enhanced agentic RAG.
- Uber's Genie write-up is on the company engineering blog under the Genie title.