Why Reranking Is the Cheapest Big Win in a RAG Pipeline
Retrieve 50, rerank to 5: how cross-encoder reranking fixes ordering in a RAG pipeline, the gains and latency to expect, when to skip it, and what to ship.
This article is published in English.
Even when a retriever finds the correct chunk, it may rank it so low that the model never sees it. A second scoring pass fixes that: pull a broad candidate set, rescore it with a stronger model and give the LLM only the winners.
Two jobs hiding in one pipeline
Retrieval in a RAG system does two separate things. It has to find the right passage, and it has to place that passage high enough for the model to use it. Teams tend to pour effort into the first and neglect the second, which is how retrieval metrics can look excellent while answers stay disappointing.
The model only sees what goes into the prompt. A correct chunk sitting at rank 7 is lost if just the top 5 get sent, and for practical purposes it might as well be at rank 700. That's why Recall@50 can look wonderful while answers are wrong: the information was found but never reached the context window.
Reranking closes that gap. As a second-stage pass, it rescores candidates the retriever already found and reorders them by how well they answer the query. For many systems it ranks among the best quality gains per unit of effort, since the architecture stays put and the rest of the stack is untouched.
Why the two stages differ
A dense retriever embeds every document in advance. At query time it embeds the query and looks for nearby vectors. Query and document never actually meet, since each is squeezed into its own vector before comparison, and that compression discards fine detail about how well they match. The payoff is speed, cheap enough to scan millions of documents.
A cross-encoder works differently. It feeds the query and a single document through the model jointly, as one input, and runs a complete forward pass on that pair. Since both texts are visible together, negations, qualifiers and the specific wording that makes a passage truly answer a question can all be weighed, instead of just a loose resemblance.
That precision has a cost. Nothing can be precomputed, so there's one pass per candidate at query time. Across a million documents that's hopeless, while across the 50 the first stage already selected it's quick and practical. Hence the two-stage idea: retrieve cheaply and widely, then rerank precisely and narrowly.
Cohere's guidance, OpenAI's RAG cookbooks and most production stacks all follow roughly the same recipe, in three steps:
- Retrieve wide. Take the top 50, often from hybrid search where dense and BM25 results are merged with reciprocal rank fusion. A broad net raises the odds that the right chunk is included.
- Rerank. A cross-encoder scores every one of the 50 query-document pairs, then the list is sorted by that score.
- Keep five. Send only the sharpest few to the LLM.
The rule of thumb that survives benchmarks is short: retrieve 50, rerank to 5. The first stage owns recall by getting the right chunk into the pool, and the reranker owns precision by lifting it to the top. Nothing needs re-embedding, no index is rebuilt and the retriever stays as it was, which explains why this fix is so contained and so rewarding.
What it buys and what it costs
Accuracy gains are consistent and large enough to matter:
- +5 to 15 NDCG@10 is typical across MTEB and BEIR benchmarks.
- 40 to 60% higher precision@3 shows up on noisy retrievers, the situation reranking targets.
- 35% fewer hallucinations appeared in Databricks testing against raw embedding similarity, because better-grounded chunks leave the model less room to invent.
Latency does go up, but the increase is minor relative to the full pipeline:
- Scoring 50 candidates takes under 50 ms on a GPU.
- On a CPU, the same 50 take 100 to 300 ms when pairs go into one batched call.
- An ultra-light library such as FlashRank needs only 15 to 30 ms.
- The LLM completion still dominates time and cost, so reranking barely changes the bill.
Paying roughly 120 to 200 ms for a 40%-plus lift in top-3 precision is, in most applications, the best trade available anywhere in the stack.
When it helps and when it doesn't
Reranking has high returns but isn't universal, and the guiding idea is that it fixes ranking, never finding. It pays off when:
- A noisy top despite good recall. The right chunk shows up in the top 50 but sits at rank 6 or 12. Strong recall@50 with weak precision@3 makes the clearest argument for reranking.
- A narrow prompt budget. If only 3 to 5 chunks reach the model, ordering decides everything, and a bad top 3 yields a bad answer.
- Hard or ambiguous questions. Where meaning turns on a qualifier or negation, like invoices to CSV as opposed to reports to PDF, scoring the pair jointly tells apart near-misses that a bi-encoder smears together.
- The corpus is full of near-duplicates. Support and policy documents hold many look-alike passages that answer different questions, and a reranker tells them apart.
It disappoints when:
- Weak first-stage recall. A reranker can only reshuffle its input. When the right chunk is missing from the top 50, improve retrieval first, through better chunking or hybrid search.
- Recall that's already excellent. If the correct chunk usually sits at the top anyway, reranking shuffles an already correct list for little gain. A fast check: when the same chunk wins on most queries with or without the reranker, it isn't earning its keep.
- A forgiving generator. A long-context model that handles 20 loosely related chunks gets less out of a tight top 3 than a model fed just 5.
- A strict latency budget. With a hard real-time SLA, 150 ms might already be too much. Choose a quick reranker such as FlashRank, or none.
To tell which case applies, measure recall@50 and precision at the top separately. Recall shows whether retrieval finds the chunk, precision shows whether it ranks it, and reranking only repairs the second.
Cross-encoder or LLM reranker
Two families exist, and the choice is a straight accuracy-versus-latency trade.
- Cross-encoders are the default. They're fast, cheap, predictable and produce a numeric score that can be thresholded. For the large majority of production systems, they're the right and final answer.
- LLM rerankers ask a big model to judge relevance itself. On truly difficult, reasoning-heavy queries they may win by 5 to 8% on some tasks, but they tack on 4 to 6 seconds and cost far more. Searchers tend to give up after about 3 seconds, so these fit offline or high-value batch work, hardly ever interactive features.
Start with a cross-encoder. Consider an LLM reranker only after measuring a real gap on hard queries and confirming the seconds are affordable.
Two ways to ship it
Teams usually pick a self-hosted cross-encoder or a hosted API, and each takes a handful of lines.
Self-hosting through sentence-transformers works well with BGE-reranker-v2-m3, the usual open-source default, which is multilingual and Apache 2.0 licensed:
from sentence_transformers import CrossEncoder
# ~568M params, multilingual, Apache 2.0; load once at startup
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
def rerank(query, candidates, top_n=5):
pairs = [(query, doc) for doc in candidates] # one batched pass
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
return ranked[:top_n]
# candidates = the top 50 chunks from the retriever
best = rerank("How do I cancel my subscription?", candidates, top_n=5)
When running a model isn't desired, Cohere Rerank is the hosted option:
import cohere
co = cohere.ClientV2()
resp = co.rerank(
model="rerank-v3.5",
query="How do I cancel my subscription?",
documents=candidates, # the top 50 from the retriever
top_n=5,
)
# resp.results carries the reranked indices and relevance scores
Both routes take the same input, the retriever's top 50, and return a sharper top 5. Frameworks such as LangChain put either behind a shared interface, so changing between them is a one-line edit.
Picking a model
The field has settled into clear choices that depend on latency budget, language and whether self-hosting is possible:
- Open-source default: BGE-reranker-v2-m3, multilingual and Apache 2.0, at roughly 80 ms on CPU for 50 candidates. English-heavy work can use bge-reranker-large.
- Easiest hosted route: Cohere Rerank needs one API call and no servers, at about $1 for every thousand searches.
- Strictest speed target: FlashRank, or a compact MiniLM cross-encoder, reranks on CPU in under 30 ms at a modest quality cost.
- Lengthy documents: pick a long-context reranker like Jina Reranker v2, since standard cross-encoders quietly cut input off past 512 tokens.
As a loose guide, a hosted API is the cheapest place to begin below about ten thousand queries daily. Past that point a self-hosted BGE model tends to pay for itself fast.
The takeaway
Reranking supplies the precision that turns solid recall into solid answers. A retriever casts a cheap, wide net and returns a noisy haul, and a cross-encoder then rescores that haul with a query-aware pass the retriever couldn't afford, moving the genuinely relevant chunks to the spots where the model actually looks.
It's the highest-return RAG fix because it stays contained: retrieve 50, rerank to 5, change nothing else and measure the lift on your own data. The accuracy gain is usually large, the latency cost small, and the decision rests on one question: does the retriever rank as well as it finds? Usually it doesn't, and reranking is what closes that distance.
Sources behind the numbers
- Cohere's reranking documentation
- Model card for BAAI's BGE-reranker-v2-m3
- CrossEncoder docs from sentence-transformers
- ARAGOG, a graded comparison of advanced RAG methods including reranking (arXiv 2404.01037)
- Databricks evaluation of long-context RAG and reranking
- FlashRank, a low-latency reranking library on GitHub