Reranking in RAG: The Narrow Window Where Cross-Encoders Actually Help

A cross-encoder helps a RAG pipeline only inside a narrow band: the first-stage recall has to be high enough, and the number of candidates you hand over has to stay small enough. Below and above that band, the reranker either has nothing to fix or starts doing harm. What follows covers the mechanism, a failure mode called Phantom Hits, and a way to measure all of it on your own data.

Picture a RAG system that found the right answer. There was no error and no lost document. The answer simply sits at position 37 in the result list, while the LLM reads only the first five, so for practical purposes it doesn't exist. No alert fires, since by every monitoring measure the request succeeded.

That isn't a quirk of one deployment. It follows from how the architecture is built. Vector search is designed to drop the right answer somewhere among the top 50 to 200 candidates, not necessarily at the very top, so a pipeline that ends at retrieval leaves answer quality to chance.

The usual reflex is to bolt on a reranker. That means a cross-encoder, roughly twelve lines, a small nDCG bump, and the ticket gets closed. The reflex isn't wrong because reranking fails. It's wrong because reranking works inside a narrow operating window that hardly anyone measures. Below, you'll see how this pricier second stage operates, where each edge of the window lies, and how to learn in one afternoon which side of each edge your system sits on.

Reranking is not better vector search

A reranker is a different architecture, switched on only after retrieval and applied to a small pool of candidates.

The bi-encoder behind your existing vector search encodes the query and each document independently, ahead of time and offline. That makes it quick and lets it scale to millions of documents. The price is that it never looks at the query and the document together.

The cross-encoder does the reverse. It reads both at once, token by token, which lets it notice negations, numeric contradictions and subtleties a bi-encoder can't see. In exchange, it must repeat that work for every candidate separately, at query time.

That gives the familiar two-stage pipeline. Retrieval narrows the field cheaply to 50 to 200 candidates and aims at recall: don't lose the right answer. Reranking then sorts that small pool at higher cost and aims at precision: of the survivors, choose the truly best.

The second layer isn't a free upgrade, either. Beyond a certain candidate count, the cross-encoder starts treating noise as signal and quality really does drop. Whether to add reranking is an engineering calculation to run on your data, not a reflex.

Why retrieval can't be the last step

A symptom that shows up constantly in production RAG sounds like this: "the model hallucinates, even though the information is in our database." In most cases that get traced to the end, the model wasn't inventing anything on its own. The correct document was inside the returned top fifty, just not inside the top five that the generator was shown.

The cause is architectural. The bi-encoder computes a document's embedding at indexing time, before any question exists. At search time it compares two compressed summaries rather than the real text of the query against the real text of the document.

A classic example of what gets lost: the query says "cheap" and the document says "$500 a night." To the vectors, both are about the price of a hotel room, so they land near each other. The contradiction becomes visible only when someone reads the two side by side, and a bi-encoder by design never does.

A better embedding model won't cure this, because it's a deliberate trade-off. Retrieval maximises recall, meaning the chance that the right document is somewhere in the returned pool, and does it cheaply across millions of documents. Pushing the right document to the very top simply isn't part of its brief.

Recall@50 and precision@5 measure different things and answer different questions. Mixing them up is how teams end up paying for a second layer to solve something it can't touch.

The weakness is that the bi-encoder never looks at the pair jointly, so let's see what a model that does look at the pair gains.

The bi-encoder catalogues, the cross-encoder reads

Think of hiring. The bi-encoder plays a recruiter who sorts a thousand résumés into folders before the job description even exists: quick, superficial, finished early. The cross-encoder plays the hiring manager with ten finalists, reading every résumé alongside the posting, line after line. Only at that point do the mismatches the recruiter couldn't see appear.

Mechanically the difference is plain. A cross-encoder joins query and document into one sequence, in the canonical layout [CLS] query [SEP] document [SEP], and passes it through a single transformer.

Inside, cross-attention does the work: every query token can attend directly to every document token and the other way around. The output is not a vector to compare but a single logit, a raw unnormalised number expressing how confident the model is that this particular pair is relevant.

So the bi-encoder yields a representation and the cross-encoder yields a verdict. That is a difference in what the model is able to see at all, not a difference of degree.

One consequence follows immediately. A cross-encoder score can't be precomputed and indexed, since it depends on the specific query and document pair rather than the document alone. Each query demands a full forward pass per candidate, tens to a hundred times costlier per pair than a vector comparison.

The figure that gave rise to the two-stage design makes the scale clear. Finding the most similar pair among 10,000 sentences takes about 50 million comparisons, which comes to roughly 65 hours with a full BERT model and around 5 seconds with an SBERT-style architecture, according to the 2019 Sentence-BERT paper from Reimers and Gurevych.

Remember it in its accurate form, because rounded variants circulate, such as "10 million documents and 50 hours." The difference is qualitative: the bottleneck is how many pairs must be scored, not how large the corpus is.

That's why a cross-encoder never serves as the first stage. It isn't worse, merely too expensive to run over a whole corpus. And since it's costly per pair, everything hinges on how many pairs you give it and where in the pipeline it sits.

Two stages, two budgets

A reranker belongs after retrieval and before the LLM. Search returns the top-k candidates, k being the number of documents one stage hands to the next, usually 50 to 200. The reranker then reshuffles that list and forwards the generator somewhere between 3 and 20 positions.

This cascade didn't originate with LLMs, which speaks to its maturity. The "cheap filter, then expensive judge" shape comes from the Learning-to-Rank era, when the second stage was an ensemble of decision trees over hand-built features instead of a transformer. The engine changed, the architecture didn't.

Neural reranking has a specific birthdate. In 2019 the paper "Passage Re-ranking with BERT" set BERT up as a binary classifier over query and passage pairs and improved MRR@10 by 27 percent relative to the best result then available. That's when the second stage stopped being an optimisation and became a layer of its own.

The best-known commercial implementation today is Cohere's flagship reranker. Across releases, what changed most was one setting. The input window went from a few hundred tokens to over 32,000, pausing at 4,096 for the v3 and v3.5 generation. This is a real design shift, since it raises the size of chunk that can be scored without truncation, so chunking can be less aggressive and less context is lost.

Vendors talk less about what comes next. Prices quoted for one and the same model span a factor of ten or more across outside sources, anywhere from a fraction of a cent to two and a half dollars per thousand searches, depending on who is counting and what they bundle into a "search."

A simple rule applies: if you can't reproduce the vendor's arithmetic on your own traffic, you don't have a calculation, you have a guess. There's also a billing trap. Documents longer than the limit are split into chunks, and each chunk is billed separately, so one logical reranking call can turn into several billed ones.

The shape to remember is this: retrieve cheaply at scale, rerank expensively over a narrow pool, and let the LLM see only what survives both. It sounds like a safe default, yet it has a threshold beyond which it stops working, in a direction hardly anyone expects.

Phantom Hits and the point where reranking starts to hurt

Conventional wisdom says the more candidates you give the reranker, the safer you are. The model is more accurate than vector search, so at K of 500 or 1,000 it surely can't miss. That intuition is wrong, and the evidence against it is documented.

The 2024 paper "Drowning in Documents" (Jacob and colleagues) reported that as K grows, reranker quality climbs, reaches a peak, then falls off visibly. When the whole collection was scanned blind, plain BM25 could beat a cross-encoder regarded as the best available. Keep one caveat in view: that result comes from blind scans on academic and enterprise datasets and says little about a typical pipeline where K is a few hundred at most.

The failure has a name, Phantom Hits. The reranker confidently gives its top score to a document with no connection to the query, neither lexical nor semantic.

The authors suggest a probable source and are careful to call it a hypothesis, not a confirmed cause. They point to the training setup, which they name exposure bias. Rerankers are trained contrastively on hard negatives, documents that look deceptively like the right answer but are wrong in substance, so the model becomes an expert in microscopic differences.

In production that expertise backfires. Shown an easy negative, plain noise it barely met during training, the model has no reference to hold onto and starts to guess. A model schooled on telling twins apart is helpless when a stranger walks in.

The opposite edge is just as real and doesn't undo the first. A few independent analyses place the lower break-even for reranking around recall@50 = 0.85. Under that figure there is nothing worth sorting, because the correct document never made it into the pool. Exact thresholds differ between sources since the corpora differ, and their provenance matters: blogs and field experience, not peer review. Use them as a hypothesis to test on your own data and never as established fact.

The two findings don't conflict. They are the two edges of one operating window. Under the recall threshold, reranking has nothing to work with. Over the K threshold, it begins to cause damage. Reranking is not a monotonic improvement, and assuming more is always safer repeats the mistake of trusting a vendor benchmark blindly, because both replace measurement with folklore.

Given that more reranking can hurt and too little may not suffice, we need a firm view of the ceiling on what this layer can fix.

The recall ceiling, and what the 67 percent really contains

A reranker is an editor, not a reporter. It can rearrange material somebody already collected, and it can't conjure what nobody brought in.

That principle, the recall ceiling, sounds trivial until it meets the industry's loudest statistic. Anthropic described a 2024 pipeline in which the top-20 retrieval failure rate fell from 5.7 to 1.9 percent, a 67 percent reduction, in its Contextual Retrieval write-up.

The number is often cited in support of reranking. Split by stage, it tells another story, though not because it's false.

The path went like this. Contextual embeddings lowered the rate to 3.7 percent, adding BM25 brought it to 2.9, and only then did reranking reach 1.9. Reranking alone therefore covers the final stretch, from 2.9 to 1.9 percent. Both numbers hold because they measure different scopes: the 67 percent is the result of the whole stack and not of any single layer.

That distinction has real operational weight. Before presenting "67 percent" at an architecture review, know that you're promising the effect of four changes together, and three of those come earlier in the chain than reranking and carried most of the gain.

The authors flag the limitation themselves: the figure averages across varied domains, and variance between them is large. One more caveat from the evidence available: every quality number for commercial rerankers is reported by the vendors, on their own datasets and with their own first-stage retrieval. No independent, market-wide evaluation exists.

No outside benchmark will tell you how reranking behaves in your system, so you need to test it yourself.

A calculation you can run on Monday morning

All the complexity of this topic reduces to one repeatable procedure. How rarely teams run it is why the "does reranking help" argument keeps going.

Step one: measure recall@50 and precision@5 on about a hundred labelled production queries. Use your own logs and your own mix of terms and intents, not a public benchmark. If recall@50 is clearly under 0.85, fix retrieval first, through chunking, hybrid search or embeddings, instead of adding a second layer on top.

Step two: look for the signature. High recall@50 combined with low precision@5 is precisely the profile reranking exists for. The right document is in the pool and just sits too low.

Step three: pick the model deliberately. A hosted API means a fast start and zero operations, but your data leaves your infrastructure. A self-hosted reranker gives control over cost and privacy, but you take on GPUs and upkeep. This is a compliance and budget decision more than a question of model quality.

Trying it out is cheap as well. Local reranking fits in a dozen lines or so:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-large", max_length=512)

def rerank(query, docs):
    pairs = [(query, d.page_content) for d in docs]   # pair: query + candidate
    scores = reranker.predict(pairs)                  # one logit per pair
    return sorted(zip(docs, scores), key=lambda x: x[1], reverse=True)

Three mistakes show up most often. One is inflating K "just to be safe." Another is reranking after the context has been truncated, so the model scores a prefix and not the document. The third is treating cross-encoder scores as absolute thresholds that compare across queries, which is especially treacherous because it works in a demo and falls apart in production.

There's a less obvious one too: "the reranker isn't helping" is sometimes a numerical bug and not an architectural verdict. A case documented in the Sentence-Transformers migration guide involved a sigmoid computed in bfloat16 that saturated, pushed the top scores together and effectively randomised their order. Once float32 was enforced ahead of the activation, the ranking recovered from near-random to normal.

Everything here needs only measurement, no architecture change.

What it means for the architect

Back to position 37. The document stuck there was never a problem of the generating model. It was a problem of ordering.

• A cross-encoder is not a better embedding. It is a separate architecture, triggered after retrieval, that reads query and document together and returns a verdict instead of a representation. No gain in embedding quality gives you what reading the pair side by side does.
• Reranking belongs in one place. Retrieval stays cheap and wide, reranking stays pricey and narrow, and the cost structure of that cascade is the whole idea.
• More candidates is not a safe default. Past some K, quality genuinely falls, and the explanation lies in how these models are trained.
• The ceiling is firm. What the first stage didn't return can't be recovered by a reranker, whatever a vendor slide claims.
• Your data is the benchmark. One afternoon spent computing recall@50 and precision@5 on a hundred labelled queries ends a debate that could otherwise drag on for weeks.

Many teams regard reranking as a standard fixture, added to every RAG system the way a cache is. That's a mistaken habit. The sounder picture appears once you see both thresholds together: a floor under which the layer has nothing to repair, and a ceiling over which it starts to damage results. In short, reranking is conditional, never automatic.

One gap stays open. In the material surveyed here, no reranker benchmarks were broken out for Polish specifically, and every multilingual number is an aggregate over dozens of languages. For a Polish-language corpus, or any smaller language, measuring on your own data isn't merely advisable. It's the only option.