Search and Judge for RAG: Why Retrieval Needs a Separate Reranking Stage

A lot of teams reach the same wall with retrieval-augmented generation. On a whiteboard the system looks tidy: embed the query, match vectors, hand the top results to the model, get an answer. In production the cracks appear quickly. Relevant documents are missed, irrelevant chunks sneak in, a slightly rephrased question changes the answer, and hallucinations persist even when retrieval seems to have worked.

The usual response is to swap in a better embedding model, raise the vector dimension, or fuss with top-k and chunk size. Those moves are reasonable, but they tune the wrong part. The deeper issue is architectural. A conventional pipeline asks one mechanism, embedding similarity, to do two quite different jobs: find candidates, and decide which candidates matter.

The remedy is to split the jobs. Call the pattern search and judge: a two-stage design where generating candidates and evaluating relevance are separate engineering problems with separate tools.

Why similarity alone falls short

Naive RAG converts the query to an embedding, pulls the top-K nearest chunks from the knowledge base, pastes them next to the question and lets the model answer. The unstated assumption is that closeness equals relevance, so the top-K list is trusted without any validation. Four things go wrong with that.

Similar is not the same as relevant. Embedding distance is a rough proxy for topical closeness. Two passages can sit near each other without one answering the question posed by the other. A text on treatment guidelines for hypertension and one on trial methodology for cardiovascular drugs may land side by side, yet only the first helps someone deciding how to treat a patient.

The ordering is weak. Vector distance is a heuristic and not a ranking function built so that position tracks relevance. The gap between result number three and number seven in a cosine ranking is frequently noise.

Context gets fragmented. Chunking breaks relationships, hierarchy and cause and effect. A dosage chunk can end up apart from the chunk describing contraindications, and neither is complete alone.

Results are unstable. Small rewordings can shift the top-K set. Asking about side effects of metformin versus adverse reactions to metformin may return different candidates, and the final answer can change with them. That is a reliability defect and not a tuning detail.

In short, search provides candidates. It does not provide correctness.

The two-stage design

Instead of one system finding and evaluating, build two stages, each tuned for its own goal.

Stage one, search, casts a deliberately wide net. Pull something like 50 to 200 candidates, many more than you would ever show the model, because the goal here is recall. The right documents only need to be somewhere in the pool, even if the pool is noisy.

Stage two, judge, applies a much stronger test. It is typically a cross-encoder reranker that reads the query and each candidate together and computes a genuine relevance score. It shrinks the pool to five to ten high-confidence passages. Only then does the assembled context go to the language model.

What embeddings actually contribute

Misreading the role of embeddings is what makes teams lean on them too much. Picture a librarian who does not read every book but tags each one with a few key themes. When a question arrives, the librarian converts it to similar tags and pulls the books whose tags match best. That is quick and usually finds something useful, but it runs on surface themes and not deep understanding, so some of the books are related without being what you need.

The consequence: your embedding model is your search behavior. It decides what gets found and what gets missed. It does not decide what is correct, and that is the judge stage's role.

Picking an embedding model is therefore a recall decision. You are choosing the lens through which the first stage views the corpus. Weigh retrieval benchmarks such as MTEB and BEIR, language coverage, model size against your hardware, vector dimension, maximum input length and inference speed. All of these are recall concerns.

Judge options and when each fits

The judge is a role in the architecture. Several implementations can fill it, with different trade-offs in accuracy, latency, complexity and cost.

Reciprocal Rank Fusion

RRF doesn't evaluate relevance at all. It merges ranked lists from several sources, for example a keyword search and a semantic search, into one ordering. For each document it adds 1 / (k + rank) across all lists, where k is a constant, commonly 60, that softens the dominance of the very top ranks. Documents that rank well in several lists rise.

It is fast, deterministic, needs no model inference and adds almost no latency. Because it uses rank positions and not raw scores, it sidesteps the mismatch between BM25 scores and cosine similarities, so no normalization is needed.

Its limit is that it only knows agreement. If several retrievers consider a document important, RRF assumes it probably is. That is a sensible heuristic but still a heuristic. It can't rescue a relevant document that every retriever missed, and it can't separate a topically adjacent document from one that answers the question.

Use it for hybrid search when you need a light way to merge lists. It belongs inside the search stage as a way to improve candidates, and it does not replace the judge.

Cross-encoders

A cross-encoder feeds the query and one candidate through the model as one joined sequence and returns a relevance score. Bi-encoders embed query and document apart from each other, whereas here every token of the question can attend to every token of the passage. The model can therefore reason about how specific phrases in the question relate to specific phrases in the text. Typical examples are Sentence Transformers models such as cross-encoder/ms-marco-MiniLM-L-12-v2, hosted rerank services and cross-encoders fine-tuned on medical, legal or scientific data.

Their strength is accuracy. On ranking benchmarks they typically beat the other approaches, because they capture fine-grained interactions that embedding similarity can't. That is the nearest thing to asking whether a document really answers the question without a full LLM call.

The cost is a separate forward pass for each query and candidate pair. A pool of 100 documents means 100 model runs per query, so latency grows linearly with pool size, and larger models make it worse.

They are the default judge wherever answer quality comes first: healthcare, legal, compliance, financial analysis, any place where a wrong retrieval has consequences. If the latency budget allows roughly 50 to 200 milliseconds of reranking, which most non-real-time systems can afford, start here.

ColBERT and late interaction

ColBERT, short for contextualized late interaction over BERT, sits between bi-encoders and cross-encoders. It encodes query and document independently into per-token vectors, then scores them by matching query tokens against document tokens. Because document vectors can be precomputed, it is much cheaper at query time than a cross-encoder, while keeping more detail than a single pooled vector. The price is a larger index, since every token gets a vector. It suits systems that need better ranking than plain embeddings with higher throughput than cross-encoders can give.

An LLM as judge

A general language model can rate candidates directly. It handles nuanced, multi-hop relevance, such as seeing that a document about drug interactions bears on a question about treatment safety although the vocabulary doesn't overlap. It can also follow explicit relevance criteria in the prompt, which makes it adaptable to domain-specific meanings of relevant.

It is also by far the most expensive option. One reranking call might take one to five seconds and cost ten to a hundred times more than a cross-encoder pass. Scores can vary between runs, and models show position bias, favoring documents shown earlier in the prompt. Hosted models add an external dependency.

It makes sense in two settings: offline or batch jobs where latency is irrelevant, such as building curated knowledge bases, evaluating retrieval during development or generating training data for a dedicated reranker; and high-stakes, low-volume tools where the value of a correct answer justifies the per-query cost, such as expert-level medical or legal research.

Choosing models for each stage

Understanding the pattern is one thing and choosing models is another. The architecture gives you two selection decisions, each driven by the task.

Start with the task

Clinical question answering, document exploration and regulatory compliance need different things. QA demands high precision at small K. Exploration rewards breadth and diversity. Compliance needs deterministic, auditable retrieval with minimal false negatives. The task profile shapes every later choice.

The embedding model sets your recall ceiling

It determines what the system can find before the judge acts. A relevant document that never enters the pool can't be recovered by any reranker. Compare candidates on these points:

• Benchmarks. Look at MTEB and BEIR results for the task type closest to yours, such as passage retrieval or question answering, and not just the aggregate score.
• Languages and domain. Multilingual models trade some monolingual precision for breadth, while domain-tuned models do the reverse. Check coverage of your terminology.
• Size and infrastructure. Bigger models usually embed better but cost more at scale. Think about indexing volume, query throughput and GPU versus CPU. A model three percent better on a benchmark that needs twice the compute may be a poor trade.
• Vector dimension. More dimensions capture more nuance but raise storage and latency. Many models offer several sizes, so test whether 768 versus 1536 dimensions justifies the overhead.
• Maximum input length. This limits your chunking. A 512-token cap forces small chunks and more fragmentation, while 2048 or 8192 tokens preserve more context per chunk.
• Speed. Embedding latency matters at index time and query time. Smaller distilled models can give good recall at far lower latency when SLAs are tight.

The principle: tune the embedding model for coverage and speed and not precision. Precision is the judge's job.

The reranker sets final correctness

It is the last filter before the LLM, so its quality feeds directly into answer accuracy. Assess it on:

• Ranking accuracy. The main metric. Check nDCG, MRR and MAP on benchmarks close to your domain. Small gains here can noticeably reduce downstream hallucination.
• Latency. The reranker runs once per candidate, so a pool of 100 means 100 passes. Per-pass time and total time are hard production limits.
• Domain alignment. General rerankers do well on broad retrieval, but high-stakes domains often profit from rerankers fine-tuned on domain relevance data. A reranker trained on medical literature will judge clinical queries better than a general one even if the general one scores higher on generic benchmarks.
• Throughput. Under heavy traffic the reranker can be the bottleneck. Consider batching, model parallelism and whether a lighter option, a distilled cross-encoder or a late-interaction model, gives acceptable accuracy at much higher throughput.

Invest in both

The strongest pipelines don't pick between a good embedder and a good reranker, because the two do non-overlapping jobs. A strong embedder with a weak reranker gives high recall and noisy final results. A weak embedder with a strong reranker gives precise ordering over an incomplete pool. Neither suffices alone.

The recipe is fast, high-recall search plus strong, precise judgment. Match the embedder to your recall needs and the reranker to your precision needs, then tune the candidate pool size between stages to trade coverage against reranking latency.

A worked example

Take a clinical assistant asked whether a patient on a particular diabetes drug can safely take a second medication. The search stage runs a keyword query and a vector query in parallel, merges them with RRF and keeps 100 candidates. Among them are the monograph for each drug, a trial-design paper, a general nutrition leaflet and several interaction tables. Similarity alone might rank the trial paper and the leaflet above the interaction table, because their vocabulary is closer to the question.

The judge stage reads the question next to each of the hundred passages and scores whether the passage really addresses combined use. The interaction table and the relevant contraindication paragraph climb to the top, the leaflet falls, and the model receives six passages instead of a noisy ten. Same corpus, same embeddings, different outcome, because a second component made a decision the first was never built to make.

Tuning the handoff between stages

The candidate pool size is the main dial. Too small and the judge never sees the right passage, which caps recall for the whole system. Too large and reranking latency and cost climb for little gain. A practical way to set it is to log, on a labeled sample, the rank at which the correct passage appears in the first stage, then choose a pool that covers nearly all of them. If the correct passage regularly sits beyond rank 150, the fix is probably in the search stage, with a better embedder, hybrid retrieval or improved chunking, and not a bigger reranker.

Other habits help as well:

• Evaluate the two stages separately. Measure recall of the pool before judging, and precision of the final list after judging, so that a regression can be traced to the right stage.
• Cache reranker scores for frequent queries when the corpus changes slowly.
• Keep the final context small. Passing five strong passages usually beats passing twenty mixed ones, since extra material competes for attention.
• Revisit the setup when the corpus or query mix shifts, because a reranker tuned on one domain can quietly degrade on another.

Treat this as a hypothesis to test on your own data and not a guarantee, since gains differ by domain and by how weak the baseline was.

The takeaway

Reliable RAG isn't achieved by finding a better embedding. It comes from separating search from judgment, with one stage maximizing coverage and the other maximizing correctness.

The shift is conceptual before it is technical. Stop treating top-K as a final answer and treat it as a candidate set. Stop treating similarity as truth and treat it as a hypothesis that needs checking. Search wide, then judge precisely.