Reranking in RAG: Cross-Encoders vs LLM Judges and Their Latency

The right chunk was in the results, just too low

Picture a RAG system whose hybrid search has finally started behaving. The vector store is up, the keyword index works, the fusion logic is written, and the chunk that holds the answer now shows up somewhere in the candidate set. Even so, the model still produces a mediocre reply.

The logs explain why. A developer asked how to debug a PAYMENTS_API_TIMEOUT error in staging. Hybrid search handed back twenty candidates: a broad authentication overview, an out-of-date incident note from the previous year, a generic page on retry logic, and, at rank eight, the staging runbook chunk with the exact fix. The prompt only had space for three chunks. The right one sat too far down, so the model built its answer from the wrong material because the right material didn't fit.

Retrieval found the evidence but didn't put it first. Retrieval is about recall, while reranking is about precision under a hard context budget.

A good candidate set isn't enough by itself. What the model actually reads depends on the order of that set, and the order is the bottleneck.

What a reranking stage does

The first pass is built to be quick and wide. A query against a vector database or keyword index has to cover millions of records in milliseconds, so these systems rely on approximations and precomputed indexes, and they retrieve more than they need on purpose.

Reranking adds a second stage that picks up a small subset of results and studies each one more carefully. It's slower and more precise, and it never touches the database. Whatever the first pass returned, usually the top twenty or fifty, gets rescored by a heavier, more accurate model.

The first pass produces candidates, and reranking decides which of them earn a place in the limited context window.

Why the first pass can't finish the job

If the embedding model is good, shouldn't the best chunk simply land on top? Not reliably, because first-pass retrieval is noisy by construction.

Keyword scoring such as BM25 depends on exact term overlap. A broad reference page that mentions an error code fifty times will often beat a short step-by-step runbook that mentions it twice, since the algorithm reads frequency as relevance.

Vector retrieval is noisy in another way. Embeddings measure semantic closeness by placing concepts in a high-dimensional space. For a troubleshooting question, they surface chunks related to the systems named. Broad overviews tend to sit very near specific procedures in that space. A chunk describing the billing architecture might outscore the exact command that restarts the billing worker. The model knows both concern billing but can't tell an operational instruction from a theoretical description.

Hybrid retrieval merges the lexical and semantic lists into a comprehensive candidate set that includes the needed evidence. It also includes duplicates, stale documents, broad summaries and chunks that are only partly relevant.

First-pass retrieval should never be treated as the last ranking layer. It works like a funnel, and reranking is the filter at the bottom.

Two ways to rerank

Once candidates exist, something has to order them. Two approaches have become standard. One uses cross-encoders, specialized models trained to score how relevant a document is to a query. The other uses large language models, where a model such as Claude or GPT receives the query and document and is asked how useful the document is.

Both raise ranking quality a lot compared with the first pass. They differ in latency, cost, determinism and the engineering they demand.

Cross-encoders

A cross-encoder is a transformer, though it works differently from the embedding models in a vector database.

A standard embedding model passes the query through the network to get one vector, passes the document through to get another, and compares the two with cosine similarity. This is the bi-encoder design, and it's fast because every document vector can be computed ahead of time.

A cross-encoder doesn't produce separate vectors. It joins the query and the document into one string and runs the whole thing through the network at once. Self-attention can therefore look at query words and document words together at every layer, seeing how the terms relate in context. The output is a single relevance score.

This design shines at fine-grained relevance decisions. Results are predictable and stable, with far less spread than you get from prompting a language model, and it reliably prefers a specific operational chunk over a broad summary.

Here's a reranking function built on a hosted cross-encoder API.

import cohere
import time

def rerank_cross_encoder(
    query: str,
    candidates: list[dict],
    top_n: int = 5,
    model: str = "rerank-v4.0-pro",
) -> list[dict]:
    """
    The practical default for second-stage ranking.
    Passes the query and candidate texts to a dedicated cross-encoder model.
    """
    co = cohere.ClientV2()

    # Extract just the text content for the API call
    documents = [c["content"] for c in candidates]

    resp = co.rerank(
        model=model,
        query=query,
        documents=documents,
        top_n=top_n,
    )

    # Reattach the original metadata and the new score
    reranked = []
    for r in resp.results:
        original_chunk = candidates[r.index]
        reranked.append({
            **original_chunk,
            "rerank_score": r.relevance_score
        })

    return reranked

In production, a cross-encoder suits the top ten to fifty candidates. Teams reach for it by default in documentation search for engineers, support knowledge bases and in-house copilots.

The main cost is speed. A cross-encoder can't scan an entire database. It waits for the first pass, receives the candidates through an API or a local model, and then waits for scores, adding a network hop and a computation step.

LLM rerankers

The alternative is a general-purpose language model acting as a judge. A strict prompt holds the user's query and one candidate chunk, and the model returns a score.

LLM rerankers cope well with subtlety. A cross-encoder hunts for deep semantic relevance, whereas an LLM can also check whether a procedure is complete, reason through contradictory evidence and separate chunks with real steps from ones with only background. The prompt can tell it to penalize vagueness or favor chunks with code samples.

This judge prompt uses Anthropic's client.

import anthropic

JUDGE_PROMPT = """\
You are a strict relevance judge. Given a user query and a candidate document chunk,
rate how well the chunk answers the query on a scale of 0 to 10.

Respond with ONLY a JSON object in this exact format:
{"score": <int>, "reason": "<one short sentence>"}

Query: {query}
Chunk: {chunk}"""

async def rerank_llm(
    query: str,
    candidates: list[dict],
    top_n: int = 5,
) -> list[dict]:
    """
    Expensive special forces. Uses an LLM to reason about nuance and completeness.
    """
    client = anthropic.AsyncAnthropic()
    scored = []

    for c in candidates:
        resp = await client.messages.create(
            model="claude-opus-4-6",
            max_tokens=128,
            messages=[{
                "role": "user",
                "content": JUDGE_PROMPT.format(query=query, chunk=c["content"]),
            }],
        )

        import json
        try:
            result = json.loads(resp.content[0].text)
            scored.append({
                **c,
                "rerank_score": result["score"],
                "reason": result.get("reason", "")
            })
        except (json.JSONDecodeError, KeyError):
            # Fallback if the model fails to follow JSON instructions
            scored.append({**c, "rerank_score": 0, "reason": "parse_error"})

    # Sort by the LLM-assigned score descending
    scored.sort(key=lambda x: x["rerank_score"], reverse=True)
    return scored[:top_n]

The model is asked to give a reason along with the score. Writing a justification first works like chain-of-thought and steadies the final number.

They earn their keep on high-value queries, in low-throughput systems where answer accuracy justifies a wait, and on complex corpora where relevance turns on policy nuance and not mere topic match.

The costs are steep. A full LLM call is made for every candidate chunk, and even when the calls run concurrently, the slowest response gates the result. Cost is well above a cross-encoder's, and the output is less deterministic.

What reranking does to latency

An added stage changes how the system performs. It brings extra model calls, extra serialization of text payloads and extra network time.

Users don't care how retrieval is arranged and don't feel recall metrics. They notice answer quality and response time. A theoretically excellent reranker is useless if it makes the assistant too slow to trust.

This helper times a reranking function.

def rerank_with_timing(
    rerank_fn: callable,
    query: str,
    candidates: list[dict],
    top_n: int = 5,
) -> tuple[list[dict], float]:
    """
    Measure the exact cost of the reranking stage.
    """
    t0 = time.perf_counter()

    results = rerank_fn(query, candidates, top_n)

    latency_ms = (time.perf_counter() - t0) * 1000
    return results, latency_ms

Passing twenty candidates to a fast cross-encoder API might add 150 to 300 milliseconds, usually a fair price for better context. Passing fifty to an LLM judge might add two to four seconds. With three more seconds for generation, the user waits seven seconds, which is past what's acceptable in a chat interface.

The number of reranked candidates needs tuning. The top ten is quick but can miss the right chunk when the first pass is struggling. The top fifty catches nearly everything and consumes the whole latency budget. For a cross-encoder in production, the top twenty is usually the sweet spot.

When reranking pays off

Add a reranker when first-pass retrieval is noisy but usually finds the evidence.

If traces show the exact evidence chunk often appears in the candidates but at rank ten or fifteen, reranking will repair the system right away. It's justified when broad documents routinely beat precise procedures, and when answer quality depends heavily on having the very best three chunks in the prompt, so precision outweighs raw recall.

Runbooks, internal policy assistants and troubleshooting tools almost always need a reranker. Picking the right command instead of a near miss can decide whether an incident gets resolved or turns into an outage.

When reranking is overkill

With a tiny corpus, skip it. A knowledge base of fifty pages can be embedded whole and served by vector search.

It's also pointless when the first pass is already near-perfect. In a lookup-heavy system where BM25 reliably puts the right API reference at rank one, a cross-encoder only slows things down without changing the result.

A very tight latency budget, like a voice assistant or inline code completion, leaves no room for another stage. Tune the first pass and accept some noise. And when the right chunk usually lands at rank one or two already, ranking isn't what holds you back.

Failure modes to watch

Rerankers fix ordering and bring their own problems. Watch for these after deployment.

Broad-summary bias. Some cross-encoders are trained on data that rewards comprehensive answers. A generic overview can outscore a small, exact code snippet because the overview shares more vocabulary with the query.

Duplicate crowding. If the corpus holds three versions of one deployment guide, the first pass will probably fetch all three. The reranker sees each answers the query well and scores them all high. The top three slots in the context window then carry the same information, squeezing out the next distinct piece of evidence. A helper like this removes near-duplicates.

def dedupe_candidates(
    candidates: list[dict],
    similarity_threshold: float = 0.85,
) -> list[dict]:

    seen_tokens: list[set[str]] = []
    deduped = []

    for c in candidates:
        tokens = set(c["content"].lower().split())
        is_dup = False

        for s in seen_tokens:
            # Calculate simple Jaccard similarity
            overlap = len(tokens & s) / max(len(tokens | s), 1)
            if overlap >= similarity_threshold:
                is_dup = True
                break

        if not is_dup:
            deduped.append(c)
            seen_tokens.append(tokens)

    return deduped

Stale-document favoritism. A cross-encoder reads text only and has no idea what year it is. A polished 2022 runbook can outscore a hurried 2026 one even though the older document is completely deprecated. Metadata boosts applied after reranking adjust scores for freshness and document status.

from datetime import datetime, timezone

def apply_metadata_boost(
    candidates: list[dict],
    freshness_halflife_days: int = 90,
) -> list[dict]:

    now = datetime.now(timezone.utc)
    boosted = []

    for c in candidates:
        score = c.get("rerank_score", 0.0)

        # Hard penalty for superseded documentation
        if c.get("status") == "superseded":
            score *= 0.4

        # Gradual decay for older documents
        updated = c.get("updated_at")
        if updated:
            age_days = (now - updated).days
            decay_factor = max(0.5, 1 - age_days / (freshness_halflife_days * 2))
            score *= decay_factor

        boosted.append({**c, "rerank_score": score})

    # Sort again based on the adjusted scores
    boosted.sort(key=lambda x: x["rerank_score"], reverse=True)
    return boosted

Latency explosion. Quality improves after deployment, but now and then the system stalls for five seconds. This usually happens when the candidate count spikes or the external reranking API degrades. Always include a timeout fallback that skips the reranker and returns first-pass results when the second stage runs long.

Evaluating a reranker properly

Teams often measure the wrong thing, checking whether the correct chunk appears anywhere in the top twenty. First-pass retrieval already covers that.

The better question is whether the reranker lifted the right chunk high enough to matter. That means measuring top-1 and top-3 relevance, answer completeness and duplicate suppression, plus tracking latency overhead and cost per query.

Build an evaluation harness that compares the plain first-pass baseline with the cross-encoder and the LLM reranker, using identical candidate sets.

from dataclasses import dataclass

@dataclass
class RerankEvalCase:
    query: str
    expected_substring: str
    category: str  # e.g., "identifier", "procedure", "troubleshooting"

def eval_reranking(
    cases: list[RerankEvalCase],
    retrieve_fn: callable,
    rerank_fns: dict[str, callable | None],
    top_n: int = 3,
) -> dict:
    """
    Compare multiple reranking strategies against a baseline.
    Measures hit rate at top-N and tracks latency overhead.
    """
    results = {}

    for name, rerank_fn in rerank_fns.items():
        hits = 0
        total_latency = 0.0
        by_type: dict[str, dict] = {}

        for case in cases:
            # Get the exact same starting candidates for every strategy
            candidates = retrieve_fn(case.query)

            if rerank_fn is not None:
                reranked, lat = rerank_with_timing(
                    rerank_fn, case.query, candidates, top_n
                )
                total_latency += lat
            else:
                # Baseline: just take the top-N from first-pass retrieval
                reranked = candidates[:top_n]

            # Check if the expected evidence made it into the final prompt window
            top_contents = [r["content"] for r in reranked]
            found = any(case.expected_substring in c for c in top_contents)
            hits += int(found)

            # Track metrics by query category
            by_type.setdefault(case.category, {"hit": 0, "total": 0})
            by_type[case.category]["total"] += 1
            by_type[case.category]["hit"] += int(found)

        total = len(cases)
        results[name] = {
            "hit_rate": hits / total if total else 0,
            "avg_latency_ms": total_latency / total if total else 0,
            "by_type": {
                t: {**v, "rate": v["hit"] / v["total"]}
                for t, v in by_type.items()
            },
        }

    return results

The test set should span several query types: identifier-heavy ones where the exact string counts, procedure-following ones where step order counts, and troubleshooting ones where symptoms point to root causes.

The numbers then show exactly what the reranker buys. If a cross-encoder lifts the top-3 hit rate from 60% to 85% for 200 milliseconds, ship it. If the LLM reranker reaches 88% for three seconds, the cross-encoder probably stays. Reranking has to be justified by better final answers within a realistic latency budget.

A sensible default stack

A reliable baseline for a production system follows a specific sequence. Split documents with structure-aware chunking. Use hybrid retrieval to collect twenty to thirty candidates. Deduplicate to drop near-identical text. Score the rest with a cross-encoder for semantic relevance. Apply a metadata boost that penalizes stale documents. Then take the top three to five chunks from the final sorted list and build the prompt.

def two_stage_retrieve(
    query: str,
    retrieve_fn: callable,
    top_k: int = 20,
    top_n: int = 5,
) -> tuple[list[dict], dict]:
    """
    The complete production pipeline for second-stage ranking.
    """
    t0 = time.perf_counter()

    # Stage 1: Fast hybrid retrieval
    candidates = retrieve_fn(query)[:top_k]

    # Clean up the candidate pool
    candidates = dedupe_candidates(candidates)

    # Stage 2: Cross-encoder rerank
    # We score slightly more than top_n to allow metadata boosts to reorder the edges
    score_limit = min(top_n * 2, len(candidates))
    reranked = rerank_cross_encoder(query, candidates, top_n=score_limit)

    # Apply business logic for freshness and status
    final = apply_metadata_boost(reranked)[:top_n]

    latency = (time.perf_counter() - t0) * 1000

    trace = {
        "query": query,
        "first_pass_count": len(candidates),
        "post_rerank_count": len(reranked),
        "final_count": len(final),
        "latency_ms": latency,
    }

    return final, trace

The default works because each tool does the job it suits. Retrieval handles candidate recall, reranking handles candidate order, and cross-encoders give the best quality-to-latency ratio for that second stage.

One caveat: a reranker shouldn't be bolted on to patch broken data. With bad chunking, it will just sort bad chunks accurately. With missing metadata, it can't guess which document is deprecated. Fix ingestion and extraction first.

What reranking still leaves open

A reranker makes sure the strongest evidence in the retrieved set reaches the model. What it can't tell you is whether a vector search was the right move at all. Take a question about how many users are active in a particular region: pointing it at a documentation index is wrong, because that number lives in a SQL database.

Retrieval quality has layers. The first pass locates text and the second orders it, yet a decision comes before both about where to look. Routing a request among vector search, SQL queries and live web search, based on user intent, is the layer upstream of everything above.