Cross-Encoder as a Judge: Gate Your RAG Pipeline With Relevance Scores

Most RAG tutorials treat a cross-encoder as a re-ranking step and nothing more. It can do a more interesting job than that. Used as a judge, it decides whether the retrieved text is good enough to hand to the generator at all.

A common alternative is an LLM acting as judge, but that's still a generative model with all the usual quirks. A cross-encoder judge sits at a useful meeting point of three concerns: how the architecture evolves, what the pipeline costs, and how long users wait.

Why a cross-encoder makes a good judge

Basic RAG setups lean entirely on a bi-encoder, the sentence-transformer family used to produce embeddings. A bi-encoder squeezes each text into its own standalone vector. It measures how close two vectors are in space and misses finer logic such as conditions, negation and structural flips.

There's a deeper limit too. Bi-encoders are tuned for recall, not precision. They find documents on the same topic, but they can't confirm that a document answers the question that was actually asked.

The common way to evaluate retrieval today is to ask an LLM to read the retrieved chunks and give them a score, using a model like GPT or Claude. In production that approach has three serious drawbacks:

1. Latency. Waiting on a heavyweight API call before generation even starts ruins the experience for a user expecting a fast answer.
2. Cost. Sending thousands of tokens of retrieved background to a judge model on every single query makes the API bill balloon.
3. Instability. Prompts drift, and model output isn't deterministic, which breaks any hard-coded threshold. Hallucination is possible as well, so a passage rated relevant on Monday can be rated irrelevant or half-relevant on Tuesday.

A cross-encoder removes all three problems. It reads the query and the document together in one transformer pass, so every token can attend to every other token. The output is a precise, repeatable score between 0.0 and 1.0.

What happens inside: cross-attention

The easiest way to see why it judges better is to compare how it reads the text with how a bi-encoder does.

A bi-encoder encodes the query (Q) and the document (D) separately. One vector comes out for Q and another for D. Inside the transformer, query tokens only attend to query tokens, and document tokens only attend to document tokens. The two sides meet at the very end, through a dot product or a cosine similarity.

A cross-encoder works differently. It builds a single input sequence that holds both texts:

[CLS] User Query [SEP] Retrieved Document Chunk [SEP]

Since both sit in the same sequence, the model applies full cross-attention. The standard attention formula still applies:

Attention(Q, K, V) = softmax( (Q * Kᵀ) / √dₖ ) * V

The difference is that each token in the query computes an attention weight against each token in the document chunk, all at once. Words from the question get tied straight to the words that answer or contradict them.

For contrast, this is the bi-encoder layout, where the two sides never see each other until the final comparison:

Bi-Encoder Architecture:
  Query Tokens  ──► [ Encoder A ] ──► [ Vector Q ] ──┐
                                                      ├──► Cosine Distance ──► Score
  Doc Tokens    ──► [ Encoder B ] ──► [ Vector D ] ──┘
                   (No token-to-token interaction)

Cross-Encoder Architecture:
  Query Tokens ──┐
                 ├──► [  Full Self-Attention Matrix  ] ──► [ [CLS] Token ] ──► Sigmoid ──► Relevance
  Doc Tokens   ──┘    (Every token attends to every)       (Dense Output)                   Probability

A worked example

Here's a case where a bi-encoder breaks down, and where a cross-encoder gets it right.

Say a knowledge base holds three documents:

• Document A: "Severe thunderstorms and torrential rain disrupted travel across the region, and the airport scrapped every flight scheduled for the evening."
• Document B: "Even with the downpour and dark skies above, airport operations carried on normally and not one flight was delayed."
• Document C: "A fault in the air traffic control software led to major delays, and the afternoon rain had nothing to do with it."

A user then asks: "Did the rain cause flight delays?"

Where the bi-encoder fails

A standard embedding model treats the query as a whole and scores each document by itself. All three texts contain the strongly weighted terms "rain," "flight" and "delays." A bi-encoder only measures the broad closeness of the vectors, so Documents B and C get very high cosine similarity. They often land right next to Document A, and sometimes above it. The retriever sees matching words and assumes the topic matches, so it treats them as relevant.

What the cross-encoder concludes

Now pass the same candidates to the cross-encoder judge. The cross-attention matrix makes the query tokens line up against the document text and form actual relationships. The outcome is quite different:

[Query] + [Doc A] ---> Score: 0.94 (High Confidence)
Mechanics: The model explicitly links "rain" -> "caused" -> "cancel/delays".

[Query] + [Doc B] ---> Score: 0.21 (Irrelevant Noise)
Mechanics: The cross-attention catches the modifier "Despite the heavy rain... continued smoothly without a single flight delay". It recognizes a structural negation.

[Query] + [Doc C] ---> Score: 0.42 (Topical Context Only)
Mechanics: The model notes that while both events happened, the text explicitly declares them "unrelated".

If you treat those scores as true probabilities, you can cut Documents B and C from the prompt right away. The generator then receives a single, high-signal chunk, Document A. That removes any chance of the model inventing a contradictory or false claim about what the weather did.

How the score is produced

A bi-encoder compares the spatial distance between two separate vectors. A cross-encoder never creates independent embeddings. It reaches its score by analyzing the whole sequence jointly:

1. Token processing. The query and the retrieved document are packed into one sequence and split by a special token: [CLS] User Query [SEP] Document Chunk [SEP].
2. The attention matrix. As the sequence moves through the transformer, each token across both texts works out its relationship to every other token. The result is a dense interaction matrix inside the model.
3. The hidden state. The model gathers its final understanding in the [CLS] token at the start of the sequence, which acts as a single summary of how the query and document relate.
4. The probability mapping. That final [CLS] hidden state goes through a linear classification layer, which collapses the raw hidden dimensions into one number. A sigmoid function, Score = 1 / (1 + e^-x), then converts it.

The sigmoid squeezes any raw output into the range from 0.0 to 1.0. The result reads as a calibrated probability of relevance, a measure of how confident the model is that the document really holds the answer to the question.

Interpreting the score

Because the value is bounded and absolute, you can sort it into tiers and act on each one at runtime:

• 0.80 to 1.00, a high-confidence signal. The document settles the core of the question. The answer is stated in the chunk, and the structure of the claims matches the query.
• 0.50 to 0.80, topical context only. The document is in the right area and mentions the right concepts, but it has no definitive answer or carries a lot of background noise.
• 0.00 to 0.50, a structural mismatch or noise. The text shares many keywords with the query yet has opposing logic, negations or an unrelated subject.

Turning the judge into a circuit breaker

In ordinary RAG, a cross-encoder is a sorter. It reorders text blocks before they move downstream. The real novelty of the judge role is turning its probability into a hard gate that can trip like a circuit breaker.

Rather than letting the generator read whatever happened to rank first, you set a firm quality floor for the pipeline, for example 0.75.

Say the retriever returns twenty documents and the judge scores every one of them. The best score in the whole batch is 0.42. The breaker trips, and the pipeline cancels the generation call on the spot.

A huge, costly language model should not be left to guess from irrelevant material, because that's exactly what triggers hallucinations. The system instead sends the user to a prepared fallback reply, something like: "Sorry, no verified internal documentation is available to answer that question reliably."

Calibrated probabilities used as a hard programmatic boundary turn an unpredictable text-generation loop into a predictable piece of software. The generator only runs when the evidence is good, and the cost of the judge stays low because it's a small encoder model, not another round of expensive generation.