Five Reranking Techniques for RAG: From Fast Retrieval to Accurate Context

Picture a RAG pipeline that's been built properly: documents chunked, an embedding model chosen, a vector database wired in. A question arrives, the retriever returns 50 chunks, and the top 5 go into the prompt. The answer is okay. Not great. Occasionally it misses completely.

Look at what was retrieved and you often find the chunk that holds the answer sitting at position 23. The embedding model rated it moderately relevant but not top-5 relevant, so the LLM never saw it. This is hardly a rare corner case. It's one of the most common failure modes in production RAG.

The retriever usually isn't the culprit. What happens after retrieval is.

The bottleneck after retrieval

First-stage retrieval is tuned for speed. Dense vectors, sparse BM25 or a mix of both, the goal is to cut a million documents down to a workable set within milliseconds. The price is a quick, shallow judgment of relevance: the model looks at the query and each document on its own, scores them, and hands back the closest.

"Closest" and "most useful for answering this question" are different things. Initial retrieval stumbles for four reasons:

• Shallow scoring. Embeddings measure similarity, not usefulness, so a chunk can sit near the query and still contain nothing helpful.
• No query-document interaction. Bi-encoders encode the query and the documents separately and never see them together, which hides subtle relationships.
• Context blindness. The retriever has no idea what the LLM needs so it can't predict whether a given chunk will lead to a good reply.
• False positives. Generic chunks about a topic often outscore specific factual ones, because they're the "safer" match.

Reranking is the fix.

What reranking does

Reranking is a second pass. After quick, broad retrieval returns a pool of candidates, a stronger model reorders them by how well they truly fit the particular query. Think of it as quality control before context reaches the LLM.

The flow has three steps:

1. Initial retrieval: a fast, wide search returns 50 to 100 candidate chunks.
2. Reranking: a smarter model scores every candidate against the query and reorders the list.
3. Final selection: the top 3 to 5 reranked chunks go to the LLM.

You get the speed of approximate search together with the accuracy of deep scoring. The retriever isn't replaced, only supplemented.

Retrieval and reranking side by side

| Aspect              | Initial Retrieval        | Reranking                     |
| ------------------- | ------------------------ | ----------------------------- |
| Goal                | Find candidates fast     | Judge true relevance          |
| Speed               | Milliseconds             | Tens to hundreds of milliseconds |
| Input               | Query + index            | Query + top-k candidates      |
| Scoring depth       | Shallow (embedding dot product) | Deep (cross-attention, token interaction) |
| Cost                | Low (local compute)      | Higher (model inference)      |
| When to use         | Every query              | On top-k candidates only      |

The idea is to avoid spending heavy compute on a million documents and to spend it on the 50 most promising ones.

Five reranking techniques

Rerankers range from cheap and quick all the way to heavy and strong. The right one depends on your latency budget, your accuracy needs and your operational limits. These five cover that range.

1. Cross-encoder reranking

Cross-encoders are the accuracy benchmark. A bi-encoder embeds query and document separately, whereas a cross-encoder reads them together as one input. That lets it capture token-level interactions such as pronoun resolution, negation and implied relationships, which separate retrievers miss entirely.

The steps:

1. The retriever returns 50 candidate chunks.
2. For each chunk, join the query and the chunk as [CLS] query [SEP] chunk [SEP].
3. Feed the pair to a cross-encoder model, for instance cross-encoder/ms-marco-MiniLM-L-6-v2.
4. Out comes a relevance score per pair.
5. Sort by score and keep the top 5.

from sentence_transformers import CrossEncoder

# Load a cross-encoder reranker
# ms-marco-MiniLM-L-6-v2 is fast and accurate for general use
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank_with_cross_encoder(query: str, retrieved_docs: list[str], top_k: int = 5):
    """
    Rerank retrieved documents using a cross-encoder.

    Args:
        query: The user question
        retrieved_docs: List of document chunks from initial retrieval
        top_k: Number of documents to return after reranking

    Returns:
        List of (document, score) tuples, sorted by relevance
    """
    # Create query-document pairs
    pairs = [[query, doc] for doc in retrieved_docs]

    # Get relevance scores
    scores = cross_encoder.predict(pairs)

    # Combine docs with scores and sort
    scored_docs = list(zip(retrieved_docs, scores))
    scored_docs.sort(key=lambda x: x[1], reverse=True)

    return scored_docs[:top_k]

# Example usage
query = "What are the side effects of amoxicillin?"
retrieved = [
    "Amoxicillin is a penicillin antibiotic used to treat bacterial infections.",
    "Common side effects include nausea, vomiting, and diarrhea.",
    "The drug was first discovered in 1958 by researchers at Beecham.",
    "Patients with penicillin allergies should avoid amoxicillin.",
    "Side effects may include rash, itching, and in rare cases, anaphylaxis.",
]

top_docs = rerank_with_cross_encoder(query, retrieved, top_k=3)
for doc, score in top_docs:
    print(f"Score: {score:.4f} | {doc}")

Pick cross-encoders when accuracy comes first and an extra 50 to 200 ms is acceptable. For most production RAG this is the best starting point. The limitation is that each query-document pair is processed separately, so latency grows in line with the number of candidates.

2. Reciprocal Rank Fusion (RRF)

Suppose you'd rather not run a neural model at all. Reciprocal Rank Fusion merges rankings from several retrievers with a deterministic formula: no training, no inference, only arithmetic.

RRF fits best when hybrid retrieval is already in place, such as dense vector search alongside BM25 keyword search. Each retriever supplies its own ranking, and RRF folds them into one better list.

The formula:

RRF_score(d) = sum over retrievers of 1 / (k + rank_r(d))

Here k is a constant, typically 60, and rank_r(d) is document d's position in retriever r. Documents ranked well by multiple retrievers rise to the top.

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """
    Merge multiple document rankings using Reciprocal Rank Fusion.

    Args:
        rankings: List of rankings, where each ranking is a list of document IDs
                  ordered from most to least relevant
        k: RRF constant (default 60, as recommended in the original paper)

    Returns:
        List of (document_id, rrf_score) tuples, sorted by fused score
    """
    scores = {}

    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            if doc_id not in scores:
                scores[doc_id] = 0.0
            # RRF formula: 1 / (k + rank)
            scores[doc_id] += 1.0 / (k + rank)

    # Sort by score descending
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

# Example: merging BM25 and vector search results
bm25_results = ["doc_5", "doc_2", "doc_8", "doc_1", "doc_9"]
vector_results = ["doc_1", "doc_5", "doc_3", "doc_8", "doc_7"]

fused = reciprocal_rank_fusion([bm25_results, vector_results])

print("Fused ranking:")
for doc_id, score in fused:
    print(f"  {doc_id}: {score:.4f}")

# Notice: doc_5 and doc_1 appear in both retrievers and get boosted to the top

RRF is the quickest option here, with essentially zero added latency. Use it when you already run hybrid retrieval and want a cheap improvement without model inference. It won't beat a cross-encoder on raw accuracy, but it works surprisingly well for something with no machine learning in it.

3. Cohere Rerank API

Some teams don't want to host and tune a reranking model. If your priority is the product and not model operations, a managed reranking API is attractive, and Cohere's is the most established in this space.

The API accepts a query plus a list of documents and returns them in relevance order. Behind it runs a modern reranking model with optimisations that would take weeks to build yourself. It handles multilingual content and long documents and scales on demand.

import cohere
from dotenv import load_dotenv
import os

load_dotenv()

# Initialize Cohere client
co = cohere.Client(os.getenv("COHERE_API_KEY"))

def rerank_with_cohere(query: str, documents: list[str], top_k: int = 5):
    """
    Rerank documents using Cohere's managed Rerank API.

    Args:
        query: The user question
        documents: List of document chunks from initial retrieval
        top_k: Number of documents to return

    Returns:
        List of (document, relevance_score) tuples
    """
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=documents,
        top_n=top_k,
        return_documents=True
    )

    results = []
    for result in response.results:
        results.append((
            result.document.text,
            result.relevance_score
        ))

    return results

# Example usage
query = "How do I handle authentication in a FastAPI app?"
docs = [
    "FastAPI is a modern web framework for building APIs with Python.",
    "To add authentication, use OAuth2PasswordBearer and JWT tokens.",
    "Pydantic models in FastAPI provide automatic request validation.",
    "The OAuth2PasswordBearer class expects a token URL endpoint.",
    "FastAPI was created by Sebastián Ramírez and released in 2018.",
]

ranked = rerank_with_cohere(query, docs, top_k=3)
for doc, score in ranked:
    print(f"Score: {score:.4f} | {doc}")

Choose it when you want strong accuracy without operational burden. The trade-offs are cost per call and an external dependency, and for very high traffic the latency of a network call can matter. For most teams the time saved on hosting and optimisation outweighs those costs.

4. ColBERT

Cross-encoders are accurate but slow, and bi-encoders are quick but shallow. ColBERT finds a middle path through "late interaction".

1. Offline: each document is encoded token by token into contextual embeddings, which are precomputed and stored in an index.
2. Online: when a query arrives it gets the same token-level treatment.
3. Scoring: every query token is matched to its closest document token (the MaxSim step), and those maxima are summed those maximum similarities across all query tokens to get the relevance score.

Because document embeddings are precomputed, this is much faster than a cross-encoder. Because it keeps token-level interactions, it beats bi-encoders on accuracy.

from colbert import Searcher
from colbert.infra import Run, RunConfig

def setup_colbert_searcher(index_path: str, checkpoint: str):
    """
    Initialize a ColBERT searcher for late-interaction reranking.

    Args:
        index_path: Path to the pre-built ColBERT index
        checkpoint: Path to the ColBERT model checkpoint

    Returns:
        Configured Searcher instance
    """
    with Run().context(RunConfig(nranks=1, experiment="reranking")):
        searcher = Searcher(
            index=index_path,
            checkpoint=checkpoint
        )
    return searcher

def rerank_with_colbert(searcher, query: str, doc_ids: list[str], top_k: int = 5):
    """
    Rerank documents using ColBERT's late interaction.

    Args:
        searcher: Initialized ColBERT Searcher
        query: The user question
        doc_ids: List of document IDs from initial retrieval
        top_k: Number of documents to return

    Returns:
        List of (doc_id, score) tuples
    """
    # Search within the candidate set
    results = searcher.search(
        query,
        k=top_k,
        filter_fn=lambda pid: pid in doc_ids  # Only rerank candidates
    )

    return list(zip(results[0], results[2]))  # doc_ids, scores

# Note: ColBERT requires a pre-built index and model checkpoint.
# For production use, build the index once and load it at startup.

ColBERT suits large-scale systems that need better accuracy than bi-encoders and lower latency than cross-encoders. The cost is the work of building and maintaining the index. If you're comfortable with vector databases and want better quality without latency that grows linearly with candidates, it's the sweet spot.

5. LLM-as-a-Judge

In the highest-stakes areas, like medical diagnosis, legal research and financial compliance, even a cross-encoder may fall short. The queries are complex, the context is subtle and a wrong answer is expensive. There the most accurate reranker can be the LLM itself.

The idea is to prompt a large model (GPT-4, Claude or Gemini) to rate each candidate's relevance to the query. With broad world knowledge and strong reasoning it can judge relevance in ways smaller rerankers can't. A typical prompt:

You are evaluating documents for a retrieval system.

Query: {query}
Document: {document}

Rate how relevant this document is for answering the query.
Respond with a single integer from 1 to 10, where 10 means perfectly relevant.

Relevance score:

Run it for each candidate, parse the scores and return the top-k.

from openai import OpenAI
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def score_document_with_llm(query: str, document: str) -> int:
    """
    Ask an LLM to score a document's relevance to a query.

    Args:
        query: The user question
        document: A candidate document chunk

    Returns:
        Integer relevance score from 1-10
    """
    prompt = f"""You are evaluating documents for a retrieval system.

Query: {query}
Document: {document}

Rate how relevant this document is for answering the query.
Respond with a single integer from 1 to 10, where 10 means perfectly relevant.
Be strict: only give high scores to documents that directly help answer the query.

Relevance score:"""

    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5
    )

    try:
        score = int(response.choices[0].message.content.strip())
        return max(1, min(10, score))  # Clamp to 1-10
    except ValueError:
        return 5  # Default on parse failure

def rerank_with_llm_judge(query: str, documents: list[str], top_k: int = 3):
    """
    Rerank documents using an LLM as a relevance judge.

    Args:
        query: The user question
        documents: List of candidate document chunks
        top_k: Number of documents to return

    Returns:
        List of (document, score) tuples, sorted by relevance
    """
    scored = []
    for doc in documents:
        score = score_document_with_llm(query, doc)
        scored.append((doc, score))

    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

# Example usage
query = "What are the tax implications of RSU vesting for employees in California?"
docs = [
    "RSUs are restricted stock units granted to employees as part of compensation.",
    "In California, RSU income is taxed as ordinary income at vesting, not at grant.",
    "Employers typically withhold federal and state taxes at vesting time.",
    "Stock options and RSUs have different tax treatments under IRS rules.",
    "California has one of the highest state income tax rates in the US.",
]

ranked = rerank_with_llm_judge(query, docs, top_k=3)
for doc, score in ranked:
    print(f"Score: {score}/10 | {doc}")

This is the most accurate technique here and also the slowest and costliest. Use it for complex, high-value queries where a wrong answer costs real money or carries real risk. For most applications, cross-encoders or managed APIs deliver around 90% of the accuracy at 10% of the cost (a rule-of-thumb figure, not a benchmark). Keep LLM judges for the cases that truly justify them.

Choosing between them

No reranker wins everywhere. This table summarises the constraints:

| Technique             | Best For                                          | Latency      | Cost           |
| --------------------- | ------------------------------------------------- | ------------ | -------------- |
| Cross-Encoder         | Maximum quality on top-k candidates               | 50-200ms     | Local GPU/CPU  |
| RRF                   | Hybrid retrieval without adding model inference   | ~0ms         | Free           |
| Cohere Rerank API     | Speed without operational overhead                | 100-300ms    | Per API call   |
| ColBERT               | Large-scale, low-latency use cases                | 20-100ms     | Index + GPU    |
| LLM-as-a-Judge        | Complex, high-value queries (medical, legal)      | 1-5 seconds  | Per API call   |

A sensible plan is to start with a cross-encoder, since it delivers the largest accuracy gain for the least complexity. If latency becomes a problem, look at ColBERT or a managed API like Cohere. Add RRF if you run hybrid retrieval, and save the LLM judge for queries where maximum accuracy justifies the cost.

Teams rarely have to pick just one. A common production layering looks like this:

1. Hybrid retrieval (dense plus BM25) with RRF fusion.
2. Cross-encoder reranking over the top 50 candidates.
3. An LLM-as-a-Judge fallback for flagged high-stakes queries.

That gives speed in the ordinary case and accuracy when it counts.

Closing thoughts

Reranking is probably the most underrated part of production RAG. Teams spend months tuning chunk size and embedding models while their best context stays buried at position 23, and a good reranker repairs that with little engineering effort.

The techniques range from free and fast (RRF) to costly and powerful (LLM-as-a-Judge), and reranking doesn't have to be all or nothing. Start simple, layer methods, and add complexity only when your metrics call for it.

• First-stage retrieval gathers candidates quickly; it doesn't settle on answers.
• Reranking acts as quality control, a second pass that judges true relevance.
• Cross-encoders make the best starting point, with high accuracy and moderate complexity.
• RRF gives free wins when you already run hybrid retrieval.
• Managed APIs save engineering time, and Cohere Rerank works well out of the box.
• ColBERT balances speed and accuracy for large systems.
• LLM-as-a-Judge is the heavy option for when accuracy matters more than cost.

If you run RAG in production without reranking, answer quality is being left on the table. Pick one technique, implement it this week and measure the difference.