Practical notes: 5 Reranking Techniques in RAG: From Fast Retrieval to Accurate

This walkthrough rebuilds the path from raw materials to a working system for: 5 Reranking Techniques in RAG: From Fast Retrieval to Accurate Context. The focus is operable steps, explicit checks, and code that you can drop into a repo without guessing intent. For the Overview stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

The Retrieval Bottleneck Nobody Talks About

When working through the The Retrieval Bottleneck Nobody stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

What Is Reranking?

When working through the What Is Reranking stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Retrieval vs. Reranking

When working through the Retrieval vs Reranking stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through the Retrieval vs Reranking stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

| Aspect              | Initial Retrieval        | Reranking                     |
| ------------------- | ------------------------ | ----------------------------- |
| Goal                | Find candidates fast     | Judge true relevance          |
| Speed               | Milliseconds             | Tens to hundreds of milliseconds |
| Input               | Query + index            | Query + top-k candidates      |
| Scoring depth       | Shallow (embedding dot product) | Deep (cross-attention, token interaction) |
| Cost                | Low (local compute)      | Higher (model inference)      |
| When to use         | Every query              | On top-k candidates only      |

The Five Reranking Techniques

The The Five Reranking Techniques stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

1. Cross-Encoder Reranking

The 1 Cross-Encoder Reranking stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

from sentence_transformers import CrossEncoder

# Load a cross-encoder reranker
# ms-marco-MiniLM-L-6-v2 is fast and accurate for general use
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank_with_cross_encoder(query: str, retrieved_docs: list[str], top_k: int = 5):
    """
    Rerank retrieved documents using a cross-encoder.

    Args:
        query: The user question
        retrieved_docs: List of document chunks from initial retrieval
        top_k: Number of documents to return after reranking

    Returns:
        List of (document, score) tuples, sorted by relevance
    """
    # Create query-document pairs
    pairs = [[query, doc] for doc in retrieved_docs]

    # Get relevance scores
    scores = cross_encoder.predict(pairs)

    # Combine docs with scores and sort
    scored_docs = list(zip(retrieved_docs, scores))
    scored_docs.sort(key=lambda x: x[1], reverse=True)

    return scored_docs[:top_k]

# Example usage
query = "What are the side effects of amoxicillin?"
retrieved = [
    "Amoxicillin is a penicillin antibiotic used to treat bacterial infections.",
    "Common side effects include nausea, vomiting, and diarrhea.",
    "The drug was first discovered in 1958 by researchers at Beecham.",
    "Patients with penicillin allergies should avoid amoxicillin.",
    "Side effects may include rash, itching, and in rare cases, anaphylaxis.",
]

top_docs = rerank_with_cross_encoder(query, retrieved, top_k=3)
for doc, score in top_docs:
    print(f"Score: {score:.4f} | {doc}")

2. Reciprocal Rank Fusion (RRF)

The 2 Reciprocal Rank Fusion stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move. The 2 Reciprocal Rank Fusion stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """
    Merge multiple document rankings using Reciprocal Rank Fusion.

    Args:
        rankings: List of rankings, where each ranking is a list of document IDs
                  ordered from most to least relevant
        k: RRF constant (default 60, as recommended in the original paper)

    Returns:
        List of (document_id, rrf_score) tuples, sorted by fused score
    """
    scores = {}

    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            if doc_id not in scores:
                scores[doc_id] = 0.0
            # RRF formula: 1 / (k + rank)
            scores[doc_id] += 1.0 / (k + rank)

    # Sort by score descending
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

# Example: merging BM25 and vector search results
bm25_results = ["doc_5", "doc_2", "doc_8", "doc_1", "doc_9"]
vector_results = ["doc_1", "doc_5", "doc_3", "doc_8", "doc_7"]

fused = reciprocal_rank_fusion([bm25_results, vector_results])

print("Fused ranking:")
for doc_id, score in fused:
    print(f"  {doc_id}: {score:.4f}")

# Notice: doc_5 and doc_1 appear in both retrievers and get boosted to the top

3. Cohere Rerank API

For the 3 Cohere Rerank API stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

import cohere
from dotenv import load_dotenv
import os

load_dotenv()

# Initialize Cohere client
co = cohere.Client(os.getenv("COHERE_API_KEY"))

def rerank_with_cohere(query: str, documents: list[str], top_k: int = 5):
    """
    Rerank documents using Cohere's managed Rerank API.

    Args:
        query: The user question
        documents: List of document chunks from initial retrieval
        top_k: Number of documents to return

    Returns:
        List of (document, relevance_score) tuples
    """
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=documents,
        top_n=top_k,
        return_documents=True
    )

    results = []
    for result in response.results:
        results.append((
            result.document.text,
            result.relevance_score
        ))

    return results

# Example usage
query = "How do I handle authentication in a FastAPI app?"
docs = [
    "FastAPI is a modern web framework for building APIs with Python.",
    "To add authentication, use OAuth2PasswordBearer and JWT tokens.",
    "Pydantic models in FastAPI provide automatic request validation.",
    "The OAuth2PasswordBearer class expects a token URL endpoint.",
    "FastAPI was created by Sebastián Ramírez and released in 2018.",
]

ranked = rerank_with_cohere(query, docs, top_k=3)
for doc, score in ranked:
    print(f"Score: {score:.4f} | {doc}")

4. ColBERT

For the 4 ColBERT stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.

from colbert import Searcher
from colbert.infra import Run, RunConfig

def setup_colbert_searcher(index_path: str, checkpoint: str):
    """
    Initialize a ColBERT searcher for late-interaction reranking.

    Args:
        index_path: Path to the pre-built ColBERT index
        checkpoint: Path to the ColBERT model checkpoint

    Returns:
        Configured Searcher instance
    """
    with Run().context(RunConfig(nranks=1, experiment="reranking")):
        searcher = Searcher(
            index=index_path,
            checkpoint=checkpoint
        )
    return searcher

def rerank_with_colbert(searcher, query: str, doc_ids: list[str], top_k: int = 5):
    """
    Rerank documents using ColBERT's late interaction.

    Args:
        searcher: Initialized ColBERT Searcher
        query: The user question
        doc_ids: List of document IDs from initial retrieval
        top_k: Number of documents to return

    Returns:
        List of (doc_id, score) tuples
    """
    # Search within the candidate set
    results = searcher.search(
        query,
        k=top_k,
        filter_fn=lambda pid: pid in doc_ids  # Only rerank candidates
    )

    return list(zip(results[0], results[2]))  # doc_ids, scores

# Note: ColBERT requires a pre-built index and model checkpoint.
# For production use, build the index once and load it at startup.

5. LLM-as-a-Judge

For the 5 LLM-as-a-Judge stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call. For the 5 LLM-as-a-Judge stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

You are evaluating documents for a retrieval system.

Query: {query}
Document: {document}

Rate how relevant this document is for answering the query.
Respond with a single integer from 1 to 10, where 10 means perfectly relevant.

Relevance score:

from openai import OpenAI
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def score_document_with_llm(query: str, document: str) -> int:
    """
    Ask an LLM to score a document's relevance to a query.

    Args:
        query: The user question
        document: A candidate document chunk

    Returns:
        Integer relevance score from 1-10
    """
    prompt = f"""You are evaluating documents for a retrieval system.

Query: {query}
Document: {document}

Rate how relevant this document is for answering the query.
Respond with a single integer from 1 to 10, where 10 means perfectly relevant.
Be strict: only give high scores to documents that directly help answer the query.

Relevance score:"""

    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5
    )

    try:
        score = int(response.choices[0].message.content.strip())
        return max(1, min(10, score))  # Clamp to 1-10
    except ValueError:
        return 5  # Default on parse failure

def rerank_with_llm_judge(query: str, documents: list[str], top_k: int = 3):
    """
    Rerank documents using an LLM as a relevance judge.

    Args:
        query: The user question
        documents: List of candidate document chunks
        top_k: Number of documents to return

    Returns:
        List of (document, score) tuples, sorted by relevance
    """
    scored = []
    for doc in documents:
        score = score_document_with_llm(query, doc)
        scored.append((doc, score))

    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

# Example usage
query = "What are the tax implications of RSU vesting for employees in California?"
docs = [
    "RSUs are restricted stock units granted to employees as part of compensation.",
    "In California, RSU income is taxed as ordinary income at vesting, not at grant.",
    "Employers typically withhold federal and state taxes at vesting time.",
    "Stock options and RSUs have different tax treatments under IRS rules.",
    "California has one of the highest state income tax rates in the US.",
]

ranked = rerank_with_llm_judge(query, docs, top_k=3)
for doc, score in ranked:
    print(f"Score: {score}/10 | {doc}")

Which One Should You Use?

When working through the Which One Should You stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

| Technique             | Best For                                          | Latency      | Cost           |
| --------------------- | ------------------------------------------------- | ------------ | -------------- |
| Cross-Encoder         | Maximum quality on top-k candidates               | 50-200ms     | Local GPU/CPU  |
| RRF                   | Hybrid retrieval without adding model inference   | ~0ms         | Free           |
| Cohere Rerank API     | Speed without operational overhead                | 100-300ms    | Per API call   |
| ColBERT               | Large-scale, low-latency use cases                | 20-100ms     | Index + GPU    |
| LLM-as-a-Judge        | Complex, high-value queries (medical, legal)      | 1-5 seconds  | Per API call   |

Final Thoughts

When working through the Final Thoughts stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Operational checklist

The Operational checklist stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope.

Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

Score single-turn answers and multi-turn trajectories separately. Aggregate chat scores bury tool-loop failures.

Write a short runbook: how to rotate keys, how to drain the queue, how to roll back the last ingest.

Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for 16f80a919c4e: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.