Auditing RAG Retrieval for Stale Chunks, Drift and Noisy Context

The quiet kind of RAG failure

Picture a support bot that tells a customer the return window is 30 days when the real figure is 14. The policy document was edited in January. The embedding index was last touched in October. No one is hurt and no lawsuit follows, but the unsettling part is that nobody can say how long the wrong answer has been going out. A week? Three months? Without a way to check, there's no telling.

That's the failure worth attention here, and it isn't the dramatic one where a model hallucinates or rambles. It's the quiet one. Dashboards look green, and a small share of users have been misled for months without a single alert.

Most teams building RAG measure the wrong thing. They ask whether the final answer sounds good, and they skip the question of whether the material feeding that answer was any good. Mostly that's because nobody gave them a simple instrument for it. What follows is one: a Python script that audits the retrieval layer automatically, built from open-source libraries and a JSON file of test queries, with roughly 30 minutes of setup and no observability platform or statistics degree.

Look at retrieval, not at the model

When a RAG system misbehaves, the reflex is to blame the model: wrong prompt, wrong temperature, wrong version. Tweak the prompt or move to a newer model and things improve for a few weeks, then the problem returns, because the real culprit sits elsewhere. The chain in production systems usually runs like this:

A document changes, but the embedding index isn't refreshed. A stale chunk gets retrieved, the LLM answers confidently from outdated context, the answer is wrong, and no one can tell why.

The model isn't lying. It does exactly what it was told, which is to produce an answer from the chunks supplied. Feed it chunks that are six months old and the answer will be six months old too, delivered fluently and with flawless grammar.

The sneaky part lies in the usual evaluation method, which is end to end: a query goes in and the answer that comes out is judged. RAGAS, among the more widely adopted evaluation frameworks, was built around the full pipeline and scores context precision, faithfulness and answer relevance (Es et al., 2023, EACL 2024). It's a solid framework. Still, it rates the system overall, not the retrieval layer on its own, so a passing end-to-end score can mask slow retrieval decay when the model occasionally gets lucky and compensates.

A second problem sits beneath that one. Liu et al. reported in 2024 that language models do markedly worse when the relevant information is buried in the middle of a long context, even models trained specifically for long inputs. So retrieving five chunks of which three are marginally relevant actively harms the system, and the tokens spent on them hurt answer quality. Neither problem shows up unless the retrieval layer is inspected directly, which is the plan.

Three failure modes that standard metrics miss

Chunk poisoning

The name "chunk poisoning" doesn't come from a paper. It describes chunks that seem relevant because their wording overlaps with the query, so they rank high in vector search yet don't contain the answer. One common cause is a split that leaves the keywords in one chunk and the real content in the next. The keyword-rich chunk scores well on cosine similarity, the useful chunk falls out of the top results, and the model pieces together a plausible but incorrect reply.

Stale content

Suppose an FAQ updated its refund policy in January and the index was last refreshed in October. The old policy chunk still ranks for refund questions and still reaches the model as "relevant". Users end up contradicted by the support team while the metrics show nothing.

Embedding drift

Slow and cumulative, this is the most dangerous of the three. An embedding model records the corpus as it stood at indexing time. Every document edit, every new term entering the domain and every change in how users phrase queries erodes that snapshot's accuracy a bit. Mean cosine similarity might fall from 0.82 to 0.74 across six months. Nothing breaks, no alert goes off, and the decay shows only once answers get visibly worse, by which point the system is months behind.

Noise in the context has a cost as well. With five chunks in the window and three of them irrelevant, tokens are wasted and quality suffers, because content in the middle of a long context gets less attention than content at the edges, as Liu et al. documented.

What the audit does

Three checks line up with those three failure modes.

1. Relevance and hits. For each query in the test set, the script fetches the top-K chunks and calculates cosine similarity between the query's embedding and every chunk's embedding. It also verifies whether the chunks known to hold the answer came back. If they didn't, the query is flagged as a miss.
2. Drift against a baseline. Every run stores its distribution of similarity scores, and later runs compare the new distribution with that stored baseline using the Kolmogorov-Smirnov test. If the two differ significantly, a drift alert fires. No need to follow the math: a p-value under 0.05 just says something changed and merits a look.
3. Context efficiency. The script measures what fraction of the retrieved tokens appear in the model's answer. A high fraction means tight, relevant retrieval, while a low one means noise is padding the context. A figure that stays below 40% is a reason to investigate.

The audit produces a structured JSON report. A weekly cron job is enough, with attention on the flag rate and the drift analysis. Most RAG systems have never had this kind of monitoring layer.

The golden query set

Before any code, a golden query set is needed: a JSON file of representative queries, each paired with the chunk IDs that ought to appear in its results.

[
  {
    "query": "What is the refund policy for digital products?",
    "expected_chunk_ids": ["faq_doc_chunk_12", "faq_doc_chunk_13"],
    "notes": "Customer FAQ, policy updated 2024-Q1"
  },
  {
    "query": "How do I reset my API key?",
    "expected_chunk_ids": ["api_docs_chunk_07"],
    "notes": "API documentation, stable"
  }
]

Size matters. Twenty queries felt like enough at first and wasn't, because the drift test needs more data points than that, and 20 gives a distribution too sparse to trust. Treat 50 as the floor and 100 or more as the point where drift signals become meaningful.

Start by taking the most frequent user queries from application logs. Run retrieval for each by hand, inspect the results and record which chunk IDs belong in the top positions. The work is dull and takes a few hours, yet it's the single part that can't be automated, and everything else rests on it. Some habits help:

• Spread queries across different topic categories instead of clustering them around one document.
• Add some queries that should return no result at all, since negative examples catch false-positive retrievals.
• Refresh the golden set when the corpus changes substantially, not continuously. It's a stable reference and not a live dataset.

The project layout and the main script

The repository is organized like this:

rag-retrieval-audit/
├── audit/
│   ├── __init__.py           # Package exports
│   ├── rag_audit.py          # Main audit logic — all three checks
│   └── config.py             # All thresholds and model settings
├── examples/
│   ├── golden_queries.json   # Sample golden set (10 queries)
│   └── run_audit.py          # End-to-end demo — runs without an existing collection
├── tests/
│   └── test_audit.py         # Unit tests for all scoring functions
├── requirements.txt          # sentence-transformers, chromadb, scipy, numpy
└── README.md                 # Setup, usage, how to read the report

And here's the structure of rag_audit.py, the file that contains all three checks. It's an outline for orientation and not the full implementation.

# rag_audit.py — structure overview
# Full implementation: https://github.com/satyam671/rag-retrieval-audit

# ── CONFIGURATION (tune to your pipeline)
EMBEDDING_MODEL      = "all-MiniLM-L6-v2"  # must match your index
TOP_K                = 5
RELEVANCE_THRESHOLD  = 0.70
DRIFT_P_THRESHOLD    = 0.05
EFFICIENCY_THRESHOLD = 0.40

# ── CHECK 1: Are the right chunks coming back?
def score_relevance(query_embedding, chunk_embeddings, expected_ids, retrieved_ids):
    """Cosine similarity per chunk + expected chunk hit/miss against golden set."""
    ...
# ── CHECK 2: Has retrieval quality shifted over time?
def detect_drift(current_sims, baseline_path=None):
    """Two-sample KS test comparing current similarity distribution to baseline."""
    ...
# ── CHECK 3: How much of the context window is signal?
def score_efficiency(retrieved_docs, answer):
    """Token overlap between retrieved chunks and the LLM answer."""
    ...
# ── RUNNER
def run_audit(golden_set_path, collection_name, chroma_persist_dir,
              baseline_path=None, answers_path=None, output_path="rag_audit_report.json"):
    """Runs all three checks, writes a structured JSON report, saves the baseline."""
    ...

The script's README explains how to clone it, point it at a ChromaDB collection and run it, including how to swap in a different vector store.

Walking through each check

Chunk relevance scoring

The relevance check does two things people often miss when building it themselves.

First, it computes cosine similarity directly from the embeddings and doesn't use the distance values ChromaDB returns. ChromaDB's default metric is L2, not cosine similarity, and the two aren't interchangeable. Mixing them gives scores that look sensible while measuring something else.

Second, it confirms that the golden set's expected chunk IDs appeared in the top-K, and that's what exposes chunk poisoning. A mean similarity of 0.75 sounds fine until you learn that the chunk meant to answer the query scored 0.62 and landed sixth, one place past the cutoff. The model never got the right answer, while the similarity numbers still looked acceptable.

The practical difference between gating and not gating is clear in code.

# WITHOUT the relevance gate — standard approach most teams use
def build_context_naive(query, collection, top_k=5):
    results = collection.query(
        query_embeddings=[embed(query)],
        n_results=top_k,
        include=["documents"]
    )
    # Pass everything back, no quality check
    return "\n\n".join(results["documents"][0])

# WITH the relevance gate - what the audit tells you to build
def build_context_gated(query, collection, model, top_k=5, threshold=0.70):
    q_emb   = model.encode([query], normalize_embeddings=True)[0]
    results = collection.query(
        query_embeddings=[q_emb.tolist()],
        n_results=top_k,
        include=["documents", "embeddings", "ids"]
    )
    passed_chunks = []
    for i, chunk_emb in enumerate(results["embeddings"][0]):
        sim = cosine_sim(q_emb, np.array(chunk_emb))
        if sim >= threshold:
            passed_chunks.append(results["documents"][0][i])
    # Empty context is better than wrong context
    return "\n\n".join(passed_chunks)

In the naive version, the top-ranked chunks go through regardless of how good they are. In the gated version, any chunk that hasn't earned its place is held back. A reply of "I don't have enough information to answer that" from an empty context beats a confident, wrong answer built on noise.

Embedding drift detection

The drift check relies on the Kolmogorov-Smirnov test. It compares the shape of the current similarity distribution to a saved baseline and flags significant divergence. Why distributions and not mean scores? Because one number hides too much. The mean could hold steady at 0.78 while the distribution beneath it flattens and spreads out, meaning some queries get much worse while others compensate. The KS test catches distribution changes that averages miss.

The sample chart plots embedding drift in a production RAG system six months after corpus updates without a re-index: a KS statistic of 0.21 and a p-value of 0.003.

One caveat: the test needs enough data to be dependable. With 15 queries the p-values bounce around and breed false confidence, while 50 or more give a stable signal. That's another reason the golden set's size matters more than it appears.

Context window efficiency

This is the roughest of the three checks, with real limits. Word-level overlap between retrieved chunks and the model's answer is only a proxy. When a model paraphrases heavily, as GPT-4-class models do, overlap undercounts how much of the context was really used.

Still, it catches systematic problems. If efficiency scores sit under 40% consistently, something upstream is off: chunks are too large, top-K is too high, or retrieval is pulling in topic-adjacent text that doesn't address the query. The check isn't precise but it's directionally right, and that's enough to start an investigation.

Reading the report

The output is a JSON file, and these fields deserve the attention:

• flag_rate is the fraction of queries that tripped a failure condition. Above 20% means retrieval needs attention, and a jump from 8% one week to 22% the next means something changed.
• drift_analysis.drift_detected is the boolean to watch. If it's true, look at mean_delta for direction. A negative delta means similarity is falling, which is the signal to re-index.
• drift_analysis.ks_statistic rises with the size of the distribution shift. Below 0.10 is probably noise. Above 0.20 is a real problem whatever the p-value says.
• query_results[i].relevance.expected_chunk_hit is per-query ground truth. A false means the chunk that should have answered the query missed the top-K. If that happens for more than 15% of the golden set, re-index.

The example is launched with one command, shown here.

python -m examples.run_audit

A cheat sheet condenses the audit into one picture.

Run it weekly and keep every baseline. The trend between runs says more than any single run.

A lesson on sample size

Running drift detection on a 15-query golden set produced p-values that made no sense. The test was technically running, but unreliably, and it was easy to lose two days suspecting the statistical implementation before realizing the data was simply too thin.

With fewer than 50 queries, the drift figures can't be trusted, so build the set first and enable drift checking afterward. A corpus covering several separate topic domains calls for one golden set per domain, audited separately, because one distribution over a mixed corpus can hide a decline in a single domain that would stand out if viewed alone.

Limits of this approach

The script verifies that the right chunks are retrieved. It doesn't judge whether chunk boundaries are placed well, whether chunk size suits the query types, or whether semantic chunking should replace fixed-size splitting.

Those questions matter. NVIDIA's benchmarks across five datasets, as summarized in a FloTorch analysis, found that the best chunk size varies by query type: factoid queries peak at 256 to 512 tokens, while analytical ones need 1024 or more. Any single chunk size is a compromise, and a golden set that mixes both query types ends up averaging over a basic mismatch. Testing chunking strategy directly, and comparing semantic with fixed-size splitting using real numbers, is a separate exercise.

References

• The RAGAS paper on automated evaluation of retrieval-augmented generation (arXiv 2309.15217, 2023; EACL 2024).
• The "lost in the middle" study of long-context behavior in language models, published in TACL volume 12 in 2024.
• NVIDIA's applied research on chunking strategies across document types, from 2024.