Debugging Quiet RAG Failures: Retrieval, Chunking, Filters and Evals

The failures nobody sees

The most annoying RAG bugs aren't the noisy ones. Noisy failures are easy: the bot admits it found nothing, the vector database is down, an embedding job crashed, or an answer arrives without citations.

Quiet failures cost more. Documents are retrieved, the answer reads smoothly, the citation looks legitimate and the dashboard reports a successful retrieval. Then a domain expert reads the reply and says it's wrong. That's usually when a team realizes it has been debugging the wrong layer.

RAG isn't one feature. It's a sequence of choices that stack up. Somebody decides how a question is read and rewritten, where documents are cut apart, which retriever gets called, what metadata narrows the candidates, how the shortlist is reranked, what finally lands in the prompt, how the model behaves when evidence is thin and whether the written answer truly follows from what was supplied. Looking only at the final prompt means arriving after the damage is done.

The pipeline being debugged

A realistic system looks more like the following than like the tidy diagram on a slide:

flowchart LR
    A[User question] --> B[Query rewrite]
    B --> C[Retriever]
    C --> D[Reranker]
    D --> E[Evidence pack]
    E --> F[Answer generator]
    F --> G[Verifier]
    G --> H[Final answer]
    C --> I[Trace log]
    D --> I
    E --> I
    F --> I
    G --> I

Any box can fail. The generator takes the blame because it's the most visible stage, but many failures begin upstream. The retriever may have pulled the wrong time period. The reranker may have promoted a lookalike chunk that doesn't matter. The evidence pack may have dropped the single useful paragraph because the token budget was too tight. Good debugging means making every stage visible.

Failure 1: similar text isn't useful evidence

Vector search finds semantic similarity, which isn't always task relevance. Take a user asking whether enterprise customers can get a refund after the Q2 policy update. The retriever may return a refund-policy chunk simply because it mentions enterprise, refund and policy. That chunk might be stale, incomplete or unrelated to Q2.

A useful trace shows the retrieved document ID, its version, its effective date, the similarity score, the rerank score, the metadata filters applied and whether the final answer cited it. If the logs say nothing beyond a score like 0.83, the mistake can't be diagnosed.

Failure 2: chunking destroyed the meaning

Chunking isn't a storage detail. It decides answer quality. A refund policy might be organized as eligibility, exceptions, an enterprise addendum, effective dates and regional differences. If the chunker separates the effective date from the exception, the answer can quote the eligibility paragraph and miss the condition that flips the outcome.

Better chunks carry a stable document ID, a section heading, a version, a date, enough neighboring text to keep the meaning and a pointer back to the original. None of this argues for enormous chunks, which just add noise. The aim is chunks that preserve the point where a decision turns.

Failure 3: missing metadata filters

Many RAG systems begin as pure semantic search. That holds up until the same topic appears across versions, products, tenants, regions or permission groups. A question about the current enterprise refund policy shouldn't make the retriever search every refund document the company ever wrote. It should filter first.

from dataclasses import dataclass
from datetime import date

@dataclass(frozen=True)
class SearchFilters:
    product: str | None
    customer_tier: str | None
    region: str | None
    as_of: date
    permission_group: str

def build_filters(user_context: dict) -> SearchFilters:
    return SearchFilters(
        product=user_context.get("product"),
        customer_tier=user_context.get("tier"),
        region=user_context.get("region"),
        as_of=date.today(),
        permission_group=user_context["permission_group"],
    )

Order matters. Apply permission and freshness filters before asking the model to reason. Fetching private or outdated context and hoping the prompt sorts it out is a bad bet.

Failure 4: an eval set of happy paths

The quickest way to improve a RAG system is a small, deliberately mean evaluation set. Skip the thousand examples. Begin with about thirty that mirror real ways the system breaks:

• questions whose answer isn't in the corpus
• questions that need two sources
• questions where an old policy and a new one conflict
• questions with similar wording about different products
• questions that depend on a table
• questions where the correct reply is to hand off to a human

A tiny retrieval harness could look like this:

from dataclasses import dataclass

@dataclass(frozen=True)
class RagCase:
    question: str
    required_doc_ids: set[str]
    forbidden_doc_ids: set[str]

def evaluate_retrieval(cases: list[RagCase], retrieve) -> dict:
    total = len(cases)
    hit = 0
    leaked_forbidden = 0

    for case in cases:
        results = retrieve(case.question)
        retrieved_ids = {item["doc_id"] for item in results}

        if case.required_doc_ids & retrieved_ids:
            hit += 1

        if case.forbidden_doc_ids & retrieved_ids:
            leaked_forbidden += 1

    return {
        "cases": total,
        "required_hit_rate": hit / total,
        "forbidden_leak_rate": leaked_forbidden / total,
    }

It's intentionally simple. It won't settle answer quality, but it catches plenty of retrieval failures before the model sees any context.

Failure 5: judging answers without the evidence

A RAG answer should be graded against the evidence actually supplied, not against the whole corpus. The distinction matters in three ways. If the right answer exists in the knowledge base but wasn't retrieved, the generator can't be expected to know it, and that's a retrieval failure. If the right evidence was retrieved and the answer ignored it, that's a generation failure. If the answer cites a source that doesn't back the claim, that's a grounding failure. Labels are needed for all three.

@dataclass(frozen=True)
class AnswerEval:
    question: str
    answer: str
    evidence_doc_ids: set[str]
    expected_claims: set[str]

def simple_claim_check(eval_case: AnswerEval) -> dict:
    answer_lower = eval_case.answer.lower()
    missing = [
        claim
        for claim in eval_case.expected_claims
        if claim.lower() not in answer_lower
    ]

    return {
        "passed": len(missing) == 0,
        "missing_claims": missing,
        "evidence_count": len(eval_case.evidence_doc_ids),
    }

Production setups may add human review or an LLM judge for subtler claims, but even a rough claim check beats a vague thumbs-up.

A better trace

Every answer should leave behind a structured trace:

import time
import uuid
from dataclasses import dataclass, field

@dataclass
class RagTrace:
    run_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    started_at: float = field(default_factory=time.time)
    query: str = ""
    rewritten_query: str | None = None
    filters: dict = field(default_factory=dict)
    retrieved: list[dict] = field(default_factory=list)
    evidence_doc_ids: list[str] = field(default_factory=list)
    prompt_tokens: int = 0
    completion_tokens: int = 0
    verifier_result: str | None = None
    latency_ms: int | None = None

def finish_trace(trace: RagTrace) -> RagTrace:
    trace.latency_ms = int((time.time() - trace.started_at) * 1000)
    return trace

With it, practical questions get answers. Was the right collection searched? Did filters remove the correct documents? Did the retriever surface the expected source? Did the evidence pack carry too much context? Did the verifier notice missing citations? Did the latency come from retrieval, generation or verification? Without answers to those, there's no RAG system yet, only a prompt with a search box attached.

Hybrid search is often the dull fix

Teams often jump from plain vector search to agentic RAG too fast. Hybrid retrieval deserves a try first. Semantic search shines when the user and the document use different words. Keyword search wins whenever precision on literal strings counts, such as an identifier, an error code, a field name or a clause of legal wording. Real systems frequently end up needing both.

def hybrid_rank(vector_results: list[dict], keyword_results: list[dict]) -> list[dict]:
    scores: dict[str, float] = {}
    items: dict[str, dict] = {}

    for rank, item in enumerate(vector_results, start=1):
        doc_id = item["doc_id"]
        scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (rank + 10)
        items[doc_id] = item

    for rank, item in enumerate(keyword_results, start=1):
        doc_id = item["doc_id"]
        scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (rank + 10)
        items[doc_id] = item

    return sorted(
        items.values(),
        key=lambda item: scores[item["doc_id"]],
        reverse=True,
    )

This reciprocal-rank style merge isn't fancy, yet it's useful. It rewards documents that show up in both result lists while still giving either retriever room to surface something important.

When agentic retrieval pays off

Agentic RAG fits problems that need investigation. Good candidates include cases where the first pass often lacks enough evidence, questions that hop across several documents, workflows choosing between SQL, vector search and API calls, models that must ask whether they have enough evidence and flows that gain from retrying with a different query.

Poor candidates are simple FAQ answers, low-latency autocomplete, stable documentation lookups and anything where each extra model call costs too much. The problem should justify agentic retrieval. If a metadata filter solves it, use the filter.

A production checklist

Before blaming the model, run through these:

1. Are documents versioned?
2. Are permissions enforced before retrieval?
3. Are outdated documents excluded by default?
4. Do chunks follow meaning rather than character counts alone?
5. Are exact terms covered by keyword search?
6. Are tables and images handled separately?
7. Do the evals include no-answer cases?
8. Are retrieved document IDs logged?
9. Are cited sources verified?
10. Is answer quality measured apart from retrieval quality?

Most RAG failures aren't mysterious. They're unobserved.

Where the evidence chain breaks

RAG makes LLM systems feel grounded, but grounding doesn't happen automatically. The model can use only the evidence it receives. The retriever can find only what was indexed correctly. The index can serve only the metadata that was preserved. An evaluator can catch only the failures someone bothered to name. This work is less glamorous than a demo, and it's where trust gets built.

So when a RAG system fails quietly, the prompt isn't the place to start. The better question is: where did the evidence chain break?