Debug RAG by Finding the First Stage That Failed

When a knowledge assistant gives a poor answer, changing the prompt is an easy reaction. It is useful only if the model received adequate evidence and then mishandled it.

A missing document, an incomplete extraction, and an unsupported conclusion are different failures. Treating them as one quality score makes debugging slower because the score does not tell you which part to change.

A practical evaluation loop should preserve the path from source material to the answer the user saw.

Build examples with evidence, not just answers

For each test question, record the passages that support an acceptable answer. Include qualifications that the answer must preserve and claims it must avoid.

An illustrative case might look like this:

{
  "question": "When should this deployment be rolled back?",
  "required_evidence": ["handbook-v3:section-8"],
  "must_include": ["the failed health-check condition"],
  "must_not_claim": ["every warning requires rollback"],
  "answerable": true
}

This record is more useful than a single expected sentence. It allows several well-written answers while retaining the conditions that make them correct.

Add questions the corpus cannot answer. Otherwise, a system that always generates something may look stronger than one that correctly acknowledges missing evidence.

Check the earliest failure first

Start with the source. Does the required information exist in the indexed text, and does it belong to the stated document? If not, fix source ingestion.

Next, inspect retrieval. Did the relevant evidence enter the candidate pool? If it did, did it survive filtering, reranking, and context assembly? These are distinct checkpoints. A passage found by search but removed before generation cannot support the final answer.

Finally, inspect the answer. Did it accurately represent the evidence, preserve exceptions, and avoid unsupported claims? If the context was sufficient but the answer failed, prompting or generation behavior becomes a reasonable target.

This order prevents a strong model from masking weak retrieval on easy questions, then failing unpredictably on questions it cannot answer from prior knowledge.

Use metrics that expose different mistakes

Recall at a chosen cutoff measures the fraction of labeled relevant items retrieved. Precision measures the fraction of returned items that are relevant. A hit-rate measure asks whether at least one relevant item appeared; that is weaker than recovering all evidence required for a multi-part answer.

Ranking metrics help distinguish evidence in first position from evidence buried near the cutoff. But relevance labels need care: one passage may mention the topic, while another directly answers the question. Treating them as equally useful can reward the wrong behavior.

For generation, assess support, correctness, completeness, and citation alignment separately. An answer can accurately repeat an outdated source. It can also contain a correct statement that the supplied evidence does not support. Those cases call for different repairs.

Keep operational quality in the same experiment

Record response time, failures, and resource usage alongside the quality judgments. Preserve the corpus version, retrieval settings, prompt version, and model identifier so comparisons are interpretable.

Change one major variable at a time when diagnosing a regression. For example, compare reranking on and off while holding the candidate pool constant. Changing chunking, retrieval, and the prompt together may improve the score while leaving the cause unclear.

Automated judges can help review many outputs, but inspect disagreements and sensitive edge cases manually. A judge's score is another measurement with limitations, not a replacement for an explicit rubric.

Evaluate what reaches the user

A correct backend response can still appear under the wrong query if the interface accepts stale requests. Include the React search interaction in end-to-end checks.

The useful outcome of evaluation is a specific next action: repair extraction, broaden retrieval, preserve context, revise generation, or fix delivery. That is a much stronger result than knowing only that an answer “felt wrong.”