How to Debug Empty Retrieval When a RAG System Returns Nothing

Retrieval-augmented generation hands a language model outside knowledge so it doesn't have to answer from training data alone, which means the final answer is only as good as what retrieval brings back. When a search yields nothing useful, the model has little or no evidence. It may fall back on a generic reply, refuse, or invent details. What follows walks through where an empty or near-empty result comes from and how to track it down.

Retrieval can fail in many places

An empty result isn't a single kind of failure. A candidate can vanish during ingestion, embedding, vector search, metadata filtering, thresholding or ranking. Debugging goes quicker when you inspect each stage separately and stop treating the whole RAG stack as one black box.

1. Begin with indexing and embeddings

Before touching the retriever, confirm the knowledge base holds what you believe it holds. A healthy ingestion run shows a sensible progression from source documents to parsed text, then chunks, embeddings and stored vectors. If 2,000 chunks were produced and only 200 vectors landed in the store, no amount of retriever tuning will help.

• Open a handful of stored chunks by hand. They should contain readable text, stable IDs, embeddings and whatever metadata your filters rely on.
• Use matching embedding models and dimensions for documents and for user queries. Swap the query model without rebuilding the index and similarity scores lose their meaning.
• Judge the parser too. PDFs, scans, tables and HTML pages may turn into blank or broken chunks while the job still says it succeeded.

2. Look at thresholds, score meaning and filters

Aggressive score thresholds are a frequent cause of zero results. Embedding scores aren't confidence percentages, and the useful range varies with the model, the distance metric, the data and the vector database. While debugging, pull the top 10 or 20 candidates with no threshold at all and read their raw scores:

results = vector_store.search(query_embedding, top_k=10)
for r in results:
    print(r.score, r.metadata.get('source'))

Then check which direction scores run. Some APIs return cosine similarity, where higher is better. Others return cosine or Euclidean distance, where lower is better. Using the wrong comparison silently discards your strongest candidates and keeps the weak ones. As a rule, write no threshold condition until you've confirmed the database's scoring convention and looked at real score distributions from representative queries.

Filters can remove good results

Metadata filtering is vital for multi-tenant systems, permissions, product scopes, dates, languages and document types, and it's also among the simplest ways to end up with an empty set. Typical culprits include strings compared to integers, inconsistent capitalisation, older records with missing metadata, wrong tenant IDs, and filters that are each valid yet impossible together.

3. Review query rewriting and chunk quality

Rewriting a query can help retrieval, but it may also strip out the very term that made the query findable. Picture someone asking how to reset the X500 while the rewriter turns that into "restore default configuration for the X-series". Should the manual be called "Resetting the X500", the user's own phrasing will likely match it better. Log both the original and the rewritten query and test each one. For exact identifiers, error codes, product names and API parameters, keep the literal tokens even if the rest of the query is normalised.

Chunking fails in a similar way. A heading such as "Resetting the X500 Router" mustn't be separated from the instructions beneath it. Chunk along the document's structure so a heading, its paragraphs, its list items and the surrounding context stay in one piece. The right chunk size depends on the workload, though a reasonable starting range is about 500 to 1,000 tokens with modest overlap, then evaluation on real queries.

What to measure: take a small set of labelled queries and record whether the expected source appears in the top-k results. That recall-focused test beats choosing chunk sizes or thresholds by gut feeling.

4. Mix lexical and vector search

Vector search shines when the user and the document phrase one idea differently. Lexical retrieval such as BM25 shines when exact tokens matter. Since hard queries are rarely all of one type, production systems often use both. BM25 tends to win for things like error codes (ERR-XK-1042 for instance), SKUs, invoice numbers, product names, class names, config keys and API parameter names. Vector search adds nearby conceptual matches, while BM25 makes sure exact lexical evidence isn't missed. A typical design retrieves a broad candidate set first and then applies a stronger reranker for precision.

5. Retrieve widely, then rerank

The first-stage retriever and the reranker do different jobs. The retriever should maximise recall by collecting anything that might answer the question. The reranker should maximise precision, comparing the query against those candidates and lifting the best evidence to the top.

A common pattern gathers 20 to 50 candidates from vector and lexical search, merges and deduplicates them, then reranks down to the best 5 to 10 chunks. Cross-encoders are efficient and built for relevance scoring, while an LLM reranker can follow richer instructions but is usually slower and costlier.

Avoid cutting candidates early. If a hard threshold runs ahead of the reranker, a chunk that really would have helped may never get seen by the stronger relevance model. Often a loose first pass followed by reranking works better than asking the vector score to carry the whole ranking.

Fusion and deduplication

When vector and BM25 results are combined, one chunk can show up in both lists and the score scales don't line up. Use a fusion method such as reciprocal-rank fusion, normalised weighted scores or or simple rank merging, and remove repeats using stable chunk or document IDs ahead of the reranker. The design goal is to tune the first stage for recall and let the reranker and context-selection logic supply precision, which makes the pipeline easier to tune and evaluate.

6. Widen the query, with a planned fallback

People and documents rarely share vocabulary. Someone typing "I can't sign in" may need pages titled "authentication failure", "login troubleshooting" or "account access". Expansion generates a few alternate searches, runs them all, and then deduplicates and reranks what comes back. Keep it on a leash: leave named entities and exact identifiers alone, don't spawn lots of near-identical queries, and limit retries so that one hard question can't trigger an expensive loop.

Only loosen what is safe to loosen. Security, tenant and permission filters are never optional and stay on. Something like an optional document category, a tight date window or secondary tags can be relaxed if your application rules permit. One simple sequence:

docs = retrieve(query)
if not useful(docs): docs = retrieve(expand(query), hybrid=True)
if not useful(docs): return ask_for_clarification_or_no_answer()

Empty versus bad retrieval

The two cases call for different remedies. If nothing comes back, suspect ingestion, filters, permissions, thresholds, score interpretation or an index you can't reach. If results arrive but are off-topic, look at chunking, embedding quality, vocabulary gaps, thin lexical coverage or ranking.

7. A more resilient production architecture

A mature RAG system doesn't push every query down the same search path. It classifies the request and picks a retrieval strategy to match the need: exact identifiers lean on lexical search, broad conceptual questions lean on vector search, and ambiguous or mixed queries use both.

After that, an aggregation step gives you one spot to merge sources, drop duplicate chunks, apply the metadata and permission rules that must hold, and send a single consistent candidate list on to the reranker. Confidence shouldn't rest on one vector score. Better signals are the reranker score, the spread between the top hits, how many sources contribute, whether the entities needed for an answer actually show up in context, and whether several chunks agree.

Confidence decides the next step

With enough confidence, the top-ranked evidence goes to the LLM. With weak confidence, the system can expand or reformulate the query, retry, or ask the user a clarifying question. That beats always answering, because the model is told outright when the evidence is too thin to support a grounded reply.

Keep security out of the retry loop

Any fallback design has to keep relevance rules apart from authorisation rules. Expansion can loosen wording, and optional topic filters can sometimes go, but tenant boundaries, ACLs, row-level permissions and privacy policies stay fixed through every retry. The retry route must never turn into a way around access control.

8. Make retrieval observable

With only the final answer in your logs, debugging RAG turns into guessing. The retrieval layer ought to emit enough data to replay what happened to any query. Record at least: the original query, rewritten or expanded versions, filters, the embedding model with its dimensions, which retriever ran, top-k, how many raw candidates came back, their scores, chosen chunks, reranker scores, latency and which fallback ran.

log({
  'query': query, 'rewritten': rewritten_query, 'filters': filters,
  'candidate_count': len(raw_results), 'scores': scores,
  'selected_chunk_ids': selected_ids, 'fallback': fallback_stage
})

A sensible debugging order

When an empty-retrieval incident happens, move from the cheapest, most deterministic checks toward the more subjective ranking changes. That keeps teams from tuning prompts or embeddings when the actual problem is a missing index or an impossible filter.

1. Ingestion first: check document counts, parsed text, chunk counts, stored vectors and a few records you know should exist.
2. Raw retrieval next: send the original query with all optional filters and thresholds off, then read the top-k hits and scores.
3. Bring filters back one by one, which pinpoints the exact filter that empties the set.
4. Verify score semantics: is higher or lower better, and is any normalisation applied?
5. Compare original and rewritten queries, keeping exact identifiers, to see whether rewriting helps or hurts recall.
6. Evaluate hybrid retrieval and reranking against a labelled query set, checking whether expected sources appear in the top-k and top-n after reranking.
7. Test the fallback by forcing empty and weak results, to confirm the system clarifies or declines and doesn't hallucinate.

What to measure

Answer quality alone doesn't tell you enough about retrieval. Track recall@k (did the expected source make the candidate set?), precision after reranking, the empty-result rate, the fallback rate, latency, and the share of queries that needed expansion or clarification. Break these down by query type, such as semantic questions versus exact identifiers, and by tenant, language and document collection, because overall averages can hide a failing segment.

Don't blame the LLM first

When RAG returns nothing, resist pointing at the model. The cause usually sits earlier in the search pipeline: incomplete ingestion, incompatible embeddings, strict thresholds, wrong metadata filters, misread scores, aggressive query rewriting, or a retrieval strategy that doesn't suit the query. Dependable systems pair sound indexing with hybrid retrieval, broad candidate generation, reranking, controlled query expansion, explicit fallback behaviour and strong telemetry. The useful mental model is that RAG is a search and ranking problem first and a generation problem second. When the retrieval layer reliably supplies relevant, authorised evidence, and knows when it can't, the model becomes more accurate, easier to explain and less prone to hallucinate.