Ten Quiet Retrieval Bugs in Production RAG, and How Each One Gets Fixed

A support chatbot once explained, in lovely prose, the wrong refund policy. The retriever had returned a paragraph that used the word "refund" four times. It was topically close and it described a discontinued product line. The LLM did what it was told, which was to answer confidently from the context supplied. Nobody had told it the context was wrong.

That is the shape of most RAG failures that end up in a debugging session. A team spends two weeks swapping the model, rewriting the system prompt and adding "only answer from context" in bold, and the hallucination rate hardly moves. The problem was never generation. The model reasoned correctly over evidence that was already wrong when it arrived.

Ten specific mistakes erode retrieval quality in production RAG. They tend to surface in a fixed sequence: chunking errors first because they sit upstream of everything, then retrieval design, then reranking and embedding choice, and finally the measurement gap that lets all of them ship unseen. Each section below covers why the mistake happens, what it costs and what to change, and where studies contradict each other the contradiction is reported openly.

Why retrieval is where RAG breaks

Practitioner write-ups on production RAG during 2026 keep landing on the same diagnosis: most bad answers trace back to retrieval. The wrong documents come back, or the right ones are missed, or they sit too low in the ranking to be used. A common estimate is around 70% or higher. That number moves with the report and the domain, so it is a rough guide only, but independent sources agree on the direction. When a RAG system gets something wrong, the likeliest explanation is that the LLM was never shown the right evidence, so the first move is to inspect what the retriever returned before editing the prompt.

The ten mistakes map onto the pipeline in four groups. Numbers 1 to 4 corrupt what goes into the vector store. Numbers 5 to 7 concern retrieval architecture. Numbers 8 and 9 sit in the ranking and embedding stage. Number 10 is that none of it gets measured.

Terms used below

• Chunking means cutting source documents into passages before indexing. Embedding models and context windows both work better on short, focused units than on whole files.
• Dense retrieval (vector search) places queries and chunks in one embedding space and returns the nearest neighbours. It is strong on paraphrase and weak on literal strings.
• Sparse retrieval such as BM25 ranks chunks by shared query words, weighting rare words more. Codes, names and jargon match well. Synonyms do not.
• Hybrid search runs both retrievers side by side and fuses the two ranked lists, most often with Reciprocal Rank Fusion (RRF). RRF uses rank positions instead of raw scores, which solves the problem that BM25 values and cosine similarities are not comparable.
• Reranking re-scores a short candidate list, typically 20 to 100 items, with a cross-encoder that reads query and passage jointly instead of embedding them apart. The result is a far sharper order before the final top-k goes to the LLM.

Four mistakes in chunking

1. Taking fixed-size splitting as the default and never testing another

Splitting every document into 500-token windows regardless of content is what tutorials use first. It takes five minutes, works well enough to demo, and teams ship it without revisiting.

Studies disagree here, and saying so is more useful than faking consensus. Several 2026 comparisons favour fixed or recursive token splitting over semantic chunking. In one large benchmark of many strategies, recursive 512-token splitting won, in part because semantic chunking produced fragments of only 43 tokens on average, too small to carry context. Other domain studies point the other way: a peer-reviewed clinical decision support evaluation reported 87% accuracy for adaptive, topic-aligned chunking against 50% for a fixed baseline (p = 0.001). A third comparison found a well-tuned fixed size could match or beat semantic splitting on real datasets, in line with the fixed 100-word passages used in the original RAG paper.

No strategy wins everywhere. What every source agrees on is that the error is not choosing fixed-size splitting. It is choosing any single strategy without benchmarking an alternative on the actual corpus.

The practical response is to measure instead of theorise. Assemble 20 to 50 representative question and answer pairs, index the same corpus under two or three candidate strategies, and compare recall@5 and context precision. Content with built-in structure, such as API docs, legal filings and financial reports, usually does better when split along that structure first, since those units were already coherent:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Recursive splitting respects structure first, falls back to
# character boundaries only when a section is still too large.
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,       # tokens, not characters, if you swap in a token-aware splitter
    chunk_overlap=75,     # ~15% overlap, a reasonable starting point
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
)
chunks = splitter.split_text(document_text)

2. One chunk size for every document type

A size is picked once, early, from a rule of thumb, then applied to a mixed collection. API references, long policy documents and chat transcripts all get the same window.

Oversized chunks dilute the relevance signal. A 2,000-token chunk covering five topics yields a vector that is a blurry average of all five and matches none well. Undersized chunks lose the context a reader needs, like a function signature without its parameter notes or a clause without the section it modifies.

Size should follow content density. Technical documentation generally holds up around 256 to 512 tokens, which keeps single concepts focused. Narrative or policy text usually needs 1,024 to 2,048 to keep the surrounding context. These are starting points to validate, because the right figure depends on the embedding model and on how users phrase questions.

3. Forgetting overlap

Overlap is set once and forgotten, or skipped because it supposedly wastes storage. Without it, a sentence crossing a chunk boundary is cut in two. One chunk ends with the setup and the next opens with the conclusion, and neither is retrievable as a whole thought. This is a common and entirely avoidable reason for the complaint that the answer is in there somewhere but the model cannot find it.

An overlap of 10 to 20% of chunk size is a sensible default: 50 to 100 tokens for a 500-token chunk, cheap compared with the failures it prevents. More is not always better. Much higher overlap mostly adds storage and noise, and at least one systematic 2026 analysis using SPLADE retrieval found no measurable benefit from overlap in that setup. The golden set should settle it, not the default.

4. Ignoring the document's own structure

Treating each file as one blob is easier, whether it is a Markdown API reference, a contract with numbered clauses or a table-heavy financial report, but structure-blind splitting regularly cuts a heading off its section or a table off the paragraph that explains it. The typical casualty is a function description separated from its parameter list, so a question about that function retrieves half an answer.

The remedy is to split by structure first, along headers, sections, lists and tables, and to use size-based splitting only inside sections that stay too large. By mid-2026 most production chunking pipelines are hybrids for this reason: structure first, semantic splitting inside long unstructured stretches, fixed size as a last resort, and no single method imposed on a mixed corpus.

Three mistakes in retrieval architecture

5. Shipping vector search alone

Starter templates default to vector search. It is one API call, and it looks excellent in a demo where the test queries are conversational paraphrases, exactly what dense embeddings do best.

Real traffic looks different from demo traffic. Users type product codes, error strings, flag names, ticket numbers and proper nouns, which are short literal tokens. Dense models trained to cluster meaning tend to confuse them with related but wrong neighbours. Benchmarks show the gap repeatedly: a tuned hybrid setup beat either method alone on an e-commerce ranking test, and a practitioner benchmark showed vector-only recall@10 trailing a properly fused hybrid, mostly through missed exact identifiers. Figures vary across corpora, but the direction holds everywhere it was tested. Whenever queries mix conversational and exact-match styles, which is nearly always, hybrid wins.

The change is to add a BM25 or SPLADE index beside the vector index. Weaviate, Elasticsearch, OpenSearch, Qdrant and Milvus all support native hybrid queries now, and both LangChain and LlamaIndex ship fusion retrievers that wrap it:

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

bm25_retriever = BM25Retriever.from_texts(chunk_texts)
bm25_retriever.k = 20

vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 20})

# weights are a starting point for RRF-style blending; tune against your eval set
hybrid_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vector_retriever],
    weights=[0.4, 0.6],
)
results = hybrid_retriever.invoke(user_query)

6. Merging with a weighted average, not RRF

After adding BM25, the intuitive step is to normalise both score lists to the 0 to 1 range and blend them with a weight. It looks reasonable in a spreadsheet. But BM25 scores and cosine similarities come from unbounded, differently shaped distributions, and min-max normalisation is fragile. One outlier document that repeats a query term dozens of times can squash every other BM25 score toward zero after normalisation, distorting the fused ranking in ways that are hard to notice without looking for them.

Reciprocal Rank Fusion avoids the problem by using each document's rank position in each list and ignoring scores altogether. That is why RRF, and not weighted averaging, is the default in essentially every production hybrid implementation cited in 2026 engineering write-ups. The standard formula gives each document the sum of 1 / (k + rank) over the lists it appears in, with k=60 a common default that tempers the influence of any extreme rank.

7. One k for every stage

Top-k is usually one global constant reused at first-stage retrieval, at fusion and for the final prompt. When the first stage fetches only the few chunks the prompt can hold, recall is already gone before the reranker can fix anything: a chunk the vector search placed 14th never reaches it. The opposite error is feeding hundreds of candidates to a reranker with no pruning afterwards, which costs time and money and fills the prompt with weak material. The LLM's attention degrades under that load (the lost-in-the-middle effect), and it quietly cancels out the gains from reranking.

Think of k as a funnel instead. Pull a wide pool at the first stage, around 20 to 100 candidates, so the reranker has room to repair early ordering mistakes, then cut to a small final set of roughly 3 to 8 chunks for the prompt. Tune each stage's k against the eval set separately, since a single value cannot serve both purposes.

Two mistakes in ranking and embeddings

8. Leaving out reranking

Reranking adds a network call or inference step, with latency and, for hosted services, a bill. Under deadline pressure it is the first thing cut, and first-stage order gets treated as final.

First-stage vector search relies on bi-encoders, which compress a passage into one fixed vector before any query exists. That is what makes them fast at scale, and it caps their precision, because the fine-grained interaction between a given query and a given passage is lost. Cross-encoders recover it by reading both at once. Independent 2026 reports describe cross-encoder reranking as the best single addition for a pipeline with decent first-stage retrieval: gains of several points in precision or NDCG are typical, and the twenties appear on lexically difficult datasets, all for under 200 ms of extra latency at sensible pool sizes. One consulting account credited reranking with a 23-point precision jump on a client project, larger than any embedding or chunk-size change there. Once hybrid search works, this is the next thing to try.

The change is to insert a reranking stage between fusion and the final prompt. For self-hosting with no per-call fee, bge-reranker-v2-m3 is a popular open cross-encoder with good multilingual coverage. Cohere's Rerank API is the usual managed alternative when GPU infrastructure is unwanted. Both integrate with LangChain and LlamaIndex:

from FlagEmbedding import FlagReranker

reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

def rerank(query: str, candidates: list[str], top_k: int = 5) -> list[str]:
    pairs = [[query, c] for c in candidates]
    scores = reranker.compute_score(pairs, normalize=True)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [text for text, _ in ranked[:top_k]]

final_context = rerank(user_query, fused_candidates, top_k=5)

9. A general embedding model on a specialised corpus

The embedding model that came with the framework often goes straight onto a legal, medical or financial corpus, justified by a few manual spot checks. General-purpose models learn from broad web and encyclopedic text, so they separate specialist jargon less sharply than models trained on domain material. The shortfall hurts most where stakes are highest. In regulated fields a near-miss, such as a passage about the standard procedure in place of the specific exception someone asked about, produces an answer that is plausible and confidently wrong, which is worse than an obvious failure.

Before committing, run at least one domain-adapted or fine-tuned embedder against the general default on the golden set. When fine-tuning an embedder is out of reach, a cross-encoder reranker trained on a small set of labelled domain queries often closes most of the gap faster and cheaper, since it fixes first-stage errors without re-embedding the whole corpus.

The mistake that hides the others

10. Shipping on impressions

Evaluation infrastructure feels like overhead beside feature work, and eyeballing a few outputs seems enough when the demo looks good. By a wide margin this carries the largest downstream cost, because it conceals the other nine. Hallucinations technically grounded in retrieved context, correct documents in the wrong order, and chunks cut at precisely the wrong boundary cannot be reliably caught by reading outputs, and by the time users report them they are in production.

The remedy is a golden set of 50 to 200 representative question and answer pairs, covering edge cases such as ambiguous phrasing, multi-hop questions needing two documents and adversarial rewordings. Score it on a standard panel with two halves. The retriever is judged by context precision and context recall, while the generator is judged by faithfulness and how relevant the answer is. Two frameworks lead this space, and teams often combine them. Ragas is the usual place to explore metrics, and its synthetic generator can bootstrap a golden set from the corpus for a person to refine. DeepEval, built on pytest, is the usual way to make evaluation a real CI gate. A common split is Ragas for exploration and data generation, DeepEval for gating each pull request:

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, ContextualPrecisionMetric
from deepeval.test_case import LLMTestCase

def test_refund_policy_query():
    test_case = LLMTestCase(
        input="What is the refund window for the Pro plan?",
        actual_output=rag_pipeline.run("What is the refund window for the Pro plan?"),
        retrieval_context=rag_pipeline.last_retrieved_chunks,
        expected_output="Pro plan purchases can be refunded within 14 days of purchase.",
    )
    faithfulness = FaithfulnessMetric(threshold=0.8)
    precision = ContextualPrecisionMetric(threshold=0.7)
    assert_test(test_case, [faithfulness, precision])

Put a suite like this in CI so that a chunking tweak, an embedding swap or a knowledge-base update that degrades retrieval breaks the build before it breaks a customer's day.

How the mistakes vary by setting

• Internal enterprise search and document Q&A, the most common setup, depends heavily on hybrid retrieval. Staff paste ticket numbers, policy codes and SKUs alongside plain-language questions, and dense-only search keeps missing the literal part.
• Customer-facing support bots benefit most from reranking. A fluent but wrong reply there turns into a ticket or a lost account instead of a shrug from a coworker.
• Regulated industries (finance, healthcare, law) call for domain-tuned embeddings or rerankers and strict evaluation, because near-miss retrieval creates compliance and liability exposure.
• High-traffic consumer products work under the tightest latency limits for fusion and reranking. The goal becomes the best accuracy that still fits a retrieval budget under 200 ms, which favours small cross-encoders or self-hosted GPU rerankers over the biggest models.
• Coding assistants and developer-tool platforms are a newer, quickly growing case in which structure-aware chunking counts for more than in prose. Code and API references are highly structured, and careless splitting wrecks the very structure that made them searchable.

Choosing the next step

In the pre-launch phase, or while still testing whether RAG is viable at all, vector-only search is the right start: it is fastest to build and answers the architecture question within a day. Once real user queries exist, scan the logs for exact-match patterns such as product codes, ticket IDs and config values. If any show up, add BM25 with RRF fusion before anything else, since sources consistently report it as the largest single retrieval upgrade, and by a wide margin. Reranking follows, when hybrid search works and the aim is precision instead of raw recall. It is inexpensive for what it delivers and needs no index rebuild. SPLADE or domain-tuned embeddings should wait for teams with a mature evaluation pipeline that has already shown hybrid plus reranking leaves a gap, because they cost more effort and upkeep and are hard to justify without a baseline metric.

Headcount and budget change the answer. With two engineers, a managed reranking API (Cohere is the usual pick) beats running a cross-encoder in-house: at modest traffic the fees are small compared with the hours that GPU servers eat. Where MLOps staff and GPUs already exist, or traffic is heavy enough that per-request billing becomes a line item, hosting an open model such as bge-reranker-v2-m3 makes more sense.

What a good pipeline still costs

RAG is not a perfect fix. Its main strength is grounding: replies can cite retrieved sources instead of leaning only on what the model memorised, and the knowledge base can be updated without retraining. The extra machinery is real operational weight, though. There are two indexes to keep aligned, a reranker to watch for drift, an evaluation pipeline to maintain as a first-class artefact and a chunking scheme that needs re-checking when the document mix shifts.

RAG also does not eliminate hallucination. It reduces fabrication from missing knowledge and leaves untouched the case where the LLM misreads or misapplies context it was correctly given. Faithfulness metrics exist because "grounded in retrieved context" and "correct" are different properties, and a mature pipeline measures both.

Production readiness

• Evaluation in CI. Run the Ragas and DeepEval pairing on every prompt, retriever or knowledge-base change instead of checking once at launch.
• Index sync. Update the BM25 and vector indexes together or on one schedule. When a stale sparse index serves alongside fresh embeddings, retrieval drifts in ways that are hard to spot.
• Observability. Record the chunks retrieved for every query, not only the final reply. That lets a bad answer be traced to retrieval or to generation without speculation.
• Credentials. Keep reranking and embedding API keys out of the repository and rotate them through the normal secrets manager.
• Latency budget. Decide the end-to-end retrieval allowance early (often below 500 ms for interactive use) and divide it among fusion, reranking and query rewriting, so the overspend does not surface after the reranker is live.
• Spend tracking. Hosted rerank and embedding pricing grows with traffic and with candidate count, and scoring 100 candidates costs clearly more than scoring 20.
• Golden-set upkeep. Refresh the evaluation set as documents change. Context recall can move when new material has a different structure from the material the set was built on.

The short lesson

Look at what the retriever returned before changing the prompt or the model. Nine of the ten mistakes (poor chunking, no hybrid search, naive fusion, mis-sized candidate pools, skipped reranking, an unchecked embedding model) are retrieval problems with retrieval fixes. The tenth, no evaluation, is the reason the other nine so often go out unnoticed.

None of the fixes needs a research team. Adding BM25 beside an existing vector store, replacing score blending with RRF or dropping a reranking call between fusion and generation is closer to a day of engineering than a redesign, and each has consistent support across independent sources even where the numbers disagree. A practical start: pull two weeks of real user queries, count how many contain exact identifiers that vector search would blur, and build a 50-query golden set from failures already on hand. That is smaller than it sounds, and it shows which of the ten mistakes costs the most.