Why RAG Fails in Production: Chunking, Embeddings, Retrieval and Evaluation

The standard RAG tutorial runs in a straight line. A PDF is loaded and cut into chunks, the chunks are embedded and stored in a vector database, and a user question is embedded, matched against the nearest chunks and pasted into a prompt. The answer comes back accurate, and it feels like the hard part is over.

Production disagrees. Questions span several documents. Relevant context gets cut at chunk edges. Retrieval returns passages that sit close to the query in meaning space but are wrong in context, with the right words and the wrong sense. The model then answers with confidence from material that is stale, contradictory or missing a qualifier that sat three sentences past the chunk boundary. The worst part is that the output sounds correct, and nothing signals that retrieval went badly.

A RAG demo is easy to build, and a version that behaves reliably over a real corpus with real users is not. The gap comes from three places: how documents are chunked, how embeddings handle nuance and how retrieval decides what counts as relevant. Each has failure modes that are easy to miss and costly to debug after launch.

Chunking, where information leaks away

The tutorial default is recursive character splitting with a fixed size and some overlap.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = splitter.split_documents(documents)

That is good enough for a demo. It fails when document structure carries meaning that character counts ignore, in three common ways.

• Split context. A spec might say that the maximum supported value is 500, and that the limit applies only in high-availability mode with the primary failover configuration active. Cut at 1,000 characters, the number may land in one chunk and the condition in the next. Someone asking about the high-availability limit retrieves the chunk that says 500 with no qualifier. The answer exists in the corpus, but the chunking severed the relationship.
• Orphaned headings. A heading such as Security Configuration and the first paragraph beneath it, which says TLS is required for every connection, can end up in different chunks. A retriever that finds the paragraph has no heading context, and the embedding of the sentence does not record that it belongs to security setup.
• Broken tables and lists. Fixed-size splitters often cut a table mid-row or a list mid-item. A column header in one chunk and the data in another leaves neither readable alone.

Production-grade chunking tends to rely on three techniques.

Structure-aware splitting cuts at the document's own boundaries, meaning headings, paragraphs and sections, in place of character counts. Markdown can be split on second- or third-level headers, and HTML on its heading, paragraph or section tags.

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "document_title"),
    ("##", "section"),
    ("###", "subsection"),
]

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
chunks = splitter.split_text(markdown_document)
# Each chunk now includes header metadata: {"section": "Security Configuration"}

Parent context enrichment stores with each chunk the heading path that leads to it. When the chunk is retrieved, the prompt receives the section and subsection titles next to the text, giving the model structure that the embedding space does not carry.

Parent-child chunking keeps small chunks for retrieval precision and larger parent chunks that contain them. Retrieval matches the small piece, and the prompt receives the parent. Small chunks embed specific ideas more accurately, while parents supply the surrounding information.

# Small child chunk for retrieval
child_chunk = "The maximum supported value is 500."

# Parent chunk injected into prompt when child is retrieved
parent_chunk = """
## High Availability Configuration
When operating in high-availability mode with primary failover enabled,
the following limits apply:
- Maximum supported value: 500
- Minimum heartbeat interval: 100ms
- Failover timeout: 30 seconds
"""

The approach costs more to implement and store, yet it yields much better answers for questions that need surrounding context, which covers most real questions.

Embeddings and the limits of similarity

Embeddings place text in a vector space where similar meanings sit close together, and retrieval returns the chunks nearest the query. That works whenever semantic closeness is a fair stand-in for relevance. It breaks when the two diverge.

• Negation. The sentences saying a system does not support IPv6 and that it does support IPv6 get nearly identical vectors, since both discuss IPv6 support. The negation barely moves the vector. A question about IPv6 may retrieve both chunks or the wrong one, and the model must reconcile a contradiction or answer from the wrong passage without noticing.
• Vocabulary mismatch. A user writes login while the docs say authentication and sign-in. These usually sit close enough for retrieval to work, but edge cases arise when the user's jargon differs from the documentation's usage.
• Specificity. A question about how to configure the connection pool and one about the default pool timeout land very near each other, so both return the same set of chunks. When the timeout value lives in one paragraph among several about pools, the best chunk is not always ranked first.

Three fixes address these limits.

Hybrid search combines dense retrieval, which captures meaning, with sparse retrieval such as BM25 or TF-IDF, which captures exact terms. A query for a version number, a configuration key or an error code is better served by keyword matching than by semantic similarity. Many vector databases now offer hybrid search natively, and the example below uses a sparse and dense combination in Qdrant.

from qdrant_client import QdrantClient
from qdrant_client.models import SparseVector, NamedSparseVector

# Many vector databases now support hybrid search natively
# This example uses Qdrant's sparse + dense hybrid search
results = client.query_points(
    collection_name="docs",
    prefetch=[
        models.Prefetch(
            query=sparse_query_vector,  # BM25 sparse vector
            using="sparse",
            limit=20,
        ),
        models.Prefetch(
            query=dense_query_vector,   # Embedding dense vector
            using="dense",
            limit=20,
        ),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),  # Reciprocal rank fusion
    limit=5,
)

Query expansion enriches the question before it is embedded. A query about authentication can be widened with login, sign-in, auth and identity verification, which places it in a richer part of the space. A small LLM call can produce the variations, or a static synonym map can cover domain terms.

Embedding model choice deserves a look as well. Many teams default to a general OpenAI model such as text-embedding-ada-002 or text-embedding-3-small. For medical, legal or technical corpora, a domain-specific or fine-tuned model often does considerably better. When retrieval stays poor despite good chunking, comparing alternatives on a representative sample of queries pays off.

Retrieval problems that appear at scale

Chunking and embeddings decide whether content can be found, while retrieval decides whether it is found for a given query. Some failures only show up with a large corpus or varied queries.

Choosing K. Most systems fetch the top K chunks by similarity and put all of them in the prompt. A small K misses facts when an answer needs support from several chunks. A large K fills the context window with marginal chunks that dilute the signal and, in long prompts, can push the truly relevant chunk beyond the model's effective attention.

The practical pattern is to over-fetch, say the top 20 or 30 candidates, and then re-rank. A cross-encoder takes the query and chunk together and scores relevance directly, rather than comparing two independently computed embeddings. It is slower because chunk embeddings cannot be precomputed, but it separates relevant from nearly relevant far better. Fast approximate retrieval narrows the field, and the cross-encoder picks the final set.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
# Retrieve broad candidate set
candidates = vector_store.similarity_search(query, k=20)
# Re-rank with cross-encoder
pairs = [(query, chunk.page_content) for chunk in candidates]
scores = reranker.predict(pairs)
# Sort by re-ranking score, take top 5
reranked = sorted(zip(scores, candidates), reverse=True)[:5]
final_chunks = [chunk for _, chunk in reranked]

Staleness. A vector store embedded six months ago holds documents that have since changed, and the embedding has no idea. Retrieval returns the outdated passage with high confidence, and the model answers from it. The remedy is metadata filtering: store each chunk's last-modified date and filter by date range before or after the similarity search. For actively maintained documentation, chunks older than a chosen threshold are dropped or ranked lower.

from datetime import datetime, timedelta

# Filter to chunks updated in the last 90 days
cutoff = datetime.now() - timedelta(days=90)

results = vector_store.similarity_search_with_filter(
    query=query,
    filter={"last_modified": {"$gte": cutoff.isoformat()}},
    k=10
)

False confidence. A model answers from whatever it receives, even when the retrieved chunks do not bear on the question. It does not know retrieval was poor. This does the most damage to user trust, because the answer reads as authoritative while being wrong. The fix is a relevance gate ahead of generation. Either score the retrieved chunks and suppress generation when the best score falls under a threshold, or tell the model in the system prompt to say it lacks enough information, and then check that it uses that escape hatch properly.

SYSTEM_PROMPT = """
You are a documentation assistant. Answer questions using only the provided context.

If the context does not contain enough information to answer the question confidently,
respond with: "I don't have sufficient information in the available documentation to
answer this question reliably."

Do not speculate or answer from general knowledge when documentation context is insufficient.
"""

Evaluation, the step teams skip

A RAG system with no evaluation has unknown quality. It may do well on some queries and badly on others, and the only way to find out is a user complaint. Evaluation has three parts.

Retrieval quality. For test queries with known correct source documents, measure how often the right document appears among the retrieved top K. Recall at 5 asks whether the right chunk is in the first five results, and mean reciprocal rank asks how high it sits. A test set of 100 to 200 representative queries should be run before launch.

Answer faithfulness. Does the generated answer reflect the retrieved context without adding claims the chunks do not contain? An LLM-as-judge setup can score this by receiving the query, the retrieved chunks and the answer, and rating how well the context supports the answer.

FAITHFULNESS_PROMPT = """
Given the following context and answer, rate whether the answer is fully
supported by the context on a scale of 1-5.

Context: {context}
Answer: {answer}

Score (1=not supported, 5=fully supported):
"""

# Automate across your evaluation set
scores = []
for query, context, answer in eval_set:
    score = judge_model.complete(
        FAITHFULNESS_PROMPT.format(context=context, answer=answer)
    )
    scores.append(score)

print(f"Average faithfulness: {sum(scores)/len(scores):.2f}")

End-to-end correctness. For queries with known answers, does the system produce the right one? It is the hardest part to automate and the most meaningful. Even a hand-built set of 50 questions with verified answers, run before and after every change to chunking, embeddings or retrieval settings, catches regressions.

The evaluation set works as a regression suite. A new embedding model, a different chunk size or an edited system prompt should each produce a measurable change in the scores. When a change lifts average faithfulness but lowers retrieval recall, or the reverse, the trade-off should be a conscious decision and not a silent deployment.

The pipeline is the product

The tutorial pipeline has four steps and no error handling. The production pipeline has six steps, failure detection at each, metadata attached throughout and evaluation before every release.

The failures are predictable. Chunking that ignores structure loses context at the edges. Embedding-only retrieval misses exact matches and handles negation badly. Retrieval without re-ranking returns a diluted top K that buries the best chunk. Generation with no relevance gate produces confident wrong answers from weak context.

None of this is unsolvable, and each fix has a known price. Structure-aware chunking requires understanding the document format. Hybrid search means running both dense and sparse indices. Re-ranking adds latency to retrieval. Evaluation requires building and maintaining a test set.

Teams whose RAG systems hold up make those decisions on purpose before launch, not in reaction to wrong answers. Tutorials skip the hard parts because a demo does not need them, but a trustworthy system does.