Why RAG Fails on Real Documents: Chunking, Hybrid Search, Reranking and Evals

Why demos break on real data

RAG tutorials usually run against one tidy PDF or a Wikipedia dump, which is the wrong exam. Aim the same pipeline at genuine company material, with uneven formatting, support tickets and sprawling internal wikis, and answers start to come back incomplete, invented or confidently wrong. Two ideas explain nearly every production failure.

1. RAG is first a retrieval system and only second a generation system. When retrieval surfaces the wrong context, the model has almost no chance of a correct answer however capable it is.
2. Overall quality is capped by the weakest stage, not the average one. A strong embedding model can't rescue a chunker that sliced a sentence in two.

Two phases

Every RAG system divides into an offline indexing phase and an online query phase:

Indexing (offline):   documents → chunks → embeddings → vector store
Query (online):       question → embedding → retrieval → context assembly → LLM

Production setups add caching, reranking, guardrails, monitoring and fallback logic around this core, but it remains the base for everything else, and no tooling downstream repairs a weak base.

Chunking pays off the most

Most systems fail at chunking, and the failure stays invisible until you're debugging a wrong answer three stages later. It matters because embedding models work best on coherent spans, context windows are finite, oversized chunks bring noise, and undersized ones drop the surrounding facts an answer needs.

Fixed-size chunking cuts every N tokens with 10 to 20 percent overlap. It's simple and predictable but blind to sentence and paragraph edges, so it can split a definition without noticing. It's a common starting point and rarely the best.

Recursive or hierarchical chunking tries natural breaks first, double newlines, then single newlines, then sentences, then words, and falls back only when a section is still too big. Most frameworks default to it today, and it's a clear improvement over pure fixed-size.

Structure-aware chunking follows the document's own layout: headers, sections, code blocks and tables. For technical docs and markdown it often wins outright. Keep parent-child links so you can match on a small precise chunk and then expand to the whole parent section when more context is needed.

Semantic chunking uses an embedding model or an LLM to find where the topic really changes and splits there. It gives the most coherent chunks at a genuine extra cost, since each boundary decision needs a model call, so it tends to be reserved for corpora where retrieval quality justifies the spend.

Sensible defaults:

• Begin with recursive or structure-aware splitting and skip fixed-size.
• Aim for chunks of roughly 300 to 800 tokens, then tune against your own evaluation set and not a number from a blog.
• Store metadata on every chunk, such as source document, section title and page or paragraph number. Filtering and citations need it later.
• Collect a small set of real questions and inspect what gets retrieved before touching the embedding model. Poor chunking quietly ruins results, so fix it before tuning anything downstream.

Embeddings and vector stores

Embedding models place text in a high-dimensional space where semantic closeness becomes geometric closeness, usually scored by cosine similarity or dot product. The model matters, but data quality and chunking typically count for more than which model sits in the pipeline. The broad families are general-purpose open models, domain-specific embeddings for legal, medical or code text, and proprietary APIs. Choose by your domain's vocabulary, not leaderboard rank alone.

A vector store's real job is fast approximate nearest neighbor (ANN) search, because exact brute-force comparison doesn't scale beyond a small collection. Most stores use graph-based indexes such as HNSW or cluster-based ones such as IVF so a query needn't be compared with every vector. The trade is a small, tunable chance of missing the true nearest neighbor in return for sub-linear query time. You rarely tune this early, but knowing it exists helps when you're asking why the obviously right chunk wasn't retrieved.

When choosing a store, ask whether it filters on metadata, whether it supports hybrid vector-plus-keyword (BM25) search natively, how easy it is to run locally versus managed, and what persistence, replication and scaling look like at your volume. Common options include Qdrant, Weaviate, Pinecone, Chroma, pgvector and OpenSearch k-NN.

In the beginning the brand of store counts for less than four other things: tidy data, solid chunking, proper metadata, and a way to mix vector and keyword retrieval. Hybrid search turns up in practically every serious deployment, since pure vector lookup keeps missing literal tokens such as error codes, product SKUs and IDs that a keyword pass handles with ease. The fusion of dense and sparse scores is usually controlled by a single weight, so one query can lean toward meaning or toward exact terms depending on the use case. A conceptual hybrid call looks like this, though the exact API varies by store:

# Conceptual shape of a hybrid query — exact API varies by vector store
results = vector_store.hybrid_search(
    query=user_question,
    alpha=0.5,          # weight between dense (1.0) and keyword (0.0) search
    filters={"doc_type": "runbook", "product": "checkout-service"},
    top_k=25,
)

Beyond top-k similarity

Grabbing the top-k closest chunks is where retrieval begins and not where it ends. Real traffic keeps exposing the same issues: the nearest chunk isn't the most useful, an answer's facts are divided among several chunks, models pay too little attention to material in the middle of a long context even when it is there (the lost-in-the-middle effect), and the embedding model may not understand your domain's vocabulary.

Techniques worth adding, roughly by return on effort:

• Reranking. Pull a wide candidate set, say the top 20 to 50 and sometimes 100, with cheap vector search, then rescore with a cross-encoder or small LLM before trimming to what enters the prompt. It's consistently among the best additions to a pipeline. A cross-encoder scores a query and document together, which is far more accurate than comparing two separately computed vectors, but it must run once per candidate and can't cover a whole collection. Bi-encoder search is cheap and approximate and narrows the field, and the cross-encoder is costly and precise and only inspects what's left.
• Hybrid search, combining dense vectors with sparse keyword search so exact tokens aren't left to chance.
• Query transformation: rewrite the question, generate a hypothetical answer document and search on it (HyDE style), or split a complex question into sub-questions retrieved independently.
• Metadata filtering, which narrows candidates by document type, date range or product before vector search runs and not after.
• Parent document retrieval, matching on a small precise chunk but returning its larger parent section so context is complete.

Measuring retrieval on its own

An evaluation set beats a gut feeling. Write 20 to 50 realistic questions, note which chunks should be retrieved for each, and check directly whether the right chunk lands in the top 5 or top 10 (Recall@k). Add an LLM judge for relevance only once manual inspection earns your trust. If retrieval quality is never measured apart from answer quality, time goes into optimizing the wrong stage. Open tooling such as Ragas covers RAG-specific metrics, so you needn't build the whole harness by hand.

LlamaIndex or LangChain, conceptually

LlamaIndex is data-centric, with strong opinionated abstractions for indexing, retrieval and query engines, and it feels natural when the core problem is a pile of documents needing good retrieval. LangChain is general-purpose, born from chains and agents, with a much bigger integration ecosystem. It's more flexible and correspondingly heavier when retrieval is all you need. Neither is magic, since both orchestrate the same underlying patterns. Pick one, learn its concepts well and don't get dogmatic, because the architectural knowledge carries over whichever syntax you write.

Recap

1. Indexing and querying are separate phases, so design them independently.
2. Chunking is a first-class design choice and not a framework default to accept.
3. Vector stores are infrastructure, and the payoff comes from filtering and hybrid search, not in the brand.
4. Retrieval quality has to be measured alone, apart from answer quality.
5. Frameworks are tools and not the architecture, so learn the patterns and treat the tool as replaceable.