Home / Articles

Chunking: The Overlooked RAG Decision Behind Wrong Answers

Why splitting documents badly caps retrieval quality, how structure-aware and parent-child chunking help, and how to measure whether your chunks work.

Publisher: REACTAPP.TOP7 min read1686 words

This article is published in English.

Sources: Chunking: The RAG Decision Everyone Skips, and Shouldn’t

When a RAG chatbot confidently gives a wrong answer, the usual reaction is to blame the model. People swap in a larger LLM, rework the prompt or try another embedding model. Often none of that touches the real cause, which sits earlier in the pipeline, at the moment the document was cut into pieces. That step is called chunking. It has an outsized effect on retrieval quality and is easy to ignore because it comes before the parts that people normally tune.

A concrete case makes it obvious. A customer asks a support bot whether a damaged item can be refunded. The bot says refunds are accepted within 30 days. That sounds fine, but the policy also says the rule doesn't apply to items marked final sale. The bot never saw that clause. The model isn't at fault. The sentence simply sat in a different chunk from the one that was retrieved.

What follows covers what chunking is, why the naive version fails in exactly this way and what production systems do about it. It leans on vendor documentation and framework docs, and it avoids guesswork where it can.

What chunking is

An AI system that searches your documents first has to split them into smaller units. Two practical pressures push in that direction.

The first is retrieval precision. Embed and store a whole document as one item and a search for "refund policy" returns the entire file, most of it irrelevant. You want the specific passage that answers the question. Microsoft's Azure AI Search documentation makes this point directly: even when a document fits within a model's input limit, splitting it into finer pieces often retrieves better, because one vector cannot represent many different sub-topics well.

The second is the hard input limit of embedding models. Chunking prevents content beyond that limit from being silently cut off.

The pipeline then runs like this. A document is split into chunks. Each chunk becomes a vector, an embedding, and the vectors are kept in a vector database. A user's question gets embedded too, and the system fetches the chunks that rank highest under whichever retrieval method is in use: pure vector similarity, hybrid search that mixes vectors with keywords, or a reranking pass on top.

That final step is where chunking choices come back to hurt. The system can only retrieve what it was given. If the evidence for an answer is spread over two chunks and only one is retrieved, the model answers from partial context and has no way to know that the missing piece exists somewhere else.

Why the naive approach breaks

The simplest method is fixed-size chunking. You cut the text every N tokens or characters whatever it contains. It's popular because it is simple and predictable and needs very little extra processing.

Even Azure, whose guidance names fixed-size chunking as a sound starting point, says plainly that the right size and overlap depend on the shape of the content and the kind of queries users send, and that highly structured data may need different handling than flowing prose. The key point is that smaller isn't automatically better. Slicing too finely can separate facts that must be read together, which is precisely what happened with the refund policy.

Replay that example. Suppose the sentence about 30 days lands in one chunk and the final-sale exception lands in the next. A query about refunds matches the first chunk strongly because of the word "refund". The chunk with the exception is never pulled in. The model reasons correctly over what it received, but what it received was incomplete.

That is the central failure of chunking: retrieval quality is capped by chunk quality, however strong the embedding model or the LLM that follows. Teams spend weeks adjusting embeddings and reranking while the real bottleneck sits one step earlier, in how the sources were divided.

What production systems do

Two patterns come up repeatedly.

Structure-aware chunking

This approach stops treating a document as an undifferentiated string. Rather than cutting at every N tokens, it first splits along natural boundaries such as headings, sections and paragraphs, and applies a size limit only when a section runs long. Azure AI Search offers this as its Document Layout skill, where a document intelligence model picks out the structure and yields coherent sections for later chunking.

As a baseline before adding structure awareness, Microsoft's current guidance suggests starting near 512 tokens with roughly 25 percent overlap, while stressing that the best settings depend on the content and use case.

Parent-child chunking

Also called hierarchical or small-to-big retrieval, this pattern attacks the same problem from another side. Instead of fetching one flat chunk and hoping it holds everything relevant, the system finds a small, precise child chunk for accurate matching and then widens the result to include the larger parent section for full context.

Several popular tools implement it:

  • LlamaIndex builds a coarse-to-fine hierarchy of nodes with a HierarchicalNodeParser. Only the small leaf chunks are indexed for vector search, and their larger parents are kept in a separate document store. An AutoMergingRetriever then swaps a parent back in once enough of its children have been retrieved together.
  • LangChain's ParentDocumentRetriever follows the same idea. It embeds small child chunks for targeted search, and at query time it looks up the parent document each hit came from and returns that larger text, aiming to balance precise retrieval with enough length to keep context.
  • Amazon Bedrock Knowledge Bases describes the pattern in its own words: child chunks are retrieved first and then replaced with their broader parents, because small embeddings are more precise while the model still needs comprehensive context.

The split works because small and large chunks have different strengths. Small ones give sharp, focused embeddings and support high-precision retrieval. Large ones give the model enough surrounding text to avoid answering from a stripped fragment. Parent-child chunking lets each size do what it is good at, instead of forcing one size to serve both retrieval and generation.

It isn't universal, though. For corpora made of already self-contained units, like FAQ entries or product listings, or for strictly tabular and schema-driven data, the parent-expansion step tends to add overhead without matching benefit.

No single best setting

No chunk size or strategy wins everywhere. What fits depends on how your documents are laid out, the questions users really ask, your retrieval and reranking setup and how you evaluate the whole system.

Semantic chunking, which groups text by meaning and not by structure or size, is often assumed to be the more advanced answer. AWS documents it as a supported strategy alongside fixed-size and hierarchical chunking, but it brings extra parameters and trade-offs, so test it against your own documents and queries rather than presuming it wins. A more elaborate strategy only helps if it solves a failure you actually have.

Checking that chunking works

Choosing a strategy isn't a one-time decision. It's something to measure and revisit. Three measurements are worth tracking once the system is live.

  • Retrieval recall. Take any question: does the set of retrieved chunks include the evidence required for a right answer? If not, no downstream prompt work can rescue that answer.
  • Context precision. Of the chunks that came back, how much was truly relevant? Low precision usually means chunks are too large or too loosely bounded and drag in unrelated text.
  • End-to-end answer quality. With the right chunk in hand, did the model use it correctly? This catches problems in generation rather than retrieval, and keeping it separate from the other two tells you whether a bad answer comes from chunking, retrieval or generation.

Tracking the three separately, rather than just asking whether the final answer seems right, is what turns chunking from guesswork into something you can iterate on.

A small way to start

A cheap first experiment helps. Collect twenty or thirty real questions together with the passages that should answer them. Run the same set through two or three chunking configurations, such as fixed size with overlap, structure-aware splitting and parent-child expansion. For each, count how often the needed evidence appears in the retrieved set. The configuration that wins on your own documents is worth more than any general recommendation.

What to remember

Chunking isn't a preprocessing step to set once and forget. It's the first retrieval decision a system makes, and everything downstream, from embeddings to reranking to generation, inherits any damage done there.

A strong embedding model can't repair a badly chunked document. If answers from a RAG system feel random, incomplete or confidently wrong, inspect the chunks before touching the model.

Sources

The details above come from the public documentation of Microsoft Azure AI Search, including its guidance on chunking documents and on the Document Layout skill, the Azure-Samples repository on designing and evaluating RAG solutions, the LlamaIndex pages on the auto-merging retriever and node parsers, the LangChain reference for ParentDocumentRetriever and the Amazon Bedrock page on how content chunking works for knowledge bases. Settings change between releases, so confirm them against the current docs.