Chunk Documents Around the Evidence an Answer Needs

Imagine asking a deployment handbook when a service should roll back. Search returns the rollback instruction, but the exception is in the next chunk: the instruction applies only after a particular health check fails. The retrieved text is relevant, yet the answer can still be wrong.

This is the central chunking problem. The unit that is easy to index may differ from the unit that is sufficient to explain something.

Before tuning an embedding model, inspect what your chunks let the model see.

Start with the structure you actually have

For a technical article, a heading and its paragraphs are a sensible starting point. A code example should travel with the explanation of its assumptions. A table needs enough labels to interpret the values. An exception belongs close to the rule it modifies.

That depends on recovering trustworthy source text. If your parser has already flattened a table into an ambiguous sequence, smaller chunks will only distribute the ambiguity.

A practical first pass is to split at section boundaries, pack adjacent paragraphs within a size budget, and break oversized sections at sentence boundaries. Code blocks and tables need special handling when they exceed the budget; preserve their structure and repeat the labels needed to interpret each part.

This is a starting design, not a universal optimum. A glossary and an incident report organize evidence differently.

Use overlap for a specific reason

Overlap can preserve a sentence that would otherwise fall across a boundary. It also introduces repeated text. With enough overlap, several top results may contain substantially the same paragraph, consuming the context budget without adding evidence.

Measure useful coverage as well as individual chunk relevance. If five retrieved chunks all explain the rollback trigger but none includes the exception, retrieving more of the same text does not solve the problem.

After retrieval, identify overlapping passages from the same source and merge or deduplicate them before constructing the prompt. Keep their source locations so citations remain traceable.

Retrieve narrowly and expand deliberately

A small passage can be a good search target because it focuses on one idea. The surrounding section can be a better generation context because it supplies definitions and qualifications.

One design is to give every small chunk a parent section identifier. Search the smaller units, then expand selected hits into their parent sections when the answer needs more context. Expansion still needs a budget; loading an entire document for every hit simply moves the original problem downstream.

For the deployment handbook, the search hit might identify the rollback condition. The parent section can then provide the exception and the recovery steps. The answer gets a coherent explanation without requiring the retriever to rank one enormous document-sized chunk.

Compare configurations with the same questions

Create questions that expose boundaries: a definition, a comparison across paragraphs, a table lookup, and a rule with an exception. Label the evidence each answer requires.

Compare a simple fixed-size baseline with your structure-aware version. Keep the corpus, questions, and final context budget constant. Inspect whether each configuration retrieves enough evidence, how much duplication it introduces, and which answers remain impossible from the supplied context.

Do not confuse words with model tokens when setting production budgets. A word-count splitter is convenient for a prototype; the eventual prompt must fit the tokenizer and limits of the model that consumes it.

Once the evidence units are sound, hybrid retrieval can help users find them through both exact terminology and ordinary language. Better retrieval is most useful when the returned passages are already worth reading.