Chunking Strategies for Production RAG: Fixed-Size, Semantic and Parent-Child

Suppose a claims handler at an insurer asks an internal assistant: what sign-off does a water-damage claim of 40,000 euros need, and what has to be checked before it can be paid? The assistant is backed by the current claims manual. The files were ingested cleanly, OCR worked, preprocessing removed the noise, the embedding model surfaced the right manual, and the reranker put it near the top.

The answer still comes back half-complete.

An experienced adjuster would not be surprised. The rule isn't contained in one paragraph. A narrative clause says that claims involving suspected prior damage need an independent inspection before payment. Under it sits an authority table. The next page holds a flow diagram showing that fraud screening and ownership verification must also be finished first. The document system adds the manual's identifier, its version and the region it applies to.

For the handler these are not four separate facts. They are one business rule expressed in four different formats: the paragraph sets the condition, the table names who can approve, the diagram gives the order of checks and the metadata says which version is in force.

A standard splitter set to 500 tokens may put the paragraph in one chunk, the top half of the table in a second, the relevant table row in a third and the diagram's extracted text in a fourth. Every chunk is valid. Every embedding call succeeds. The dashboard stays green. Yet the knowledge has been fundamentally rearranged. At query time the system must link the claim type to the amount band, map that to an approver, find the inspection requirement and recover the checks, all from fragments that can't be read alone. The language model receives pieces and writes an incomplete answer.

This is why RAG debugging misleads. Teams swap embedding models, raise top-k, buy a stronger reranker or rewrite the prompt. Sometimes it helps. More often it compensates for damage done at segmentation. Poor RAG quality is frequently a chunking problem wearing an LLM costume.

Most discussion of chunking shrinks to a single number: 256 tokens, 512 or 1,000, with how much overlap. Those are fair implementation questions, but the architectural one is different:

What must stay together so that it still means the same thing when retrieved on its own?

What follows runs from the simplest strategies to the techniques that appeared in 2024 and 2025, notes where each repays its cost, and ends with a production design that copes with messy enterprise documents.

Where chunking sits in the pipeline

Chunking is not an isolated text-processing step. It is the boundary between understanding clean documents and building searchable knowledge. Everything upstream decides what is available to split, and everything downstream depends on whether the pieces keep enough meaning, structure and provenance to be found correctly.

Picture the full path. Sources such as PDFs, tables, images, JSON, databases, APIs and code are ingested, parsed, cleaned and normalized. Only then does a chunking layer decide how each content type becomes a retrieval unit. Narrative can use structural or semantic splitting. Dense rule statements may suit proposition-level representations. Tables need boundaries aware of their schema. JSON should follow business objects. Images need visual descriptions linked back to the original.

So a production system rarely settles on a single chunk size for everything. One corpus can hold representations at the document, section, paragraph, proposition and token level together. Parent-child retrieval pairs small chunks for precision with larger parents for context, whereas late chunking, contextual retrieval and RAPTOR reconstruct information that cutting would otherwise discard.

Chunking also doesn't finish when vectors exist. Chunk metadata, document versions, parent-child links, access controls and source references must survive indexing, retrieval, reranking, context assembly and generation. Evaluation closes the loop: retrieval misses, duplicated context, wrong citations and shifting query patterns should flow back into chunking decisions. Production chunking is less about cutting documents and more about designing the retrieval units that everything else works on.

The central trade-off

Take a short policy excerpt:

High-risk corporate customers must complete Enhanced Due Diligence.

Facilities above EUR 10 million require Group Credit Committee approval.

Fully cash-collateralised facilities may follow the exception defined in Section 7.4.3.

Keeping everything together preserves context but yields a broad, blurry vector. Splitting aggressively gives sharp retrieval, but the approval rule can drift away from the customer class it applies to, and the exception can end up detached from the rule it modifies.

That is the whole tension: small chunks improve precision, large chunks preserve context, and production systems need a deliberate balance. Every chunking technique is a different route through this trade-off, and none removes the need to know your documents and the questions users actually ask.

Simple baselines

Fixed-size chunking

Fixed-size chunking cuts text into windows of a set length, usually in tokens, with optional overlap. It is simple, deterministic and cheap.

Counting tokens generally beats counting characters, because embedding models see tokens. Five hundred characters of German or Chinese can carry far fewer meaningful units than the same count in English. The method encodes the text into tokens, steps through them at a fixed interval and decodes each window. Overlap means each window starts before the previous one ended.

• Works well for: uniform prose, baseline benchmarks, cases where operational simplicity matters most and early prototypes that need a working pipeline first.
• Fails when: meaning depends on structure. A condition may end at token 499 while its exception starts at token 501, and the boundary knows nothing about the business logic.

What overlap does

Overlap is insurance against boundary failures. Without it, a sentence can be sliced across two chunks, and neither half stands alone. With it, each chunk carries a little of its neighbor, so both stay interpretable.

The price is real: more vectors, more embedding spend, more storage and result lists clogged with near-identical passages. A figure of twenty percent is a habit and not a law, so treat it as a parameter to test against retrieval metrics.

Fixed-size chunking earns its place not by being optimal but by providing a stable baseline. If a fancier method can't measurably beat it on your corpus, the extra complexity isn't justified.

Recursive splitting

Enterprise documents already encode useful boundaries in headings, sections, paragraphs, sentences and lists. Recursive splitting uses these before falling back to arbitrary cut points. LangChain's recursive character splitter, for instance, tries double newlines first, which usually mark paragraphs. If a piece is still too large it tries single newlines, then sentence punctuation, then spaces and finally single characters. The size limit is the ceiling that triggers each fallback.

The separator order matters more than the size number. For well-organized policies, procedures, manuals and technical documentation this is often a strong default.

One production caution: if ingestion already extracted the hierarchy, don't throw it away and ask a splitter to rediscover it. A section numbered 7.4 under a lending chapter already tells you which text belongs together. Keep that structure in metadata and use it to constrain the splits. Put plainly, don't destroy structure while chunking and then try to rebuild it during retrieval.

Recursive splitting assumes structure roughly follows meaning, and that isn't always so. One paragraph can contain three unrelated rules, and five paragraphs can describe a single process. That is where meaning-based methods come in.

Sentence and paragraph units

Tools such as spaCy and NLTK offer trained sentence detectors that handle the cases naive period-splitting breaks on: abbreviations, decimals, titles and ellipses. Paragraph chunking is simpler and often effective, since a paragraph is a grouping the author chose on purpose. For clean documents, splitting on paragraphs and merging short neighbors up to a minimum size is a sensible midpoint between fixed-size and semantic approaches.

The cost is variable chunk size. Some paragraphs hold 50 tokens and others 800, which complicates batching, raises tail latency during embedding and makes index size harder to predict. Whether the better boundaries justify that depends on the corpus.

Sliding windows as a recall tool

A sliding window is fixed-size chunking where overlap is treated as a retrieval mechanism and not just boundary insurance. The reasoning: a query vector matches best the passage that answers it directly. If that passage straddles a boundary in one segmentation, neither chunk may score well. With generous overlap, every passage of reasonable length appears whole in at least one chunk.

More overlap raises recall at the price of redundancy. Beyond roughly 30 to 40 percent, top-k lists fill with near-duplicates that burn context budget without adding facts, and 10 to 25 percent suits most corpora. Evaluate overlap by measuring the duplicate retrieval rate next to Recall@k, because lifting Recall@5 while doubling duplicates may not improve answers.

Meaning-aware strategies

Semantic chunking

Semantic chunking replaces the token-count rule with a question: has the topic changed enough to start a new unit? Imagine a section whose first two sentences cover inspection requirements, whose third turns to payment limits and whose fourth cites fraud rules. A fixed splitter can't see those turns, while a well-tuned semantic chunker can.

The usual recipe has three stages. Split the text into sentences with a linguistic model. Embed each sentence, or a small window of adjacent ones, with a sentence-transformer. Then compute the cosine distance between neighboring embeddings and place a boundary wherever the distance exceeds a percentile cutoff, often somewhere between the 85th and 95th percentile of distances in that document.

That threshold is the critical setting. Too aggressive and related ideas shatter into tiny, meaningless chunks. Too lenient and distinct topics stay merged, restoring the noise you wanted to remove.

It also costs more, because a full embedding pass must happen before boundaries are known. For short, tidy documents the cost buys little. For long, inconsistently formatted narratives, finding real topic shifts can noticeably improve retrieval. One honest limit: the method assumes topics shift gradually and detectably between sentences. In dense legal or regulatory prose, where one sentence may hold three conditions and two exceptions, sentence-level distances produce noisy signals.

Parent-child chunking

This is among the highest-impact patterns in production RAG because it resolves a tension every single-unit approach ignores: the best unit for retrieval is not the best unit for generation.

Take a 2,000-token section covering limits, exceptions, controls and escalation. Embedding it whole preserves context but produces a vague vector. Cutting it into 200-token pieces sharpens retrieval, yet the matched fragment may lack what the model needs to reason well.

Parent-child chunking separates the two jobs. A parent is a larger coherent section, such as a whole clause, procedure or subsection. Children are smaller focused fragments derived from it: single conditions, thresholds or exceptions. Only children are embedded and searched. When a child matches, its parent identifier fetches the larger source text, and that is what the model actually reads.

LlamaIndex and LangChain offer versions of this through hierarchical nodes, small-to-large retrieval and parent-document retrieval. Each child stores a parent ID in metadata, and a lookup after retrieval expands matches to their parents before context assembly.

In regulated settings, identifiers need real lineage. A readable ID combining the document, version and section is safer than an arbitrary UUID, because a child from version 7.2 that resolves to a parent from 7.1 produces context that looks plausible and is operationally wrong. The pattern also works across modalities: a child can be a serialized table row, a visual description of a diagram or a proposition, all sharing a parent that stands for one logical section.

Proposition chunking

Take a sentence like this one from a lending policy: risky corporate borrowers need enhanced due diligence plus a yearly review, and a committee must sign off whenever their exposure exceeds a set limit. It carries three distinct facts. A retriever answering how often such customers are reviewed shouldn't have to wade through the approval threshold to score a match.

Proposition chunking breaks text into atomic, self-contained statements, each retrievable alone. Extraction is typically done at index time by a language model with a carefully structured prompt that preserves every condition, threshold, qualifier and exception, resolves pronouns where the text allows and infers nothing that isn't stated. For the sentence above, the output might be three statements: such customers require due diligence, require annual review, and above the exposure threshold require committee approval.

Each proposition gets its own embedding, so the vector space becomes a space of atomic facts and not passages, which sharpens precision for narrow, fact-seeking questions. Two production constraints apply:

• The proposition is not the source of truth. It is a model-generated representation. In regulated work, retrieve with the proposition, then return to the authoritative text for grounding, reasoning and citation. The proposition is an index entry and not a claim.
• Cost scales sharply. One 10,000-token document can yield hundreds of propositions, each needing an embedding call, so the index grows quickly.

Restoring lost context

Late chunking

Conventional chunking splits first and embeds each piece on its own. Late chunking, which Jina AI introduced in 2024, flips part of that order.

Late chunking first runs the whole document through a long-context embedding model to obtain contextual token-level representations. Chunk boundaries are applied afterward, pooling the token vectors that fall inside each window. The difference from simply using bigger chunks is that the retrieval unit stays small. The passage is still focused, but its vector reflects the setting in which it appeared. Pronouns, abbreviations, forward references and terms defined earlier are resolved in the embedding and not left for the model at generation time.

The infrastructure cost is genuine. You need a long-context embedding model that exposes token-level outputs, extra indexing memory and careful pooling code. It also cannot fix bad segmentation: a badly serialized table or a diagram dropped in preprocessing stays lost. What it solves is context loss at embedding time and nothing upstream. For corpora with dense cross-sentence dependencies, such as regulations, contracts and manuals, the gain in embedding quality can be material.

Contextual retrieval

Anthropic's contextual retrieval, published in 2024, attacks the same isolation problem differently. It doesn't change the embedding architecture. It enriches each chunk's retrieval text at index time.

An isolated sentence stating a limit may not match a query mentioning customer class, jurisdiction and approval authority, since those terms live in the surrounding document. So a short, document-specific context statement is generated for each chunk before indexing, for example noting which section of which policy version the passage comes from and what it defines. That statement is prepended, and the combination is embedded.

In Anthropic's evaluation, combining contextual embeddings with contextual BM25 cut top-20 retrieval failures by 49 percent, and adding reranking cut them by 67 percent. Those numbers come from Anthropic's own corpora and aren't guarantees. Real gains depend on your documents, queries, embedding model and retrieval setup. The lesson isn't the figures but that adding document-specific context to isolated chunks can materially help whenever important meaning lies outside the chunk boundary.

It helps most with ambiguous fragments: stray table rows, image captions, mentions of defined terms and chunks whose scope came from a parent section they no longer include. It also pairs naturally with parent-child retrieval. One discipline applies: the generated context is a retrieval aid and not a source statement. Give every chunk the same generic blurb and the vectors become too similar, hurting retrieval. Good contextualization adds only what a chunk needs to be understood without its neighbors.

RAPTOR: several levels of abstraction

RAPTOR, short for Recursive Abstractive Processing for Tree-Organized Retrieval, targets a limit shared by single-level retrieval: fine-grained chunks answer narrow factual questions well but struggle with questions that need synthesis across sections. Compare asking who approves one particular claim with asking for the organization's entire control framework for that claim type. The first has one authoritative answer in a table. The second needs evidence from inspection rules, delegated authority, collateral or recovery policy, review procedures and escalation paths, probably across sections and documents.

RAPTOR builds a hierarchy. Source chunks form the bottom layer. Related chunks are clustered by embedding similarity and a language model summarizes each cluster. Those summaries are clustered and summarized again, repeating until a few high-level summaries represent the whole document or corpus. At query time, a narrow question can match a leaf while a broad one matches a higher-level summary, so one system handles both without architectural change.

The main production worry is provenance. Summaries are interpretations, and interpretations can distort thresholds, edge cases and exceptions, the very details that matter most in regulated work. Use upper-level nodes to find the relevant parts of the corpus, then resolve back to original passages before generation. The summary helps locate evidence, and the original text is what the model reasons from. RAPTOR earns its cost on long policies, regulatory frameworks and large manuals where users ask both detailed and synthesis questions in one application. For short collections or uniform query types the complexity rarely pays off.

Structured and non-text content

Tables: structure is meaning

Return to the authority table. It holds explicit relationships: claim type water damage, amount band above 25,000 euros, approver regional claims manager. Flatten that into a run of words and the column relationships that give each value meaning are gone, so tables need table-aware handling.

With small tables, keeping the table whole is often best, particularly when queries compare rows or need the full range of conditions. Write it out so column-to-row links survive, for instance by turning every row into label-value pairs that name each column. With large tables, rows make more precise chunks, but each row must carry its headers, because three bare values with no labels mean nothing alone.

A bigger decision lurks beneath serialization. If users ask deterministic questions such as listing every approver for amounts above a given figure, vector retrieval may be the wrong tool altogether. Tables queried through SQL or a dataframe will beat any embedding approach for exact-filter lookups. The principle: don't destroy explicit structure just to make everything look like text.

JSON and structured records

The same reasoning applies to JSON. A record with segment, rating, region, amount, approver and a review flag already shows which fields belong together, so running a token splitter over serialized JSON ignores that. Small records should stay whole, with a readable retrieval form that prints each field with its label. For large payloads, find natural business entities such as customers, accounts, transactions, screening events or case decisions, and chunk around them.

Inside a record, divide semantic fields from filterable ones. Fields that natural-language questions will touch belong in the embedding. Fields such as region, version, business unit and effective date usually work better as metadata filters on the index than as part of the vector. Pre-filtering on metadata is faster and more exact for categorical limits, and it keeps the embedding about meaning.

Code: syntax is the boundary

Code poses the same problem in another domain. A generic splitter that cuts a Python file at token 500 may sever a function from its return statement or a class from its methods. For code search, boundaries should follow syntax: functions, methods, classes and modules. Python's ast module can parse source into a tree and extract each function, async function and class as a complete unit, so chunks follow the language's semantics instead of approximating them with character counts. For repositories in many languages, tree-sitter offers one parsing interface across dozens of them and handles edge cases that regular expressions miss. Parent-child fits naturally: retrieve a method precisely, then expand to its class or file when the model needs wider context.

Diagrams and images

Don't treat OCR text as a full representation of a visual. A flow diagram carries meaning through arrows, sequence, branching, grouping and spatial layout, none of which OCR captures. Use a vision-capable model to write a retrieval description of what the image communicates and not only what words appear in it. Store a pointer to the original. The generated description is the index entry, while the source image is what the system shows as evidence. In regulated systems the citation should point to the original visual and not to its description.

Routing by content type

Real documents are mixed. A single manual may contain narrative rules, an authority table, a process diagram, JSON metadata and embedded references. There is no sensible reason to run one splitter over all four, since each carries meaning differently: a paragraph through linguistic continuity, a table through row and column intersections, a diagram through position, direction and grouping, and JSON through explicit keys.

The production answer is a chunking router, a dispatcher that identifies each extracted block's content type and sends it to the fitting strategy. The router need not be clever. Its value lies in making content-type routing an explicit design choice and not an accident of applying one generic splitter. For the claims manual, narrative would go to structural or semantic splitting, the authority table to table-aware serialization, the flow diagram to a vision model for a description and the metadata JSON to object-aware separation.

Every resulting chunk should carry the same core metadata, such as document ID, version, parent ID, section and region. That shared lineage lets later stages see that a table row, a paragraph and a diagram description are related evidence from one section.

When bigger units win

Much of the progression above moves toward finer granularity, which can create the false impression that smaller is always better. Research on LongRAG explored the opposite direction, testing whether much larger retrieval units, in some experiments up to around 4,000 tokens, could beat fine-grained retrieval on complex multi-hop questions. The finding wasn't that large chunks are universally superior. It was that retrieval granularity should match query granularity.

A question about how often an account type must be reviewed has a proposition-sized answer. Asking for the full control framework for a claim type needs evidence from several areas, and fine-grained retrieval may force the system to stitch a broad answer from many isolated fragments, so assembly errors appear even when each fragment is correct.

Choosing and measuring

How to pick a strategy

No chunking strategy is best everywhere. The choice depends on document structure, real query types, latency budget, indexing cost tolerance and the downstream generation design. Query type matters as much as document type. The same policy set may serve narrow lookups about a minimum review frequency and broad requests to explain an entire framework, and those want different retrieval granularity. If your evaluation data covers only one type, the winning strategy may underperform on real traffic.

A rough guide:

• Uniform prose, early prototype: fixed-size with modest overlap.
• Well-structured manuals and procedures: recursive or structure-based splitting, with hierarchy kept in metadata.
• Long narrative with inconsistent formatting: semantic chunking, if the added cost shows a measurable gain.
• Questions that need both precision and context: parent-child.
• Dense rule statements and narrow fact queries: propositions with parent retention.
• Chunks that lose meaning in isolation: contextual retrieval or late chunking.
• Mixed query types over long documents: hierarchical methods such as RAPTOR.
• Tables, JSON, code and diagrams: type-specific handling through a router.

Benchmark instead of guessing

Glancing at a few chunks and concluding they look reasonable helps debugging, but it isn't evaluation. The real question is whether a strategy helps the retriever recover the evidence needed for the queries users actually send.

Build an evaluation set covering different needs: a narrow fact tied to a single table row, a multi-evidence question requiring a narrative clause plus a process description, a conditional question about an exception and a synthesis question across sections. Link each query to the IDs of the retrieval units that should be found.

Retrieval metrics are not sufficient on their own. Record how large the index is compared with a fixed-size baseline, what embedding costs per document, the indexing and retrieval latency, context tokens per query, duplicate rate, answer correctness and citation correctness. Three extra points of Recall@5 are poor value if the index quadruples and the results fill with near-copies.

On a typical policy corpus a plausible pattern is that recursive splitting beats fixed-size by a few points at equal index cost, semantic chunking adds a bit more at modest extra cost, parent-child lifts recall further with a notable index increase, and propositions plus parents reach the highest recall at far greater cost and complexity. Contextual retrieval may match that recall with less index growth, which can justify its model call at indexing time, and RAPTOR may trail parent-child on narrow facts while winning on synthesis. Treat this as an illustration of the shape of the trade-offs and not a measured result. The right operating point depends on accuracy needs, latency limits, indexing budget and whether each increment justifies its complexity.

Don't optimize retrieval alone

Suppose heavy overlap improves Recall@5, but the top five now contain four near-identical passages from one clause, pushing a different relevant section outside the context budget. One metric improved and the system got worse. Evaluate the full path from retrieval through reranking and context assembly to generation, asking at each stage whether the needed evidence survived, whether duplicates consumed useful space, whether the answer is correct and grounded and whether the citation points to the right source and version.

In regulated systems, citation correctness isn't a nicety. An answer can be factually right while grounded in the wrong version of a policy. That is still a compliance failure, and it stays invisible unless citation provenance is explicitly tested.

A production architecture

The central change in thinking is that a mature enterprise pipeline hardly ever wants one universal chunker. It wants a routing layer that sends each content block to a suitable strategy, keeps shared lineage metadata on all resulting chunks and holds explicit links back to the authoritative source.

For the claims manual, a plausible setup is: narrative sections go to structural or semantic splitting; dense rule clauses become propositions with the parent retained; small tables are serialized whole; large tables become header-aware row chunks; diagrams get vision-model descriptions; and JSON records are separated by object. Every chunk, whatever the type, carries the same identifiers: document ID, version, parent ID, section, region and a reference to the source artifact. This shared lineage matters as much as the chunks themselves, since it lets retrieval reconnect a table row, a clause and a process description as evidence from one section of one version.

Back to the water-damage claim

The question was never whether 500 or 800 tokens was the right size. The source held different kinds of evidence. The paragraph established that an independent inspection was needed. The table established who approves the amount band. The diagram established the checks that precede payment. The metadata established which version and region applied.

A well-built chunking layer keeps each piece in its native structure and records that they share a section. That leaves retrieval with choices: approval questions fetch the table row, inspection questions fetch the clause, process questions fetch the diagram description, and complicated questions pull several related children and expand to their common parent. That is the gap between splitting text and designing retrieval units.

The order that counts is not fixed-size, then semantic, then proposition, with the newest idea winning. Rather: learn the source, learn the queries, keep existing structure, pick retrieval granularity, restore missing context, preserve provenance, measure retrieval and answers, and add complexity only where it earns its keep.

A last checklist before indexing

Before chunks reach the embedding pipeline, each should satisfy three properties:

• Retrievable. Specific enough to match the queries that matter, and not so broad that it matches everything loosely.
• Interpretable. Carrying enough context to keep its meaning alone, or having a reliable way to recover that context through a parent lookup.
• Traceable. Pointing back to the authoritative source: the exact document, version, section, page, table, image or record it was taken from.

When one of these is absent, the chunking layer is passing its problems down to retrieval, reranking, context assembly and generation. A vector store will happily index a poor unit, an embedding API will return a vector without complaint and a model may write a confident answer on top of it, none of which shows the representation is right. Fixing a broken unit is cheapest before it becomes an embedding.

Before shipping, also plan logging, secrets and configuration handling, deterministic IDs, versioning of models and prompts, chunk lineage tracking, access controls, evaluation datasets, index lifecycle management and monitoring of retrieval quality, latency and cost.