Parent-Child Chunking and Hierarchical Indexing for Better RAG Retrieval

When a RAG system returns an answer that is almost right but missing the one detail that mattered, the weak point is usually how the text was split, not the embeddings. Two techniques address this directly: parent-child chunking, which searches small pieces but reads big ones, and hierarchical indexing, which narrows the search by following a document's own outline. Both aim to give a language model precision and context at the same time.

Why splitting is the hard step

The basic pipeline is simple: cut documents into chunks, embed them, store them, fetch the top-k for a query and hand them to a model. When it breaks, the break usually traces back to the very first step. Chunks that are too large dilute relevance and blur the embedding, and chunks that are too small cut away the surrounding sentences a question depends on. Fixed-size, recursive, semantic and sentence-window splitting are all attempts to balance that tension within one unit of text.

A different premise is worth trying: stop hunting for one perfect size and use two sizes together, each tuned for a different job. Before introducing that fix, it helps to see precisely where a single size fails.

A small failing example

Consider a two-sentence passage from a technical document. The first sentence says caching makes a system much faster. The second warns that poor cache invalidation can leave stale data behind.

A small-size splitter could cut it into two chunks. Chunk A is the sentence praising performance, and chunk B is the sentence about stale data from bad invalidation. Now ask: what is a risk of caching? The script below embeds the question and both chunks, then compares them by cosine similarity.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

query = "What is a risk of caching?"
chunk_a = "Caching improves system performance significantly."
chunk_b = "However, improper cache invalidation can lead to stale data issues."

query_embedding = model.encode(query)
score_a = cosine_similarity(query_embedding, model.encode(chunk_a))
score_b = cosine_similarity(query_embedding, model.encode(chunk_b))

print(f"Chunk A similarity: {score_a:.4f}  - '{chunk_a}'")
print(f"Chunk B similarity: {score_b:.4f}  - '{chunk_b}'")

Running it prints the similarity of each chunk to the question.

Chunk A similarity: 0.5891  — 'Caching improves system performance significantly.'
Chunk B similarity: 0.5103  — 'However, improper cache invalidation can lead to stale data issues.'

Chunk A scores higher, around 0.59 against roughly 0.51, even though chunk B is the one that answers the question. Both mention caching, so both look related, and the embedding model has no special reason to connect the word "risk" with a sentence about stale data that never uses that word. A retriever that takes only the single best match would confidently pass the wrong half of the passage to the model.

The lesson is that semantic similarity does not guarantee completeness. A chunk can be rightly judged on-topic and still lack the fact the question needs, and the smaller the chunk, the more likely that fact sits in a neighboring sentence.

Parent-child chunking

The idea is to store two granularities of the same text at once, each doing a different job.

Child chunks are small and precise. They typically run 100 to 400 tokens and are meant for embedding search with high recall. Their small size produces sharper vectors, because less unrelated material dilutes what each vector represents. Going down to a handful of words backfires, though, since a fragment may lack the standalone meaning needed for a reliable match.

Parent chunks are large and contextual. They often span 800 to 3000 tokens or more, covering a whole section or logical unit, and exist for reasoning and not matching. In the caching example, a parent holding the full passage contains both halves of the idea, so whichever child matches, the model still sees the complete picture. The upper bound comes from the model's context window, since several large parents assembled into one prompt can crowd out everything else the prompt must contain.

The relationship fits in a sentence: children are what gets searched, and parents are what gets read. A child must be precise enough to win the similarity contest, and its parent ensures the model receives the context that child's answer depends on.

How it looks in practice

Imagine a section titled "Database Performance Optimization". A parent-child splitter works in two passes. First it identifies the parent boundary, the whole section covering indexing, query optimization, caching and trade-offs. Then it splits that same text into small children inside it, for example:

• a child about indexing improving query speed by cutting scan time,
• a child about caching reducing repeated computation at the risk of staleness,
• a child about rewriting SQL queries for optimization.

What makes the scheme work is that each child remembers where it came from. The simplest method stores the parent's full text in each child's metadata, as in this indexing function.

def build_parent_child_index(document_section, parent_text, child_splitter, embedding_model):
    child_chunks = child_splitter(parent_text)
    records = []
    for child_text in child_chunks:
        records.append({
            "text": child_text,
            "embedding": embedding_model.encode(child_text),
            "parent_text": parent_text  # full parent stored directly on every child
        })
    return records


parent_text = """Indexing improves query speed by reducing scan time across large tables.
Caching reduces repeated computation but may introduce staleness if not invalidated properly.
Query optimization involves rewriting SQL queries to use more efficient execution plans."""

child_texts = [
    "Indexing improves query speed by reducing scan time across large tables.",
    "Caching reduces repeated computation but may introduce staleness if not invalidated properly.",
    "Query optimization involves rewriting SQL queries to use more efficient execution plans.",
]

records = [
    {"text": t, "parent_text": parent_text} for t in child_texts
]

for r in records:
    print(f"Child: {r['text'][:50]}...")
    print(f"  → linked to parent ({len(r['parent_text'])} chars)\n")

Duplicating the parent onto every child costs extra storage but needs no separate lookup at query time, because the parent already sits in the child's record. The alternative is to store only a parent_id and fetch the parent when needed, which saves space at scale and adds one lookup step. Either way each child stays independently searchable, compared against queries like any other chunk, while carrying a direct path back to its fuller context.

Retrieve small, expand big

This flips the usual flow. Standard RAG goes retrieve, send, answer. The parent-child version goes retrieve a child, expand to its parent, send the enriched context, answer. Here is that flow implemented, continuing the caching example.

def parent_child_retrieve(query, child_records, embedding_model, top_k=3):
    query_embedding = embedding_model.encode(query)
    scored = [
        (np.dot(query_embedding, r["embedding"]) /
         (np.linalg.norm(query_embedding) * np.linalg.norm(r["embedding"])), r)
        for r in child_records
    ]
    scored.sort(key=lambda x: x[0], reverse=True)
    top_children = scored[:top_k]

    # Expand each matched child to its parent - but deduplicate first
    seen_parents = set()
    expanded_context = []

    for score, record in top_children:
        parent = record["parent_text"]
        if parent not in seen_parents:
            expanded_context.append(parent)
            seen_parents.add(parent)

     return expanded_context

for r in records:
    r["embedding"] = model.encode(r["text"])

context = parent_child_retrieve("What is a risk of caching?", records, model, top_k=2)
print("Context sent to the LLM:\n")

for c in context:
    print(c)

Pay attention to the seen_parents check. When the top two children belong to the same parent, which is common because neighboring chunks of one section often score alike, expanding each one separately would send the same parent text twice and waste context. Deduplicating on the parent makes each piece of context appear once however many children matched.

For the caching passage, the model now receives the sentence that won the match plus the explanation around it, so both halves arrive together regardless of which child ranked first.

Why it works

• Precision with context. Children find the right spot, and parents supply what a single child can't carry.
• Better reasoning. Models struggle with disconnected fragments, and a parent restores the narrative thread.
• Fewer invented details. Seeing surrounding constraints and explanations, the model has less reason to fill gaps with guesses.
• Technical material. API references, SDK docs, engineering wikis, papers and legal text rarely stand on a single sentence, and the exception or condition two sentences later is often what matters.

Hierarchical indexing

If parent-child is two levels, hierarchical indexing is many. Instead of one parent and its children, you build a structure that mirrors how the document is organized. A sample hierarchy might look like this.

Document: "Cloud Architecture Guide"
  Section: Networking
    Subsection: Load Balancing
      Chunk: Round-robin method
      Chunk: Least connections method
    Subsection: CDN usage
  Section: Security
    Subsection: IAM policies
    Subsection: Encryption

Note where this hierarchy comes from. Some systems route by category, such as department, topic or metadata tags assigned independently of any one document. Here the hierarchy comes from the document's own headings and sections, the way its author arranged it. Both narrow the search space before searching, but one organizes a whole knowledge base by topic and the other organizes a single document by its table of contents.

Most documents aren't flat text. Headings and subsections already encode how ideas relate. Hierarchical indexing keeps that structure and lets the system find the right section, then the subsection, then the exact chunk, much like a person skimming a table of contents before reading closely.

Retrieval in action

Suppose the query is "How does least-connection load balancing work?" The system finds the Networking section, narrows to the Load Balancing subsection, and then retrieves the chunk explaining that the least-connections method picks the server with the fewest active connections.

Each narrowing step needs a real mechanism, because none of it happens by itself. Options include a trained classifier that predicts which section a query belongs to, an LLM call that reads the query and picks the likeliest section from a known list, or rule-based matching on section titles and keywords. Whichever is chosen runs at every level, once for the section and again for the subsection, before the chunk-level search starts.

Is the extra depth worth it? Two levels are enough for most documents. A full multi-level hierarchy pays off for long, deeply structured material where one section contains enough distinct subtopics that searching all of it at once would dilute relevance. A short FAQ page or single-topic article rarely needs more than two levels, while a hundred-page architecture guide with dozens of subsections might. As with any indexing decision, measure before adding complexity.

Quick tuning tips

Start with child sizes near the middle of the range, around 200 tokens, and check recall on a handful of real questions before changing anything. Log which parents get expanded for each query so you can spot sections that are always pulled in and may be too broad. Cap the number of parents sent to the model, since three or four large sections can already fill a modest context window, and keep the original child text available for citations so users can see exactly which sentence matched.

Comparing the two

• Scope: parent-child works on two granularities of the same text, while hierarchical indexing models the whole outline.
• Main benefit: parent-child rescues context around a match, while hierarchy narrows where the search begins.
• Cost: parent-child needs links between children and parents, while hierarchy needs structure detection and a routing step at each level.
• Fit: parent-child suits most long documents, while hierarchy suits large, deeply structured sources.

They combine well, since a hierarchy can end in parent-child chunks at the bottom.

A pipeline you can ship

1. Ingestion. Parse documents and detect headings. The loader must keep heading information, meaning which text belongs to which section, instead of flattening everything into one stream, because the next stages depend on it.
2. Chunking. Build parent chunks from full sections, then split each into child chunks, identifying the parent boundary first.
3. Indexing. Embed the children and store the parent links, either by copying the parent text into each child's metadata or by storing a parent_id for lookup.
4. Query. Embed the question, search the children, take the top matches, fetch their parents with deduplication, assemble the enriched context and send it to the model.

Mistakes to avoid

• Children only. You gain precision but lose reasoning context, which is the failure in the caching example, where chunk B never gets the sentence that completes the answer.
• Parents only. You keep context but lose retrieval accuracy, because embedding a multi-topic section as one vector blurs it the same way an oversized chunk does.
• Ignoring structure. If the loader discards headings, there is no section boundary left for hierarchical routing to use, however good the router is.
• Weak metadata. Without explicit parent links such as parent_text or parent_id, expansion becomes impossible later. Metadata has to be captured when the chunk is created, since it can't be recovered once discarded.

Where it pays off most

These patterns shine wherever documents are long, structured and interlinked: enterprise knowledge bases, internal documentation, support bots, legal and compliance archives, engineering wikis and research assistants. In such material no single chunk size can satisfy both precision and context.

RAG is drifting from plain search-then-generate toward a model that first walks through organized knowledge and then reasons about what it found. Parent-child chunking improves local understanding, so wherever a match lands the surrounding explanation comes with it. Hierarchical indexing improves global navigation, so the search starts in the right neighborhood and not across everything. Flat chunking resembles ripping pages from a book and trusting the model to guess the plot. These two techniques leave the book bound while still letting you look up a precise line. In practice that means fewer answers that are nearly right, which is the failure that frustrates users most, and a retrieval layer whose behavior is easier to explain when someone asks why a particular passage reached the model.