Index-Time RAG Quality: Chunk Types, Cleanup and Retrieval Labels

Where a demo turns into a production system

A RAG parser hands over an ordered list of nodes, and each node knows which section it came from. Whatever happens from there until a model reads a query is index-time work, and that stretch is what separates a demo from something you can run in production. It breaks into three jobs. Nodes get turned into chunks, of which there are three kinds with their own builders. Chunks are scrubbed of material the parser extracted dutifully but the model has no business seeing. And every chunk receives a short generated label from a language model. On an internal evaluation over banking documents, those labels roughly halved the number of failed retrievals, making them the biggest single payoff in the pipeline.

It's also the least glamorous. Hardly anyone writes about noise removal, which is exactly why so many of the wins are there. The figures below come from a small internal eval and not from a rigorous benchmark, and they should be read that way.

Three kinds of chunk, three builders

The naive pipeline uses a single chunker: split the text and move on. Real documents contain at least three kinds of content that behave nothing alike. A paragraph of prose, a fee table and a screenshot each need their own treatment, and pushing all of them through one splitter is how half-tables and orphaned steps appear.

Text chunks follow structure

Splitting by character count is off the table. Instead every line is typed first, as a heading, a numbered step or a plain sentence, and the merge rules depend on those types. That keeps a numbered step from being glued to an unrelated paragraph. Oversized prose is cut at sentence ends, and slicing through the middle of a sentence is the fallback of last resort. Neighbouring chunks overlap slightly, so a procedure crossing a boundary stays understandable from either side.

Tables stay whole

Tables are built separately because their meaning is in their layout. The builder finds the header row, repeats it on each piece if the table must be divided, and tidies the rows. It also rejects tables that carry no real content, most often the table of contents, which parsers output as a neat grid of headings and page numbers.

The previous iteration of this system had a defect right here. Tables of contents were detected with a crude test, many rows with most of them finishing on a small number. Fee tables fit that description too, so an actual fee schedule could vanish from the index without any warning. The replacement test is narrow: in a real contents page, page numbers only climb, but fee amounts such as $35, $30 and $7 fall and rise. A table counts as an index only if the trailing numbers never go down. Fee schedules now stay, contents pages still go, and a single condition removed the bug.

Images are decided one at a time

Each image can cost money, so every one tries three paths in order and stops at the first that works.

1. The free path uses text the parser already extracted. If the document supplied a usable caption or annotation that isn't a placeholder like "image," that text is used and no vision model is called. In many documents, between a third and half of images take this route, which is pure savings.
2. The paid path calls a vision model. This applies when no usable text is available and the image is bigger than a tiny divider. A tightly worded prompt makes the model classify first. If the picture is a logo or ornament, it answers with the one word DECORATIVE and the chunk is thrown away. Otherwise it fills two fields. SCREEN is a single line saying what the screen is for, for example an outgoing wire entry screen for business accounts, written so it sits close to user questions in vector space. CONTENT lists every field label, button and code visible, in reading order, giving searchers the exact strings they might type. That way a screenshot can be found by purpose and also by its printed wording.
3. The fallback path uses the heading. If both fail but the section is known, a placeholder is stored that says what the image is and where it sits, so it remains reachable through its section at query time.

A small flaw sits in the size check. Tiny images skip the vision call based on a width-and-height threshold, but both dimensions have to be small, so a very thin and very wide divider line passes through and is still sent to the model. That wastes a little money on decoration without hurting correctness, and it's on the to-do list.

The diagram shows the three image paths tried in order. The vision model runs once at index time, and every query afterward is free.

Cleaning up what the parser got right

Faithfulness is exactly what makes parsers a nuisance here. They pull out all of a page: a CONFIDENTIAL stamp repeated across 200 pages, page numbers, copyright footers, a sentence cut in two by a page break. Each of these becomes a chunk, and every one is clutter. Clutter also distorts ranking, since a user query can just as easily hit a banner as a procedure.

So once chunks exist, they run through a cleanup pass in which each step targets a kind of junk actually seen in real documents. In order:

1. Boilerplate filtering removes bare page numbers, revision stamps and link lines like "contact us" or "read more" near the start. The rule set varies with the file type, because PDFs, Word files and slide decks each have their own kind of clutter.
2. Remove repeated artifacts. Text that shows up identically across many pages is furniture. "CONFIDENTIAL, INTERNAL USE ONLY" on fifteen pages gets counted and culled, whereas "See page 3" appearing twice on one page stays, since it doesn't span pages.
3. Repair broken breaks. A chunk that begins mid-sentence or with a dangling bullet was split in the wrong place. If it can join the previous chunk in the same section, they merge. If not, it's tagged as a continuation so its label carries that context.
4. Merge orphan fragments. A stranded 40-character chunk at a section boundary is absorbed by a neighbor, but only when both share exactly the same section and heading. Nothing merges across sections, and a short callout that really stands alone is left as it is.
5. Deduplication collapses a sentence repeated under one heading into a single chunk. If it appears under two headings, both copies are kept, because with the section path attached they carry different information. Images are exempt entirely, as two similar-looking screens are still two separate screenshots.

None of the rules is sophisticated. Each came from watching a real document defeat the simpler version, and together they explain why a query reaches a procedure and not a copyright line.

The retrieval label

The step with the biggest payoff, and the one most systems omit, comes next. Take a bare chunk like this:

Enter amount and press F4.

It carries its section path, Wire Transfer > Initiation > Amount Entry, which already helps. Before embedding, though, a language model also writes a short retrieval label of 18 to 22 words, stored with the chunk. For this chunk the label reads something like:

Procedure step: type the transfer amount, then press F4 to confirm while starting a wire transfer.

Consider the effect. Without a label, the bare instruction sits in embedding space beside every other step that says to type a value and press a key. With the label, it sits close to wire transfers and confirmations, which is vocabulary a real user would use. A reworded question like "how do I confirm the amount on a wire transfer" then finds this chunk. In the internal evaluation the labels removed about half of the retrieval misses. That was a small test on a single corpus, so treat it as a strong hint and not a universal constant.

The technique descends from Anthropic's Contextual Retrieval, which found that putting a chunk-specific context sentence in front of each chunk before indexing sharply reduces retrieval misses. Here it was tuned for content where exact identifiers matter. Transaction codes and function keys must be copied unchanged and never paraphrased, and each label starts with a role word like "Procedure step" or "Reference table", so chunks group with the right kinds of questions.

What the label writer gets to see

A label is only as good as the context handed to the model that writes it. The bare chunk isn't enough, since it's precisely what lost its meaning when it was cut out of the page. So each label request carries several things:

• the chunk text itself
• the section path and heading
• the column headers, when the chunk is a table
• a previous and a next chunk, to show what surrounds it
• a short summary of the whole document, for the big picture

The document summary comes from a separate call made before labeling starts. It samples chunks from the beginning, middle and end plus one table and one image, and returns two or three sentences naming the document type, its audience and the products and codes involved. Every label request then includes it. The phrase "enter the amount" should be labelled differently in a wire transfer manual than in a loan disclosure, and the summary is how the model learns which document it's reading.

Neighbors deserve a closer look, because "previous and next" doesn't mean the two chunks physically adjacent. The chunk next door may belong to an entirely different procedure. So neighbors are picked by relevance, in three tiers, taking the first that matches:

1. the nearest chunk in the same section path
2. otherwise, the nearest chunk in the same parent section
3. otherwise, the nearest chunk under the same heading

Candidates are scanned outward from the current chunk, and anything outside the right section is skipped. The model therefore sees the next chunk that truly belongs with this one. Each neighbor is cut to a few hundred characters, which is enough context without the cost of passing around full chunks.

For the F4 example, the model sees that the previous chunk in its section was about choosing a beneficiary and the next about confirming the transfer. That lets it produce a firm label about entering a wire transfer amount, not a hazy summary. Well-chosen neighbors keep labels precise.

The figure shows the label model reading its same-section neighbors and then writing one specific sentence, stored beside the chunk and feeding both search lanes.

Why this is affordable

Three properties make the step practical and not ruinous.

• One-time cost. The label is generated during indexing and saved, so queries never pay for it.
• Low unit cost. A small, quick model labels chunks in groups of 25, several groups running concurrently, and the expense comes to well under a cent per chunk.
• Graceful failure. A batch whose labelling errors out is simply indexed unlabelled while the run continues. Those chunks keep their text and section path and just miss the boost, so an outage hurts quality but never stops indexing.

The label lives in its own field, apart from the chunk body. That separation matters later, at query time, when the label gets extra weight in the keyword search. The point for now is that the costliest reasoning is done before anyone searches, and it's what makes paraphrased queries work.

Determinism and the versioned prompt

Label generation is deterministic: temperature zero and a fixed seed mean that re-indexing a document reproduces the same labels. This is what backs the claim that the cost is paid only once. Re-indexing never reshuffles the vectors, it just recreates them.

Determinism also depends on versioning the prompt. A seed only gives identical output for identical input, and the instruction text is part of that input. Edit the enrichment prompt between releases and every later re-index yields different labels, so the index silently divides into two generations and nothing reports it. The prompt is therefore saved with a version tag, loaded by that tag when indexing runs, and changed only through a tracked, intentional update. Reproducibility across releases has to be built, because a seed alone won't deliver it.

Embedding and storage

After cleaning, placement and labeling, the final index-time job is producing the vector. One embedding model covers every chunk type: text-embedding-3-large, cut down to 1536 dimensions, so text, tables and screenshots all rank against a query in the same space.

What goes into the embedder is a composed string in a fixed sequence: structural prefix, label, then body text. The prefix draws on facts already known about the chunk. These include a document type guessed from the section path, such as fee schedule or wire transfer procedure, along with the heading and section path, and for pictures the page and location. Tagging a procedure-like chunk as a procedure pulls it toward questions that start with how to do something.

The section path is also added to the text at this stage, which is where the older version had a bug. Back then the prefix was written into the chunk during cleanup, and any later edit could prepend it a second time and quietly spoil the vector. The current code removes any stored prefix, then builds a new one at embedding time. It exists in a single place and is added exactly once, just before the model reads it, which eliminated a whole family of silent double-prefix errors.

The finished vector is written to the store alongside the chunk body, label, structural metadata and a full-text column that keyword search reads from. Then the document turns active. If any pipeline step throws, the document is marked rejected and the reason is kept for debugging, while a previous active version stays searchable the whole time. A half-indexed document is never visible.

What a finished row looks like

To make this concrete, here's the "Enter amount and press F4" chunk as it lands in the store. Every field on it came from a step above.

text              : "Enter amount and press F4."
                     (built + cleaned + section-prefixed)
retrieval_context : "Procedure step entering the amount
                     and pressing F4 during wire transfer
                     initiation."          (the label)
chunk_type        : TEXT
section_path      : "Wire Transfer > Initiation >
                     Amount Entry"         (Part 2)
preceding_heading : "Amount Entry"
table_headers     : null                  (TABLE only)
page_start / end  : 12 / 12
image_object_key  : null                  (IMAGE only)
embedding_model   : text-embedding-3-large
embedding_dim     : 1536
embedding         : [ ...1536 floats... ] (body+label)
text_search_vector: label weighted high, body lower

On a table chunk, the column headers fill the spot where a text chunk has nulls. An image chunk stores a blob key, MIME type and page position, with the CONTENT dump in text and the SCREEN line in retrieval_context. All three flavors share one row layout. Retrieval relies on two columns: embedding for vector search and text_search_vector for keyword search, each derived from body plus label. The other columns locate the chunk, display it or limit it to the right tenant.

That is the entire output of index time: rows like this, one per chunk, cleaned, labelled and vectorized, waiting for the first query.