Contextual Retrieval for RAG: What the 67% Figure Hides, What It Costs and When It Fails

"The company's revenue grew by 3% over the previous quarter." A vector database can hold thousands of sentences like that one, and every one of them is functionally useless until it is clear which company is meant and which quarter.

The fault does not lie with the model that writes the final answer. It lies with a chunk that entered the index without an identity. Any RAG pipeline built on filings, contracts, technical documentation or source code is exposed, since those corpora repeat the same linguistic pattern in dozens of places and give the retriever no means of telling the repetitions apart.

The cure is cheap. Its price is counted in dollars per million tokens, and it takes a single generated sentence, written once while the index is built. In Anthropic's tests such a prefix reduced failed retrievals by up to 67%. That headline number hides some structure, and the use case chosen to showcase it has since attracted a counterexample. There are also four settings in which the spend is hard to justify.

The core idea in brief

When an index splits documents into chunks of a few hundred tokens, some chunks lose their subject and their timeframe. "Revenue grew by 3%" names no company and no quarter, and no embedding model can restore that, because the information is not in the text.

Contextual Retrieval handles the problem once, at indexing time. A language model is shown the entire document together with one chunk, and it writes a short situating prefix of fifty to a hundred tokens that says what the chunk is, whom it concerns and when.

A second step gets skipped in many summaries, and it is the important one. The enriched text is fed to the vector index and to the BM25 index, the lexical ranker that predates embeddings and matches exact words and not meaning. One intervention therefore repairs both halves of the search stack, at a one-time indexing cost and not a cost added to every query.

In Anthropic's evaluation, the embedding prefix alone cut failed retrievals by 35%. The full stack, with BM25 and reranking, cut them by 67%. Prompt caching is what keeps it cheap, because the whole document is read once and not from scratch for every chunk.

The technique is not universal, and that is where it gets interesting. First, the mechanism and the prompt itself.

A prefix that gives a chunk its identity

The intuition comes before the mechanics. A chunk cut from a document is a page torn out of a binder, with no header, case number or date. Read in isolation, it would leave a person unable to say which company's revenue grew, or when. An embedding model faces the same blank, except that it has no way to flag it.

The mechanism restores the missing header. At indexing time the model receives two inputs, the full source document and one chunk from it, and has a single job: write a short context that places that chunk within the whole.

Anthropic's prompt fits in a few lines:

<document>
{{WHOLE_DOCUMENT}}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{{CHUNK_CONTENT}}
</chunk>
Please give a short succinct context to situate this chunk within the
overall document for the purposes of improving search retrieval of the
chunk. Answer only with the succinct context and nothing else.

Applied to the example chunk, the output could look like: "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023; the previous quarter's revenue was $314 million." The original 3% growth sentence comes right after. Roughly a hundred tokens convert a stray fragment into a record with a key.

The enriched text is then written to two indexes. One is the embedding index (Contextual Embeddings), the other the keyword index (Contextual BM25). With ACME spelled out inside each chunk, BM25 can match the company on any page of the report, where before only the cover mentioned it.

Nor was this the first idea tried. Anthropic reported earlier experiments with generic document summaries attached to chunks, which gave "very limited gains," and with HyDE and summary-based indexing, which were not very effective either (Anthropic, September 2024). The prefix is what remained.

What separates a summary from a prefix is mechanical. One summary is shared by all chunks of a document, so it cannot tell chunk A from chunk B inside the same annual report, whereas each situating prefix is written for its own position in the document.

Corpora made of filings, contracts, technical documentation or a codebase, where one linguistic pattern recurs in many places, are exactly where chunks get lost today. The embedding prefix alone gives a 35% error reduction, and that is only the first tier.

A staircase inside one number

"67%" often gets quoted as the effect of a single technique. The published table says something different, and it reads better as a staircase than as one jump.

Results are measured as one minus recall@20: the proportion of queries whose correct chunk missed the top twenty. Anthropic calls this the fail rate and reports it over codebases, fiction, arXiv papers and scientific publications.

Four configurations build on each other:

• Baseline, with plain embeddings over raw chunks: 5.7% of queries fail.
• With Contextual Embeddings only: 3.7%, a 35% reduction attributable to the prefix by itself.
• With Contextual BM25 added: 2.9%, a cumulative 49% reduction, with the keyword index contributing the extra gain.
• With reranking added: 1.9%, which is where the 67% headline comes from, thanks to the final tier.

A reranker is a second model that reads the query and a candidate chunk jointly. It is slower than the first pass but much more exact. Here it reduced 150 candidates to twenty. All four figures describe the strongest embedding setup in the test, Gemini Text 004, and are not an average across the embedding models Anthropic tried.

The breakdown changes the decision and not just the story. A team that already runs hybrid search and reranking in production should expect contextualization alone to land nearer thirty-five percent than sixty-seven, since the other two tiers are already paid for. Reading the number as one effect is an easy mistake, and a good share of the reduction comes from standard retrieval components doing their ordinary job.

A Hacker News thread adds a sharp caveat: no one published how plain BM25 performs, with no contextualization at all, in that same test. Without that column, nothing settles how much of the jump to 49% is due to context in the lexical index and how much to plain hybrid search.

A trap when quoting the figures

Anyone quoting these results can also fall into a calculation trap. Anthropic's cookbook includes a smaller dataset of nine codebases and 248 queries, scored by another measure: Pass@20 moves from 90% for the baseline to 97% for the full stack. Different corpus, different metric, so mixing those figures with the earlier ones is simply an error.

The reduction is not free. It costs money at indexing time, but one mechanism keeps that cost nearer a dollar than a thousand.

Why a dollar and not a thousand

Consider the naive approach. An 8,000-token document is cut into ten chunks. Without any optimization each call re-reads the entire document, so identical text is processed ten times to describe ten different paragraphs.

That cost profile is what blocked the idea for years. People dismissed it as brute force and vetoed it in budgeting, long before the quality benefit was ever measured.

Prompt caching alters the math in one place. The document goes out once as a fixed prefix. The first request pays a cache write at about 1.25 times the base price, while each following request on that document pays a cache read at about a tenth of the base price.

Assuming typical chunk and document sizes, contextualization costs a one-off $1.02 per million tokens of source text. The number is stated along with its inputs, 800-token chunks and 8,000-token documents, which cost figures seldom are. That makes it possible to recompute with local numbers rather than trust it blindly.

In a cookbook example covering 737 chunks, close to 62% of input tokens came from the cache. The total fell from roughly $9.20 to roughly $2.85. A better model was not responsible for the two-thirds saving. Reading the document only once was.

The ordering detail that decides everything

A practical detail determines whether any of this materializes. Caching works on prefix matches, so all chunks of one document need to be processed back to back to reuse a single cache entry. Parallelism across documents is fine, one worker per document, but mixing chunks from several documents in a single queue is not. A typical naive pipeline distributes chunks over a shared thread pool, which hits the cache only now and then and pays the full rate.

The figure also deserves a warning. $1.02 reflects 2024 model pricing and is no constant of nature. The next generation of that model charges four times as much for input, so repeating the number today without recomputing quotes an outdated price.

The economics look excellent for SEC filings, the very document type used as the flagship illustration. Yet someone tested that very document type and saw the reverse.

A flagship example that fails its own test

Here begins the part most summaries skip.

The illustration used a chunk from an SEC filing, but the list of measured domains leaves filings out. It names codebases, fiction, arXiv and scientific publications, so financial documents serve as a narrative example and not a tested corpus.

Others filled that gap, and they found the reverse. Snowflake's engineers (Tarakad, Yu, Samdani, Merrick, March 2025) used about 23,000 genuine 10-K and 10-Q filings as a corpus and judged their pipeline on roughly 500 hand-written queries.

Enriching each chunk with a heavy, unique, model-written summary, which the team itself called contextual retrieval, lowered scores by 5.8 points for one model and 0.4 for the other. The outcome was a loss, with considerably higher compute spend.

When cheap metadata beats an expensive model

The more interesting finding is what worked in the same experiment. Injecting document-level metadata, namely company name, filing date and form type, raised answer accuracy from around 50 to 60% into the 72 to 75% range, with no language-model call at all.

The contrast carries the lesson. The problem was genuine, since the chunk really did not know its company. Yet the cheapest remedy outdid the costliest, because the company name stays constant through a whole filing and attaching it needs no model.

The comparison is not a clean replication, and that should be stated openly. Snowflake used its own prompt, embedding model, chunk size and evaluation, so the results cannot be read side by side as if from one instrument. They do involve the same type of document, which is why they matter.

Two further results moderate the excitement. A 2026 benchmark based on T²-RAGBench, separate from the Snowflake study, tested contextualization on financial documents with tables. It found recall@5 gains of 2.8 percentage points with dense retrieval and 2.2 with hybrid retrieval (arXiv). The gain is genuine and repeatable, and it is nowhere close to 67%.

That same benchmark, which spans over 23,000 queries across 7,318 documents, also contains a finding that should worry anyone dealing with reports: most of the errors left over are table-structure mismatches. A text prefix cannot help there, because it accurately labels a chunk that remains unreadable.

The second point is about compute. ConTEB (2025) pitted the Anthropic-style method against an embedding model trained for the purpose. The trained model scored marginally higher on average and processed each document roughly a hundred times faster (arXiv). Quality differences are small, as the prefix approach wins outright on some datasets, such as an insurance corpus, and loses narrowly on others. What the trained model really offers is lower compute cost.

A rival design also exists. Late Chunking (Jina AI, September 2024) addresses the same problem at a lower layer by embedding the entire document and cutting it into pieces only after the transformer, without calling a language model. Its structural weakness is that nothing is added to the keyword index, which forfeits the half of the benefit that Contextual BM25 provides.

For most corporate corpora, that asymmetry, more than the metric gap, probably decides the choice. That is a judgment call, and the cited ConTEB benchmark does not test it directly, since BM25 is not among the components it evaluates.

Once the showcase example turns out to have a counterexample, the useful question becomes where the technique fails to pay, and how a given corpus can be checked in five minutes.

Five minutes of testing before any spending

Start with a test and not an implementation. Take ten random chunks from the index and read each one in isolation. If it is possible to say what every one is about and where it came from, the chunks already carry context, and a prefix would add tokens and not information.

The test needs no code and takes five minutes. It is also the only evidence that concerns the actual data and not somebody else's.

Four documented situations where the technique does not help:

• A corpus below two hundred thousand tokens. Anthropic's own advice at that scale is to put the whole knowledge base into the prompt and skip retrieval entirely. Contextualization is meant for when the corpus no longer fits in the context window.
• Chunks that already stand alone. FAQ pairs, product cards and dictionary entries state their own subject, so a prefix merely repeats it.
• A corpus that changes often. The cost is one-time only if documents are stable. Documents revised weekly are paid for again every week.
• Tables and financial reports. If table structure is what breaks, rather than a missing company name, the chunk gets described accurately and stays unreadable.

The framing fits in three lines. A small, stable corpus suits long context with caching. A large corpus with ambiguous chunks suits Contextual Retrieval. Short, self-contained entries suit plain hybrid search, with no extra charge.

The order of adoption matters, and it is the point that most often leads to disappointment. Contextualization multiplies the value of hybrid search and does not replace it. A team without BM25 and reranking should add those first, Skipping that order means purchasing the third tier while the first is still missing.

Work out the cost ahead of deployment, not afterward. The formula is published, chunk and document sizes are known, and the one real unknown is whether the pipeline actually hits the cache.

What to carry into the design review

Return to the opening sentence. "Revenue grew by 3%" was never wrong. It was subjectless. The fix was neither a smarter model nor a better embedding, only one header appended once, at indexing time.

• However strong the answer-generating model, a chunk without context is useless. The cause is indexing, not hallucination.
• Three interventions stack up to the 67% headline, not one. Prefix alone is close to thirty-five percent, and hybrid search with reranking supplies the remainder.
• Prompt caching made the technique affordable, not a stronger model, and the saving vanishes without document-by-document processing.
• The document type used in the illustration has a documented counterexample: on SEC filings, inexpensive document metadata outperformed an expensive per-chunk model call.
• A five-minute check of a real index outweighs any headline. Read ten chunks separately.

In practice that comes down to three moves, in order. Run the ten-chunk test, check whether the corpus matches one of the four negative cases, and only then work out the indexing cost with the published formula. Only the third step needs a budget, and the first two decide whether it is worth requesting one.

One caveat cannot be avoided. Anthropic published the results breakdown in an appendix, as a static PDF and not as a raw dataset, and independent teams that tried to reproduce the numbers flag this as a reproducibility problem. First-hand evaluations on corpora in languages other than English are also scarce. Until both gaps close, every figure here comes from a specific setup and not from a law of physics, and what is missing most today are production measurements on non-English corpora.