RAG Distractors: Why Raising k Buys Competitors, Not Answers

Imagine a retriever that returns ten chunks for a question. Exactly one holds the answer. Think for a moment about what the other nine are. The ranking itself dictates their nature: out of the entire corpus, they are the passages closest to the question that still fail to answer it.

Nothing is broken here. The retriever did what it was specified to do. Those nine chunks aren't noise that slipped through, they're its product.

The figures below come from published measurements by other people, and each one is attributed. The argument connecting them is a claim about production RAG: raising k mostly doesn't buy answers. It buys competitors for the answer, and those are the most expensive tokens that can go into a context window.

A name for what gets shipped

Chroma's Context Rot report, published on 14 July 2025, draws a distinction much retrieval writing ignores. Irrelevant content has nothing to do with the question. A distractor is related to the topic but doesn't answer it.

The two behave very differently. Across eighteen models, even a single distractor lowered accuracy compared with a needle-only baseline. Four distractors made the damage compound, and the effect grew as inputs got longer. In one haystack and needle pairing, two of the four hand-written distractors showed up in hallucinated answers far more often than the other two. Model families split as well: Claude models tended to abstain when things were ambiguous, while GPT models tended to answer with confidence and be wrong.

Now compare that definition with a ranked list of ten results. Ranks two through ten fit it exactly, and they're sorted so that the most convincing ones sit closest to the top.

One consequence follows: better embeddings make this worse. An improved retriever's whole purpose is moving passages nearer to the question. For rank one, that's the goal. For the other nine, it means each near miss is sharper than anything last year's model would have surfaced. The spend goes into distractor quality, and gets labeled as recall.

The common fix repeats the mistake one layer higher. A cross-encoder reranker judges relevance more carefully than the first pass, yet relevance is still what it scores, and a passage can sit squarely on topic while containing no answer whatsoever. Reshuffling nine near misses doesn't remove them. Only a different question does: does the passage answer, as opposed to does it match?

The hidden half of the cost

A second charge never shows up on the invoice.

With an unfocused window, a single call has to do two jobs: find the relevant passage, then reason over it. Retrieval has quietly moved inside the model, where it can't be logged, tuned or watched while it fails, and where it fights for the same attention as the reasoning that was actually paid for.

Chroma put a price on the gap. In their LongMemEval runs, each question went in twice. One version used the full input, averaging 113,000 tokens. The other used a focused version averaging 300 tokens, holding just the relevant passage. Every model family did worse on the full input. Claude models showed the widest gap, mostly because they refused to answer once things turned ambiguous.

Identical question, identical model, answer present in both windows. What surrounded the answer was the one thing that changed.

A bigger window won't rescue this

Anthropic published a post on context engineering in September 2025 that gives the architectural cause. In a transformer, each token looks at all the others, which creates n² pairwise links for n tokens, and the model's hold on those links weakens as n rises. What's left is a limited attention budget, so extra context pays back less and less.

That decay is a slope and not a cliff, which is exactly why people miss it. Nothing throws an error. Precision just leaks away.

Chroma's data points to where the loss is steepest. All eighteen models did better on text that had been shuffled into nonsense than on text in its natural order. Another pattern: the less the phrasing of a question resembled the phrasing of the passage that answered it, the faster accuracy fell as inputs lengthened, although the same pairs worked fine on short inputs.

The practical reading is uncomfortable. A neatly organized corpus isn't helping, and the users hit hardest by long windows are those who ask in their own vocabulary and not in the documentation's.

Four kinds of tokens and how each one fails

Kind	Examples	Failure mode
Static	System prompt, rule files, tool definitions	Sent on every turn, so the price scales with the number of turns. It dilutes.
Retrieved	Chunks fetched for each query	Billed per fetch. It brings in near misses.
Accumulated	Chat history, tool outputs	Grows in one direction. Breaks late, often several turns after the poisoning event.
Externalised	Notes, files, summaries from sub-agents	Inexpensive to keep. Goes stale and gives no warning.

Vector database docs frame the retrieved row as a recall issue and never cost out the other three. Teams tune retrieval intensely, while the preamble keeps multiplying unnoticed behind it.

Replaying with focused context

A single experiment can tell a context defect from a model defect, and it takes no new infrastructure.

Pull thirty failing cases from production. For each one, assemble by hand the smallest context that contains the answer and nothing else. Then run the same question against both versions. The normal path looks like this:

chunks = index.query(question, k=10)   # one may answer, the rest merely resemble
prompt = SYSTEM + "\n\n".join(c.text for c in chunks) + question

And the comparison loop looks like this:

for case in failing_cases:
    full = assemble_context(case.question)
    focused = case.gold_passage
    log(case.id, ask(full), ask(focused))  # the gap is your context tax

Compare the two columns. If the focused version passes where the full one fails, no model upgrade will help.

After that, instrument the ledger. Log tokens per turn in four buckets: static, retrieved, tool results and history. Then look at the split on the turn where quality collapses. Very few teams have ever seen that number for their own agent, and the static share usually surprises them most.

Arithmetic for one support session

Chroma's figures are the baseline. From here on, the numbers are simple calculation.

Consider a support agent with a 6,000-token preamble, a median session of twelve turns, and top-8 retrieval at around 700 tokens per chunk.

Across the session, the preamble accounts for 72,000 billed tokens carrying information that never varies. Retrieval contributes 5,600 each turn, so 67,200 in total. Of that, seven out of eight chunks were picked for looking like the question and not for answering it. The sum is about 59,000 tokens of manufactured distraction, repeated for each session and each user.

Compare it with a focused prompt of roughly 300 tokens, which scored better than the 113,000-token input on the very same questions. Spend is only half of the damage. Those tokens also push accuracy down.

Changes to make first

Treat every token like a seat that needs a licence.

Static context must justify its place by changing behaviour on every turn it sits in, and it sits in all of them. Something that governs one task in fifty belongs in a skill that loads when triggered, and not in the always-on preamble.

Retrieved context must justify its place by being able to answer. So rank by whether a chunk bears the answer, not by similarity, and keep reducing k until the replay shows no more gains. A cheap method is a second pass that shows each candidate to a small model with one yes-or-no question, whether the passage contains the answer, and discards every no. That typically removes most of the fetch.

Each token is billed on every turn and draws on the same attention budget. Asking whether a document is relevant is therefore the wrong test. What matters is whether it outweighs whatever it displaces.