A Bigger Context Window Will Not Fix Your Agent: Attention Budget and Caching

A provider can advertise a million-token context window, and the model will still use far less of it reliably. NVIDIA's RULER benchmark checks the length at which a model keeps performing well. For two of the seventeen models it tested, Yi-34B and GPT-4 (1106 preview), the reliable length came to roughly 32K of a claimed 200K tokens and 64K of a claimed 128K. The weakest models fell off at a tiny fraction of what the label promised. Past that point, the advertised size is a line on a pricing sheet and not a production capability.

You see the same underlying failure in several places. An agent forgets an architectural decision from the first turn. A RAG system retrieves the right document and still answers wrongly. An automatic session summary drops precisely the config line you needed. The usual reflex is to buy a bigger window, and for agents that fails, because capacity was never the problem. The problem is how the model spreads attention over everything already stuffed into that capacity.

The short version

The context window is not uniform memory. It is a resource with an attention budget that erodes well before the token limit. Material at the start or end of the context is used better than material in the middle, to the point where a model handed a document buried mid-prompt can answer worse than one handed nothing. A concrete case: an agent that receives a long tool log, a pile of retrieved chunks and its instructions all in one prompt will tend to honor whatever sits at the edges and quietly blur the rest.

Three tools manage that budget, and each loses something different:

• Prompt compression removes low-information tokens. It looks good on reference benchmarks, but independent reproductions show cracks on structured content such as numbering, exact strings and code fragments.
• Sliding windows and compaction control session length. The first throws away the oldest turns and the second swaps them for a summary, and their losses differ.
• Prompt caching can cut input cost by about 90 percent without losing information, provided the prefix is byte-identical between calls.

Here is the trap: compression and compaction both break the cache, and cache savings frequently exceed compression savings. Using the wrong tool for the wrong problem produces silent production failures that are painful to diagnose afterward.

Context is a budget, not storage

Two terms first. A token is the unit of text the model works on, around four characters or three-quarters of a word in English. The context window is the maximum number of tokens processed in one call, which is less everyday memory and more the largest size of the attention matrix in a single pass.

Andrej Karpathy has described context engineering as the craft of filling the window with exactly what the next step needs, with the model as the processor and the window as its working memory. That analogy is useful but covers only half of it. The other half is the attention budget. Anthropic's guidance on context engineering for agents treats context as a finite resource with diminishing returns, where each added token draws on a limited pool of attention. This is arithmetic and not poetry: attention pairs grow quadratically, so a thousand tokens imply about a million relationships, and a hundred thousand imply ten billion.

So adding context just in case is never free. Three extra few-shot examples, a full conversation history or twenty tool definitions in place of five all draw from the same pool as the facts that matter. The first question in an architecture review shouldn't be how many tokens are available. It should be which part of the budget is actually working toward the result.

The U-shaped curve

Liu and colleagues (Stanford and Berkeley, 2023) measured accuracy on multi-document question answering and found a U shape. Answers were most reliable when the relevant passage came first or last and least reliable when it sat in the middle. The label Lost in the Middle stuck quickly, well ahead of any agreed explanation.

What really unsettles is a side result. Once the prompt held twenty or thirty documents, a correct answer placed mid-context scored lower than the closed-book baseline, where the model saw no document whatsoever. Handing over the right evidence in the wrong slot can leave a system worse off than giving it nothing.

Why it happens: still open

Years later there is no consensus on the cause. Several explanations compete:

• an artifact of training data, since authors tend to put key points in openings and closings,
• how positional encoding and causal masking behave,
• plain decoder geometry that exists from initialization onward, before training starts,
• an adaptation that emerges during pretraining as the model balances competing memory demands.

Each cause calls for a different remedy: other training data, another attention mask, calibration of attention, or just a reordered prompt. Merging them into a single neat explanation would be misleading.

What is solved and what is not

Asking whether the problem is already solved frames it badly. Simple fact retrieval is largely solved: current frontier models can find one literally stated needle in a very long haystack wherever it sits. Multi-step reasoning is not solved. Synthesizing several documents or editing code across a large repository is a different task from locating a single fact, and published long-context coding evaluations show success rates falling sharply as input length grows for the same model and task type.

There is also evidence against the curve being universal. Chroma's Context Rot study tested eighteen frontier models and found that a shuffled, logically incoherent haystack beat a coherent one across all of them, and its experiments did not reproduce a consistent positional advantage either. The reasonable conclusion is that more than one degradation mechanism exists, and Lost in the Middle is only one.

One sentence that moved a score from 27% to 98%

Anthropic reported a telling case with Claude 2.1 on a needle-in-a-haystack test. Accuracy was 27 percent until a single line was added to the start of the model's reply, telling it to pick out the most relevant sentence from the context. After that the score was 98 percent. No capability was added; one instruction merely changed place. It was a single evaluation, one model version and one kind of task, so treat it as an illustration and not a law for all prompts.

The mechanism: attention sinks

Early tokens in a sequence soak up an outsized portion of attention, content or not. The cause is plain math. Softmax makes attention weights add up to one, so the model must put all of its attention somewhere even when nothing present deserves it. Abstaining is not an option, and the first positions become a dumping ground. These are called attention sinks.

A team at MIT (Xiao and colleagues) turned this quirk into a tool. Their finding was that a simple sliding window collapses at precisely the moment the first token is evicted from the cache. Pinning only the first handful of tokens as fixed sinks keeps generation steady across streams of millions of tokens, running as much as 22.2 times faster than recomputing each window. Popular summaries often skip one limit: it adds zero tokens of real memory. Output stays fluent while old tokens vanish, and any fact that slides out is lost for good. Stability and memory are different things.

The practical advice, then, is to place the key instruction where attention is strongest, usually at the end, and to measure on your own setup. It isn't universal: some research shows sinks appearing mid-sequence that are better suppressed than protected, and behavior varies by model.

Compression, window and compaction

Think of a suitcase that won't hold everything. You can drop less important items, pack only the newest or write a note about what stayed home. Each choice loses something different.

Prompt compression

The LLMLingua family scores each token's surprise with a small helper model and strips the low-information ones. The claims are impressive: up to 20x compression for LLMLingua, and for the query-aware LongLLMLingua an improvement of up to 21.4 percent on NaturalQuestions with a quarter of the tokens. Those are two variants and not a single result.

Independent reproductions complicate the picture. One issue filed against the reference repository reported the full uncompressed prompt clearly outperforming the compressed one, the opposite of the paper. A separate study of compression in the field reported fragment-counting accuracy collapsing, since compression destroys numbering and exact strings. The takeaway isn't that compression fails. Its reliability is contested and depends on data type. Narrative text tolerates it well, while numbered passages, identifiers, code and numbers do not.

Three ways to manage a session

• Sliding window. Plain first-in-first-out: keep the last k messages or N tokens and drop the rest. It costs nothing computationally and loses brutally, so a decision from turn one vanishes by turn five and the model may propose something that contradicts it.
• Compaction. Replace the older conversation with a summary. The risk is a summary so aggressive that it loses details whose importance appears later. Thresholds across tools range widely, but a common recommendation is to trigger compaction around 70 to 75 percent of capacity and not 95, because a model with no spare room writes a worse summary.
• External memory. In the MemGPT style, data lives outside the window and the model pulls it back through a function call when needed. Nothing is lost for good, but each recall costs latency and an additional model call, and the model has to know that a given fact is worth fetching in the first place, which is itself a place where mistakes creep in. Use it for facts that must be retrievable on demand, like user preferences or earlier decisions, and not for everything.

The trap: compression kills the cache discount

With prompt caching, the provider remembers the unchanging opening of your prompt and bills later reads of it at a small fraction of the usual rate. The KV cache underneath holds the computed state for earlier tokens so the model needn't redo that work.

The price of the discount is rigidity: the prefix must be byte-identical across calls. That rules out a variable timestamp at the top, a random session ID before the instructions and reshuffled keys in a JSON dict. Anything variable belongs in the volatile tail. Teams running prominent production agents have called the cache hit rate their single most important system metric, with about a tenfold cost gap between hit and miss.

Here the trap snaps shut. Compression rewrites the prompt on each call, since the tokens it drops depend on the question at hand, so every request is a cache miss. A saving of about sixty percent on tokens runs into the loss of a ninety percent discount, and it loses that contest more often than teams assume.

One anecdote circulates in marketing material about a monthly bill falling from about $42,000 to $2,100 after deploying compression, with no company, date or method given. It is impossible to tell whether that comparison counted the lost cache, which is exactly the variable that decides whether such a result repeats.

A plan for Monday

This list doesn't replace testing on your own data. It prevents the most common mistake, which is reaching for compression or a bigger window before checking cheaper, safer options.

1. Classify the task. Simple fact retrieval or multi-step reasoning over long context? That sets how serious positional degradation really is.
2. Check where the key instruction sits. If it is buried mid-prompt, move it to the end and measure. It's a one-sentence change and the only free lever here.
3. Verify prefix stability. When the system prompt differs even by a byte between calls, caching does nothing, so repairing that comes first. It gets raised in reviews far less than model choice, though it can move the bill more.
4. Name the problem before choosing a tool. Cost trouble means checking the cache before compression. Quality that degrades over a long session suggests compaction at a 70 to 75 percent threshold. Forgotten specific facts point to a sliding window or external memory, depending on whether the facts must return on demand.
5. If you still use compression, test it on your own data, and specifically on data with exact numbers, identifiers and code, where published results most often fail to replicate.

What to take away

A bigger window doesn't fix an agent because it enlarges the capacity of a resource whose problem was never capacity.

• The window is an attention budget and not storage, and every token added just in case draws from the pool that serves the important ones.
• The Lost in the Middle debate is framed wrongly: fact retrieval is largely handled on frontier models, while multi-step reasoning over long context still degrades noticeably.
• Position is the cheapest lever you have, so check it before spending on anything pricier.
• Compression has documented weaknesses in independent tests, especially for content that must survive verbatim.
• Caching and compression cancel each other out economically, so calculate both savings first.

The real shift in everyday work is the order of questions in a design review. Ask first what the context contains and where each item sits, and only afterward how large it is. Reverse the order and you pay for a bigger window that simply gives important facts a wider middle in which to disappear. A short rule of thumb to pin up: inspect position, then prefix stability, then volume, and reach for compression last.