Context Rot Explained: Why Long Inputs Weaken LLM Accuracy

Feed a long prompt to any of the frontier models tested in 2025 and the answer quality slips, even when the thing you're asking about is sitting right there in the text. Chroma's 2025 study across 18 of them found accuracy on facts buried in the middle of the input dropping by more than 30%. The reason has little to do with some fuzzy "attention span". It comes from how transformers encode where a token sits.

People call this context rot, and it's a different animal from hitting the context limit. A 2023 Stanford experiment shows why. Give a model 20 retrieved documents, about 4,000 tokens, which is nothing next to modern limits. If the relevant fact is in the first position, the model gets it right 70-75% of the time. Move that same fact to the middle and accuracy falls to 55-60%. The information is still there and still correct. The model just gives it less weight.

Where the decay comes from

Most current transformers use Rotary Position Embeddings (RoPE) to mark token order. RoPE has a known long-term decay: the further apart two tokens are, the lower the ceiling on how much one can attend to the other (in theory) shrinks, whatever the two tokens actually mean. Softmax makes it worse. It pushes attention toward tokens that already score high, which favors the start and the end of the context and gives you the U-shaped curve researchers call "lost in the middle".

You can see the shape of the decay with a small script. It's built only from RoPE's frequency structure, not from any real model's weights, so read it as an illustration. The only import it needs is numpy:

import numpy as np

And here's the function that measures how the similarity bound shrinks as distance grows:

def rope_similarity_decay(distances, dim=64, base=10000):
    freqs = 1.0 / (base ** (np.arange(0, dim, 2) / dim))
    decays = []
    for d in distances:
        bound = np.mean(np.cos(freqs * d))
        decays.append(bound)
    return np.array(decays)

Run it over larger and larger token distances and you get this:

Token distance | Relative similarity bound
             1 |  0.9662
            10 |  0.6579
            50 |  0.4898
           100 |  0.5586
           500 |  0.2006
          1000 |  0.2789
          5000 | -0.1309
         10000 |  0.0494
         50000 | -0.0898

Tokens that are 1 to 10 positions apart keep a bound near 1.0. By a few thousand tokens of separation it has collapsed toward zero and even dips below it at points, so the geometry between far-apart tokens stops reliably saying "these two are related". Notice that it isn't a smooth slide either. There's a falling envelope with real oscillation inside it. None of this benchmarks a specific model, but it does explain why the problem is structural and doesn't just vanish when you train a bigger one.

A bigger window doesn't fix it

Researchers now separate the advertised context window from the Maximum Effective Context Window (MECW), the length at which a model's performance actually holds up. On complex tasks the effective ceiling can sit as much as 99% below the number on the spec sheet, according to 2025 research on that gap. A million-token window means the model won't error out before a million tokens. It says nothing about whether it uses all of them well.

Why coding agents get hit hardest

For coding agents, context rot has been singled out as a primary failure mode, and it's a garbage-in problem more than a capability gap. Over a long task the agent searches, explores and backtracks, and every dead end leaves noise in the history. That noise drags down each later step, even though the same model would solve the task cleanly from a fresh context. So a long session can fail without ever running out of tokens: the instructions from turn one are still in the conversation on turn fifty, but the model's attention may no longer reach them.

What helps

RAG and long windows aren't interchangeable. Retrieval earns its keep by cutting a large collection down to the relevant slice, because stuffing everything in at once runs straight into the positional decay above. Past retrieval, the fixes all say the same thing: keep the working context small and clean rather than big and complete.

• Summarize or throw away stale conversation history.
• Put whatever the decision hinges on near the start or the end of the prompt, never in the middle.
• Remember that "it fits" and "the model will use it" are two separate claims.

Quick answers

What is context rot? Output quality declining as input length grows, even when the window isn't full and the needed information is present.

Why is the middle worst? RoPE weakens the link between distant tokens by design, and softmax then concentrates attention on positions that already score high, which are usually the first and last ones.

Does a larger limit solve it? No. The usable context is often far smaller than the advertised one, especially on complex multi-step tasks.

Is it just running out of space? No. Accuracy measurably drops at a few thousand tokens, far below the hundreds of thousands that frontier models accept.

One caveat on the code above: it's a simplified frequency model. Actual behavior depends on trained attention weights sitting on top of this geometric baseline, so the baseline explains part of the effect and not all of it.