The KV Cache Explained: Why Long Context Is a Memory Bill, Not Smarts

Memory arithmetic for local models keeps running into a term called the KV cache. It's easy to calculate and surprisingly hard to explain. This piece is pure theory, with no hardware shopping and no file formats. By the end, the reader should know what the cache holds, why a model can't work without it, and why a bigger context window is really a request for memory instead of a request for intelligence.

Why the cache exists

A language model writes one token at a time, and to pick the next one it looks back over every earlier token. That look-back is called attention, and it's the costly step.

Now picture no cache at all. To produce token 5, the model processes tokens 1 to 4. For token 6, it processes 1 to 5. For token 7, 1 to 6. The same work gets repeated on the same tokens, and the waste grows with every added word.

A cache converts repeated work into stored work. The model still attends to every earlier token, but it doesn't rebuild them each time. Compute each token's contribution once, save it, and reuse it for the rest of the conversation. Two things get saved per token, which explains the name.

Queries, keys and values

A token starts as a list of numbers called an embedding, which you can think of as the token's meaning written as coordinates. The model multiplies that embedding by three separate learned matrices. Training fixed those matrices, and they don't change afterwards. Each one turns the same input into a different vector.

The three matrices belong to the model's weights and are identical for every token. Only the incoming embedding differs, so the three outgoing vectors differ for each token.

Think of a library. Each book carries a label on the spine, and the text sits inside. The spine label plays the key, the text plays the value, and the question a visitor arrives with plays the query. Nobody reads the whole shelf. Visitors scan spines, pick the labels that look relevant, and open only those. Attention does the same thing with one twist: it opens every book, then blends the contents, giving more weight to books whose labels fit the question better.

Keys are for matching and values are for content. That split explains why both must be kept. One decides how much a past token matters, and the other supplies what it says.

What gets thrown away and what stays

Only the newest token asks a question. Once its turn ends, the question is answered and never needed again, so queries are discarded.

Keys and values are another story. Token 5's key and value will be consulted when the model writes token 6, token 600 and token 60,000. They never change, either, because nothing that happens later alters what token 5 offered. Computing once and keeping forever is exactly the right trade.

Why the cache gets so big

Back to the library. Every arriving token writes two cards, a label and a contents card, and files them. Nothing is removed while the conversation stays open. Two facts make that filing job larger than it sounds.

First, models stack many layers, and each layer keeps a private library. Eighty layers means eighty buildings, and each files a card pair for every token. Second, each layer divides its work among several readers called heads, and every reader files its own pair, so the count multiplies once more.

On this model, one token of conversation costs 320 KiB. A thousand tokens cost 320 MiB. The cache is tiny per token and huge per conversation.

The whole thing as one formula

All of that compresses into a single line, and every term can be read from a model's config.json:

cache bytes = 2  ×  layers  ×  tokens  ×  kv_heads  ×  head_dim  ×  bytes_each
              │        │          │           │            │            │
              │        │          │           │            │            └ 2 at FP16, 1 at 8-bit
              │        │          │           │            └ numbers per head
              │        │          │           └ how many key/value sets per layer
              │        │          └ your context length
              │        └ every layer keeps its own
              └ one key and one value

Only one term is yours to pick. Layers, heads and dimensions are set at training time. Bytes per value follows from the cache precision. Tokens is the context length, and that's the dial you control.

Since tokens sits in a plain multiplication, growth is linear and merciless: double the context and the cache doubles. Longer conversations get no discount.

Two real models compared

Real numbers make this less abstract:

• Qwen3.6-27B maintains a growing cache in just 16 of its 64 layers, with 4 key/value sets in each. Cost: 64 MiB for every 1,000 tokens.
• Llama-3.1-70B maintains one in all 80 layers, with 8 key/value sets apiece. Cost: 320 MiB for every 1,000 tokens.

That's five times the cost per token, and architecture explains it rather than size. A model's parameter count won't tell you its cache bill.

One more wrinkle: Llama-3.1-70B runs 64 readers yet stores only 8 card sets. That mismatch is intentional. Groups of eight readers share a single set instead of each hoarding a private copy. The technique is grouped-query attention, and almost every current model uses it. It's like sixty-four people each buying their own newspaper versus eight copies circulating around an office. Everyone still reads, but the newsagent's bill drops eightfold.

The saving is eightfold for little loss in quality. Without it, long context on a 70B model would need more memory than any single machine holds.

Turning tokens into pages

Token counts are hard to visualise, so here are those windows in ordinary reading terms, using roughly three quarters of a word per token:

8K tokens  ≈   6,000 words  ≈   25 pages   a long report
32K tokens  ≈  25,000 words  ≈  100 pages   a short book
128K tokens  ≈  98,000 words  ≈  390 pages   a full novel
256K tokens  ≈ 197,000 words  ≈  790 pages   two novels

Asking for a 128K window means asking the model to hold a novel in its head and keep holding it. Put that on real hardware: Llama-3.1-70B with 8-bit weights on a machine with about 114 GiB usable.

context      weights      cache      total      left over
   8K        65.2 GiB     2.5 GiB    67.7 GiB    46.0 GiB
  32K        65.2 GiB    10.0 GiB    75.2 GiB    38.5 GiB
 128K        65.2 GiB    40.0 GiB   105.2 GiB     8.5 GiB

At 25 pages, the cache hardly registers. At novel length it grows to 40 GiB. For comparison, the entire Qwen3.6-27B, every weight included, occupies 25 GiB, so a single novel-length chat on the 70B model demands more memory than a good 27B needs in total.

The weights didn't move. The only change was how much conversation the model was asked to remember.

A long context window isn't a setting. It's a standing memory reservation, paid again for every conversation left open.

People often miss this part. Weights resemble a store's inventory, purchased once and browsed by every shopper alike. The cache resembles a changing room. Each shopper needs a separate one, held until they leave. Two readers each working through a novel need two 40 GiB caches. That's why a model that feels spacious on a personal machine buckles when several people use it at the same moment.

Four ways to shrink the bill

Here are four levers, in the order worth reaching for.

1. Cap the context to the job. This is the cheapest fix by far. A typical chat uses a few thousand tokens, and 32K covers the bulk of coding tasks. Requesting the maximum window for every task pays the maximum price every time.
2. Quantize the cache, with care. Storing each number in 8 bits rather than 16 halves it. The catch is that keys and values tolerate this unequally. Published measurements show 8-bit keys keeping about 98% agreement with the full-precision run, while 4-bit keys fall to around 12%. Values handle coarser treatment, so compress values first and keep keys at 8-bit or better.
3. Prefer architectures built for it. Grouped-query attention is everywhere now, and hybrid designs go further by giving most layers a small fixed-size summary instead of a growing stack of cards. That's how a model can advertise 256K and still fit.
4. Revisit the weights. Dropping weight precision releases memory for the cache. In the 128K row above, 4-bit weights turn 8.5 GiB of spare room into 37.8 GiB.

Choosing a model with the cache in mind

Run the numbers for three models side by side, all with 8-bit weights on the same machine, and the picture can shift in an unexpected direction.

The same cache budget buys sixteen times the context on the best of them. Parameter counts, compared the way people usually compare them, barely differ, yet cache cost differs by a factor of 16.

Qwen3.6-35B-A3B makes a good everyday model here. At 20 MiB per 1,000 tokens, context is almost free, and even the full 256K window needs only 5 GiB of cache. Rationing context stops being a concern, which is the reason to own a large machine at all. When a shorter task needs more reasoning depth, the dense 27B is the step up, at three times the per-token cost, which is still small.

Llama-3.1-70B is harder to justify at long context on a 114 GiB box. It isn't a weaker model. At 128K it takes 105 GiB, and at 256K it wants 145 GiB that don't exist. Holding its cache to a sane 20 GiB means capping context near 66K. If a model forces context rationing anyway, memory is better spent on one that doesn't.

A useful habit follows from this: look up the cache cost per 1,000 tokens before looking at parameter count. Parameters tell you the price of loading a model. The cache figure tells you the price of using it. They answer separate questions, and the model's name only advertises one.

The short version

The KV cache is a record of the conversation so far, written once per token so the model never has to recompute it. For each token, in each layer and for each key/value set, it stores a key, which decides how much that token matters, and a value, which records what the token contributes.

It's also the one slice of the memory bill that grows while the model is in use. Weights cost a fixed amount at load time, whereas the cache is a running charge that scales with tokens and with open conversations.

Next time a context length needs setting, treat it as a figure in gigabytes, not tokens. Across the models covered here, a 128K window lands anywhere between 8 GiB and 40 GiB depending on which one is chosen, so it's worth deciding deliberately.