Context Engineering for Long-Running Agents: Vector Database, KV Cache, Claude Code

Plenty has been written about what context engineering is. Far less addresses the question a builder faces in practice: how should the context pipeline be designed for an agent that runs for 200 steps? A million-token window sounds generous, yet a 2025 Chroma study of 18 frontier models reportedly found quality starting to slip once the window is 50 to 70 percent full. What follows is a blueprint for dealing with that, aimed at people shipping production agents. It covers why long contexts decay, a memory model borrowed from computer architecture, four patterns for managing context, the economics of the KV cache, how three well-known products differ, and a decision path with metrics.

An agent with no memory design has amnesia

Picture an agent that carries out steps 1 to 40 correctly. At step 41 it seems to forget the goal and starts repeating decisions it already made. That is rarely a logic bug. It's context rot, the measurable drop in answer quality as input length grows. The more the model receives, the more it effectively loses.

Reported research from Zylos AI in 2026 points the same way: doubling a task's duration is said to quadruple the chance of failure, and agents appear to degrade after roughly 35 minutes of equivalent task time. The Chroma result mentioned above found that every one of the 18 models degraded as context grew. The NoLiMa benchmark (ICML 2025) was starker still. At just 32K tokens, 11 of 13 models reportedly fell below half of their baseline performance even though they advertised 128K or more, and GPT-4o went from 99.3% to 69.7%.

Why the decay is built in

Transformer attention compares every token with every other token. Doubling the text therefore costs about four times the computation, not two. At 200K tokens that is on the order of 40 billion attention pairs per layer. Dumping everything into the context simply doesn't scale for an agent working through hundreds of steps.

A second effect is "lost in the middle". Models handle the start and end of a long context best and lose precision in the middle, so a key fact buried there can be close to invisible. Oddly, text that is coherent and well organised but irrelevant hurts accuracy more than the same sentences shuffled at random. Semantic similarity inside the noise makes the real signal harder to find.

The advertised window size is also misleading. A model that accepts a million tokens does not offer a million tokens of useful capacity.

Context anxiety

Users observed, and Anthropic acknowledged, one more quirk: Claude Sonnet 4.5 tended to wrap up tasks too early as it neared the context limit. This has been called context anxiety. The model behaves as if it is running out of RAM and backs off before its resources are actually spent.

Since degradation can't be avoided, the agent's architecture has to manage it. Systems engineering has a long-standing analogy that helps here.

The model is the CPU, the context is RAM, the vector database is disk

In mid-2025 Andrej Karpathy endorsed "context engineering" over "prompt engineering", describing it as the delicate art and science of filling the window with just the right information for the next step. He wasn't coining a buzzword. He was pointing at a change of viewpoint: the LLM is the CPU, the context window is RAM, and the engineer building the agent plays the part of the operating system.

The analogy carries into practice quite naturally. Compaction resembles garbage collection, state injection resembles a boot sequence, sub-agents resemble process isolation, and a file like CLAUDE.md is the configuration loaded at startup. What changes is the focus: less effort goes into how the prompt is worded, more into what gets loaded into memory and when.

A memory hierarchy for agents

An agent has a hierarchy just as a computer does, and each layer trades speed against capacity:

• In-context working memory. No added latency, 200K to 1M tokens. It matches registers and L1 cache: the fastest, the priciest and the most constrained.
• KV cache. Under 10 ms. Comparable to L2 or L3 cache, it holds attention matrices precomputed for earlier tokens and kept between requests.
• Vector database or semantic search. Roughly 5 to 50 ms and millions of entries, the counterpart of RAM. Figures quoted include about 0.34 ms per query for FAISS running locally and 7 ms p99 for Pinecone.
• Filesystem or notes. Above 100 ms with no real size limit, like a disk.
• Archival storage. Seconds to read, comparable to network or backup storage.

Each decision about where a piece of information lives is a trade between cost, latency and capacity. Pick the wrong layer and the result is either a big bill or an agent that forgets. The MemGPT work from 2023, widely cited, formalised this idea under the name "virtual context management", modelled on how operating systems manage tiered memory. It continues today as the Letta framework.

If the context is the agent's RAM, which patterns manage it in a real system, and how do they fit together?

Four patterns: write, select, compress, isolate

Context engineering rests on four basic patterns. They aren't alternatives to choose between. They're complementary layers of one composite design, and every production agent ends up using all four.

Write: put state outside the window

Writing means externalising state. The agent keeps notes in a file such as todo.md, NOTES.md or progress.md and reads them back in at later steps. Manus is known for "todo list recitation": the agent keeps rewriting its task list at the very end of the context so the model's attention is pulled toward the overall plan. That is deliberate attention steering, and it counters the lost-in-the-middle effect.

Claude Opus 4 in a Pokémon-playing experiment reportedly kept accurate notes over thousands of game steps, covering character levels, maps and strategies, with no explicit instruction about how to structure its memory.

Select: fetch just in time

Selecting means retrieving context when it's needed rather than preloading it. CLAUDE.md acts as a static preload, much like an onboarding document handed to a new hire on day one. After that, the agent explores the codebase with grep and glob and pulls in only what the current step needs.

Claude Code reportedly tried vector databases and dropped them because code drifts out of sync and permissions get complicated. Cursor went the other way and bet on dynamic discovery through semantic indexing. An A/B test there reportedly showed a 46.9% cut in tokens when the agent loaded MCP tools on demand.

Compress: shrink the volume

Compressing means cutting volume by summarising or truncating. Compaction asks an LLM to summarise the conversation so far, freeing window space while keeping what matters.

Claude Code reportedly warns at about 80% capacity and compacts automatically near 95%, keeping architectural decisions. Compaction that is too aggressive discards subtle but vital context, and the agent starts re-exploring ground it already covered. In practice, 3 to 6 times compression seems to be the sweet spot for production work. Academic benchmarks report much higher ratios (LLMLingua cites 20 times with only a 1.5-point exact-match loss on GSM8K), but production use demands higher fidelity.

Isolate: separate through sub-agents

Isolating means keeping contexts apart. A specialised child agent works in its own context window and hands a condensed result back to its parent. It may explore 10K tokens or more and return a summary of only 1K to 2K, far less than a shared context would accumulate. Manus applies a principle borrowed from Go: don't communicate by sharing memory, share memory by communicating.

The point isn't which pattern to pick but how to compose them and when each should fire. Writing plus selecting is the first line of defence, and compressing plus isolating is how a system scales.

One part of context management has an outsized effect on production costs and is often ignored at design time.

KV cache hit rate, the number that matters

The KV cache stores the attention matrices already computed for earlier tokens, so the model doesn't have to recompute them for every new token. Think of a database that precomputes joins for frequent query prefixes. In one GPT-2 measurement, generating 200 tokens took 11.9 seconds with the cache and 56.2 without, around a 4.7 times speedup.

Why it changes agent economics

A typical production agent has an input-to-output ratio near 100 to 1, compared with about 1 to 1 for a chatbot. Prefill, the recomputation of the entire input, dominates each call. Prompt caching is a provider-side mechanism that stores computed tokens for identical prefixes between requests, and it makes reads about ten times cheaper. For Claude Sonnet the quoted prices are $0.30 per million cached tokens against $3.00 uncached.

So reshaping a context design around the KV cache can plausibly separate an $8,000 monthly API bill from an $800 one, which lines up with Anthropic's tenfold caching discount. Yichao "Peak" Ji, a Manus co-founder, has said that, for an agent in production, nothing matters more than how often the KV cache is hit.

Three rules for keeping the cache warm

• A stable prefix. A one-token difference is a cache miss. A mistake Manus calls out is putting a timestamp at the top of the system prompt: it lets the model tell the time and wrecks the hit rate.
• Append-only context. Only add to the context, never edit it retroactively. History stays immutable, because any change causes a miss. JSON serialisation also has to be deterministic, since key order counts.
• Explicit cache breakpoints. Mark points in the context that tell the provider where to cache the prefix.

There's also a scale trap sometimes called the "MCP tax". Suppose 10 to 15 tools each carry 10K to 15K tokens of definitions. That's 100K to 225K tokens of overhead before the user has typed anything. Across 1,000 developers with five sessions a day, a rough estimate puts it near $1.2 million a year spent only on tool definitions. The remedy is progressive disclosure: load tools just in time instead of all at once.

Cache hit rates and four patterns are theory. What do they look like in systems that have already faced the problem in production?

Claude Code and two other products, three philosophies

Claude Code, Cursor and Manus give three coherent and different answers to the same problem. Each follows from its own product context rather than from ignorance of the alternatives.

Claude Code: static preload plus just-in-time reads

Claude Code, from Anthropic, loads CLAUDE.md at session start as an onboarding document and then navigates the codebase with glob and grep, with no vector database. It compacts automatically near 95% and preserves architectural decisions. Combining a memory tool with context editing reportedly gave a 39% improvement over the baseline.

Context anxiety with Sonnet 4.5 meant the model needed external context resets. Anthropic reportedly addressed this in later model versions without altering the tool's architecture.

Cursor: two tiers and dynamic discovery

Cursor (made by Anysphere) splits "intent context", meaning what the user is asking for, from "state context", meaning the live picture of files, edits and errors. Out of the box it reads just the first 250 lines of any file, a conscious choice that saves tokens and money.

A notable change came in January 2026. Cursor said that as models got better as agents, giving fewer details up front and letting the agent pull relevant context itself worked better. The A/B test mentioned earlier showed a 46.9% token reduction from on-demand loading.

Manus: gradient descent by trial and error

Manus describes its approach as "stochastic gradient descent", with the filesystem acting as the ultimate context. It has reportedly rebuilt its framework five times since March 2025. Ji's summary: context engineering turned out to be anything but straightforward, and is an experimental science.

Six principles are attributed to the team:

• Put the KV cache first.
• Mask tools instead of removing them, to preserve the cache.
• Treat the filesystem as unlimited external memory.
• Recite the todo list to steer attention toward the global plan.
• Keep errors in the context, since erasing a failure removes evidence.
• Vary serialisation templates to avoid the few-shot trap.

What the comparison teaches

No single philosophy is correct. Claude Code is tuned for long coding sessions that need stability. Cursor targets short assisted sessions that react dynamically to editor context. Manus is built for fully autonomous web agents making 50 or more tool calls per task. Five rebuilds at Manus show something basic: context engineering is an empirical discipline, not a blueprint that gets assembled once and shipped. Every production system will go through its own iterations.

It's easy to assume that growing windows will solve the problem, since a million tokens should be plenty. The NoLiMa and Chroma data correct that assumption quickly. Claude 4 Sonnet shows noticeably better resistance to context rot than older models, but that's an exception that proves the rule, not evidence the problem is gone.

Those systems and patterns cover what works. What is left is turning them into concrete design choices for a given agent, and finding out whether the context design is doing its job.

A blueprint: what to use, and what to measure

A paper from March 2026 on corporate multi-agent architecture (arXiv:2603.09619, about moving from prompts to context engineering) puts forward five criteria for context quality, and frames the context as the agent's operating system:

• Relevance. Is the information pertinent to the current decision step? Irrelevant context feeds rot directly.
• Sufficiency. Does the agent have everything it needs, without guessing?
• Isolation. Are agent contexts properly separated, so that tenant A's data never leaks into tenant B?
• Economy. Is the context the smallest possible set of high-signal tokens? Every needless token drains the attention budget.
• Provenance. Can you trace where each piece of context came from? That matters for debugging.

A decision path

For a short task, under about 20 steps and with no long-range dependencies, a sliding window is enough. Keep the last N turns plus anchored system instructions. There's no overhead and the implementation is the simplest.

For a long-running task, where a step-5 decision affects step 50, use compaction that preserves architectural decisions. Warn at about 80% capacity and compact automatically at about 95%.

For independent sub-tasks with more than 10 tools, use sub-agents that return 1K to 2K token summaries, each working in a clean window.

For many tools but short sessions, load tools just in time instead of preloading everything. Progressive disclosure removes the MCP tax.

What to instrument

Without these metrics, a context architecture has no observability:

• Token growth rate. Steady monotonic growth means there is no compaction strategy. A sawtooth pattern of rise, reset and rise again shows compaction working.
• KV cache hit rate. Below 50% suggests something in the system prompt prefix keeps changing. Check timestamps and other dynamic elements first.
• The 35-minute heuristic. Agents seem to degrade after about 35 minutes of equivalent task time, so plan checkpoints with that in mind.
• Post-compaction amnesia. If the agent re-explores ground it already covered, compaction is probably too aggressive.

One gap stands out for anyone designing these systems: there's no standard metric for information lost in compaction. The error shows up later, not at the moment of compaction. The best timing for compaction is still an open problem, because too early wastes window and too late brings on rot.

SWE-ContextBench (arXiv:2602.08316) backs this up, finding that unfiltered or badly selected experience gives limited or even negative benefit. More context isn't always better, and bad context actively hurts.

Context engineering rules for long-running agents

Designing a context pipeline is a different question from asking what context engineering is, and a handful of conclusions follow.

• Context rot is a physical property of transformers. A better model doesn't fix it; a memory architecture manages it.
• The model is the CPU, the context is RAM, and context engineering is the agent's operating system, which moves the discussion from linguistics to systems engineering.
• Write, select, compress and isolate are complementary layers, not alternatives, and every production agent uses all four.
• Hit rate on the KV cache is the number to watch, and one stray timestamp in a system prompt can add tens of thousands of dollars to a monthly bill.
• There's no one correct philosophy, as Claude Code, Cursor and Manus show.
• Good context engineering means the five quality criteria plus measurable metrics, not intuition.

A practical first step is to give context the opening slot in the next agent design review: where state is stored, how the KV cache is handled, what the compaction threshold is, and which sub-tasks go to sub-agents. This isn't a performance tweak. It's the foundation of a system that works.