Cutting Agent Token Costs: Caching, Routing and Context Hygiene

A first agent often starts out with a 500-token system prompt and two tools, and both numbers tend to climb quickly. The leaked Claude system prompt is said to have been about 24,000 tokens long. OpenClaw users have reported that more than 150,000 input tokens went to Gemini 3.1 Pro on the very first turn, for a reply of 29 tokens. Left unoptimized, an agent that handles 100 messages a day with 166K input tokens apiece costs roughly $996 monthly on Gemini 3.1 Pro and about $2,490 on Claude Opus 4.6. A handful of design choices can pull that down to somewhere near $50 to $100.

The ideas covered here are prompt caching, semantic caching, lazy-loading tools and MCP servers, routing and cascading between models, delegating to subagents, and keeping the working context clean. Each one saves money in a different way, and each one has a price in complexity or quality. None of them is free.

The material falls into four groups. The first is reusing tokens when possible, through prompt caching and semantic caching. The second is trimming the tokens that are always present, like memory and tool definitions. The third is routing work to smaller models, or escalating to bigger ones, and weighing the quality risk against the savings. The last is keeping the context clean, for performance as much as for cost, with a short look at compaction.

Stop paying for the same tokens twice

Model costs don't come only from calling the model too often. They also come from paying again and again to process identical text. Two very different mechanisms help. K/V caching is the machinery under prompt caching, and semantic caching is a separate idea. For long system prompts, prompt caching is the easy gain; semantic caching demands more effort and brings more risk.

How the K/V cache works

Before a model produces anything, it has to process the prompt, and that takes compute, which means latency and money. It makes sense not to reprocess the same content repeatedly.

When a large language model gets a prompt, the text is tokenized, the tokens become vectors, and inside each attention layer those vectors are projected into K/V tensors.

Caching means holding on to those tensors so they don't have to be computed again. An inference engine already keeps K/V tensors during generation, because otherwise the math wouldn't run at a reasonable speed. The step beyond that is keeping the cache after the response ends and not discarding it.

On the next request, the system checks whether the start of the prompt matches something it already has tensors for. If it does, it loads them and skips the processing.

The economics are easy to see with round numbers. Say processing 2,000 tokens takes one second and the system prompt is 10,000 tokens. Skipping that recomputation saves 5 seconds on every call, though real prefill throughput varies a lot with the setup.

The match against the stored cache has to be exact. If any token changes, the tensors for that part of the prompt no longer apply and it has to be processed again. People trip on this constantly: an extra space, a reordered tool definition or a timestamp in the wrong place breaks the match.

So a stored cache speeds up requests and, in turn, makes them cheaper. Storing it isn't free, though. Cached K/V takes memory on the serving side, which is why many providers expire entries after about 5 to 10 minutes.

Nobody has to build this from scratch, because frameworks and API providers handle it, each with its own rules. Teams hosting an open-source model would normally use a serving framework such as vLLM, which has a caching layer built in. API users follow each provider's policy instead.

Prefix caching in vLLM

The caching layer in vLLM chops the prompt into blocks, hashes each block from its own tokens plus the tokens before it, and stores the K/V tensors under those hashes. As with most setups, the static part to be cached belongs at the start of the prompt.

Three flags matter:

• --enable-prefix-caching turns the feature on.
• --block-size sets the number of tokens per block. With a block size of 16, the engine cuts off after 16 tokens and starts a new block.
• --kv-cache-memory-bytes sets the K/V cache size per GPU explicitly. More memory lets cached blocks last longer, but many different long requests at once fill it faster and old blocks get evicted sooner.

Other options exist, including SGLang with RadixAttention for prefix caching, and LMCache, which is meant to plug into serving engines. The mechanics match what's described above.

Caching through API providers

Most teams use API providers, and each has policies to follow so prompts hit the cache. The prompt has to be structured for it. OpenAI is a clear example: caching a part of the prompt requires an exact prefix match, meaning the same static input at the start. So stable instructions, examples and tools go first, and variable content comes after.

Sending a prompt-cache-key can route similar requests together and raise hit rates. Caching switches on automatically for prompts of 1,024 tokens or more, but the first 256 tokens are used to route requests back to the same cache, so the static portion needs to be longer than 256 tokens.

Anthropic works differently: caching has to be enabled with the cache-control parameter. Eviction usually follows 5 to 10 minutes of inactivity at both providers, and it can be extended. Anthropic lets you stretch it to an hour at twice the price.

When self-hosting, the time saved is also money saved. With API providers the benefit arrives as cheaper cached input tokens. OpenAI discounts cached input by as much as 90% off the normal input price. Anthropic gives the same discount on cached input but also bills for keeping the cache, so misuse can end up costing more than not caching.

When 90% of a prompt is static, and the cache is used properly, the cost drops sharply, as the chart shows.

An interactive calculator was built for this comparison so the numbers can be adjusted. Overall, prompt caching is a solid win for anyone with a long, unchanging system prompt.

Semantic caching

Semantic caching is another thing entirely. It matches on meaning: if a new request is similar enough to an earlier one, the stored answer is returned. That sounds easy, but there are clear pitfalls.

Matching text by meaning uses embeddings, which are vectors compared with cosine similarity. High similarity should mean similar meaning, though that depends on the embedding model.

The proposal is to pair similar requests with answers that already exist. A question like "What is the capital of France?" and a hurried "Quick, give me the capital of France" should both land on one stored answer, with no LLM call to produce it again. This works nicely when many people ask near-identical generic questions and the underlying data goes stale slowly.

So why not do it everywhere? Because a lot can go wrong. The design has to settle several questions:

• What similarity threshold counts as a match.
• How long an answer stays valid.
• How multi-turn conversations are handled.
• What actually gets stored.
• Whether a router should sit in front.
• How users are kept separate.
• What happens if a wrong answer gets cached.

Time to live deserves special care, since information goes stale at different speeds for different questions. The mechanics are simple, but production use needs metadata filters and tags, such as user, workspace, corpus version, persona, session scoping, a smart TTL and some rule for whether a returned match is good enough. It turns into a small project.

A workable design uses the semantic index to find an earlier question and lets different questions point to the same stored answer, which limits storage growth. TTL can follow usage: keep frequently reused entries longer and drop the rest. It's also wise to build this after the logs show repetition and not at the outset, since some use cases simply don't suit it.

Many databases can do the work, and libraries handle the plumbing, among them semanticcache, prompt-cache, GPTCache, vCache, Upstash semantic-cache and Redis with LangCache.

The savings can be real. Redis claims up to 68.8% fewer API calls and a 40 to 50% latency improvement, though that figure is partly marketing, since it comes from a clean question-and-answer scenario. Results depend completely on the setup. A Q&A bot with many redundant calls saves more, and a coding agent whose calls are all unique saves less.

Prompt caching shines when a changing question sits inside a big static prompt, and semantic caching shines when people keep rephrasing the same question. A 15% semantic match rate adds only modest savings, but a rate above 45 to 50% starts to look worthwhile.

The two also differ in what can go wrong. A prompt-cache miss only costs the normal price, so the failure is cheap and visible. A semantic-cache miss is harmless too, but a false hit is not: a user receives an answer written for a different question, with full confidence, and nothing in the logs flags it. That asymmetry is why semantic caching calls for conservative similarity thresholds, user scoping and a way to review hits, whereas prompt caching mostly needs disciplined prompt layout.

Ordinary caching also deserves a mention. Expensive deterministic work such as SQL query results, tool outputs and retrieval results should be cached so it never runs more often than necessary. One example is a tool that gathers keyword data for summarizing, caches it until the data goes stale, and reruns it only when that route is hit again.

Semantic caching is an interesting concept that saves tokens in certain use cases, but doing it well takes engineering.

Don't preload dormant tokens

A system prompt can swell because of bulky tool definitions or growing memory. Small agents don't feel it, but agent specs that keep growing can be slimmed down by fetching information on demand, or at least trying to.

Keep the always-loaded layer small

When an agent's prompt gets past a certain size, the layer that is always loaded should stay tiny and steady, with growing detail stored apart from it. Loading a few hundred tools, or sending full MCP server descriptions that keep changing, makes everything noisy.

The issue is performance as well as cost. When any layer keeps changing, prompt caching also becomes much harder to hit. The top layer should be compact and stable, enough to tell the model where it is and where to go next, without carrying the whole world up front.

Claude Code is reported to follow this pattern for memory. An index file is always loaded and is not supposed to grow past 200 lines, while detailed topic files live elsewhere. How the agent behaves in practice compared with what the system intends is a separate question. Similar thinking appears in Claude's advanced tool setup, in the layered design of Claude Skills and in attempts to lazy-load MCP tools instead of putting every server definition in the prompt up front.

Does it work?

The principle holds: when context gets large, a model has more trouble choosing the right action. The space is still young, so one concrete tool shows how it can work.

A few months back Anthropic released an advanced Tool Search feature, meant to shrink what sits in context even when the model can reach hundreds of tools. The company reports having observed 55K to 134K tokens of tool definitions before optimization, and that picking the wrong tool is a common failure once context gets that large.

A search tool optimizes the context by letting the LLM look up tools and not define them all up front. A configuration looks like this:

tools=[
        {
            "type": "tool_search_tool_bm25_20251119",
            "name": "tool_search"
        },
        {
            "name": "search_contacts",
            "description": "Find a contact by name or email.",
            "input_schema": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"}
                },
                "required": ["query"]
            }
        },
        {
            "name": "send_email",
            "description": "Send an email to one or more recipients.",
            "input_schema": {
                "type": "object",
                "properties": {
                    "to": {"type": "string"},
                    "subject": {"type": "string"},
                    "body": {"type": "string"}
                },
                "required": ["to", "subject", "body"]
            },
            "defer_loading": True
        }
    ]

One tool named tool_search is defined, using an out-of-the-box option, BM25 or Regex, or a custom one. Then a single tool is marked as deferred, as an illustration. That's only worthwhile with 10 or more tools. Anthropic performs the search, so the way it inserts the tool schema into the system prompt and the search itself stay hidden. Once a tool matches, its definition is appended inline to the conversation as a tool_reference block.

The idea is neat, since it gives a smaller starting context at the price of one extra search step. Outside testing with 4,000 tools gave fairly lackluster results, so there's room for more testing. Tools also have to be described well enough to be found, and debugging gets harder when the intermediate step is invisible.

The pattern turns up elsewhere under the plain label of good AI engineering. Don't expose the agent to a huge messy context. Give it a way to narrow things down, and let it inspect or load a tool only when needed.

The savings here depend on how many tokens are being sent to begin with. A second calculator compares tool search with prompt caching.

The sample above shows estimated savings for prompt caching combined with lazy-loaded tools. Both bring savings, but combining them isn't a huge change on top of either alone. Tool search isn't only about money, because it also keeps context clean for performance. For savings alone, adopting at least one of the two is the biggest win.

Use cheap models for cheap work

This section covers routing prompts to different models and using cheaper subagents for certain tasks. Both can cut token costs and can put quality at risk. Many argue that 60% or more of incoming questions are easy and don't need the strongest model, certainly not a reasoning one.

Commercial products already do some of this. ChatGPT is reported to route on signals such as conversation type, complexity, tool needs and explicit intent like "think hard", and Claude relies on description-based delegation plus built-in subagents such as Explore. The concept is easy to grasp; doing it without hurting quality is the hard part. What follows covers predictive routing and output-checked approaches such as cascades and subagents, and a third calculator covers the savings.

Route by difficulty

Request-level routing guesses difficulty and intent before any output exists. The upside is large, but one bad choice can poison a whole session, so quality is a real concern. Some kind of router model has to decide where each request goes.

OpenAI's routing signals aren't public, and users often feel they've been moved to a weaker model at an irritating moment. The open-source world offers more to examine. RouteLLM from LMSYS, the Berkeley group behind Chatbot Arena, learns from real preference data gathered in the Arena. It combines standard embeddings with a tiny router head, which should keep hosting costs low, and its authors report big cost reductions while keeping most of GPT-4's performance. That claim hasn't been independently tested here.

The LLMRouterBench paper adds a sobering note: many learned routers barely beat simple baselines such as keyword or heuristic routing, embedding nearest-neighbor and kNN-style routing.

The fancy routers may not give much of a boost over something simple. People haven't dropped routing, and they keep experimenting, but savings aren't a sure thing when answer quality can't keep up. Hosted options exist too, such as OpenRouter Auto and Switchpoint, though nothing public describes their routing internals or accuracy. The calculator compares LLMRouter, heuristics, a self-hosted classifier, LLM-as-router, RouteLLM and OpenRouter Auto. Real-world quality needs proper testing with clear numbers, which this space deserves in its own right.

Start cheap, escalate on low confidence

Skipping any attempt to predict a request's difficulty from its prompt, the cheap model can answer first and a check then decides whether to keep its answer or escalate. Google's "Speculative Cascades" write-up describes the tradeoff: small models first for cost and speed, larger ones only when needed.

The cheap model generates, and then a lightweight checker looks at signals like logprobs, entropy or margin-style uncertainty, or semantic alignment.

The approach is attractive because prompt difficulty is hard to predict and most routers aren't perfect, while quality is easier to judge once an answer exists. It only makes sense if most questions can be answered by the simpler model, since escalated ones pay for two calls. Teams using it say validation latency between calls can stay under 20 ms.

An open-source implementation named CascadeFlow claims 69% savings with 96% quality retention compared with GPT-5. The prompts it was tested on had verifiable ground truth, though, such as math answers and multiple choice. The main worry is that small models are often confidently wrong, so it can be sensible to start with conservative thresholds and escalate more often, which pushes costs up. Adding cheap-first cascading to the calculator suggests savings near 50%, provided larger models are needed for some requests at all.

Delegate to subagents

Subagents delegate work to isolated agents, sometimes running smaller models, so it's a form of routing too. The savings are smaller here. The value is also in keeping context clean so each agent can focus fully on its own task.

Claude Code ships with built-in subagents. The Explore subagent is explicitly a Haiku worker for exploring and searching a codebase, which applies the idea of cheaper models for cheaper tasks. The main session also delegates by matching descriptions, invisibly, and the user only sees a lower total bill. Since the orchestrator normally stays involved in planning, synthesis and retries, the saving falls short of what routing delivers.

By the calculator's numbers, subagents shave about 11% off the no-routing option, so they aren't the main lever for cost cutting alone.

Keep the context clean

Good context engineering is mostly a performance matter, but it can affect cost as well. Agents collect clutter over time: stale attempts, old plans, duplicated state, logs, repeated observations and raw tool outputs. First-time agent builders are especially prone to it, since they keep dumping results into the main agent's working state.

It happens everywhere. OpenClaw users complain about context build-up, and so do Claude Code users, because adding stuff is easier than cleaning it up.

The hard part is a state pipeline

This is a two-tier problem. It isn't just compressing the chat. New material also has to stay clean as it enters the working state, and that's tedious engineering. A bad pattern looks like this, where the agent keeps dumping raw output into context:

bad state:
agent does work
→ dumps tool output into context
→ reads files
→ dumps files into context
→ runs tests
→ dumps logs into context
→ retries
→ keeps everything

The real task is preserving the right state while deleting the exhaust along the way. Raw output can go to an archive, with only what's needed in active context. The enemy is usually tool-output bloat, so making tools less noisy by default is part of the work. A healthier active context looks like this:

Good active context

[system rules]
[project rules]
[user task]
[current working state]

Keep:
+ auth flow lives in auth.ts + session.ts
+ bug only happens on refresh path
+ failing test: session_refresh_keeps_user
+ likely overwrite during refresh
+ files in scope: auth.ts, session.ts, auth.test.ts

Drop:
- raw grep results
- full test logs
- duplicate file dumps
- dead-end retries

Certain pieces of context could also carry a lifecycle or an expiry. Once compression becomes necessary, it's then easier to tell what the LLM actually needs.

Anthropic's notes on long-horizon tasks say compression should preserve architectural decisions, unresolved bugs and implementation details. LangChain's autonomous compression lets the agent decide when to compact, and does not wait until the context is already bloated, which appears to be Anthropic's approach. Teams are starting to treat compression as a systems problem, with benchmarks and agent-specific policies, and no longer as a generic summarization trick.

A recent paper by Jia et al. argues that at 6x compression it achieved a 51.8 to 71.3% token-budget reduction along with a 5.0 to 9.2% improvement in issue resolution on SWE-bench Verified. So this isn't only about cost. It helps performance generally.

Removing junk can plausibly clear 30 to 70% of a context, which saves about as much in dollars. For a 10k context window, cleaning up 30 to 50% over 100k runs might save up to $1,500, and at 40k the figure rises to $6,000. Compressing agents that already use very small cheap models may cost more than it saves. Still, clean context doesn't sacrifice quality the way semantic caching or routing can, so done well it's a clear gain. The catch is the effort to do it.

Choosing among them

There are four ways to cut token costs, and the right mix depends on the use case:

• Use prompt caching with large system prompts that stay unchanged across looped LLM calls.
• Use semantic caching for a generic Q&A bot that has to stay cheap.
• Test routing when the agent must handle both easy and hard questions.
• Keep the context as clean as possible to avoid sending unnecessary tokens.