Cutting LLM Costs: Caching, Model Routing and Prompt Trimming

A prototype with an LLM behind it rarely costs much. Then traffic arrives, and a system that looked cheap during development gets processed thousands or millions of times a month. At that point the inference bill becomes a line item people ask about in meetings. Swapping in a smaller model everywhere is the obvious reaction, and it's often the wrong one. Most of the savings come from three decisions: how models get called, what gets sent to them, and which model handles which request. Below are caching, routing and prompt strategies that cut spending without hurting the quality users actually notice.

What drives the bill

Several factors move LLM cost:

• how many requests arrive
• input tokens and output tokens
• which model serves each call
• the size of the context window
• how often the same question repeats
• agent and tool calls
• retries after failures
• embedding and retrieval workloads
• batch processing needs

A simple chatbot makes one model call per question. An agent can make a chain of them for one task: user, agent, retrieval, tool, model, tool, model, final answer. If every link in that chain runs on an expensive model, the total climbs fast. For that reason, cost control belongs in the architecture from the start, and bolting it on after the invoice shocks someone is much harder.

Measure before you change anything

Optimizing blind is guessing. Start by tracking:

• requests per day and tokens per request
• input tokens versus output tokens
• cost per request, per user, per feature and per model
• cache hit rate
• latency, error rate and retry rate
• agent tool calls

Real data often holds surprises. You might find that 20% of users generate 70% of the spend, or that one RAG workflow stuffs huge amounts of useless context into every call. Neither shows up until somebody looks.

Cache repeated work

Caching is among the easiest ways to skip inference altogether. Picture a support assistant. Thousands of customers ask some version of "What is your refund policy?" When the answer rarely changes, generating a fresh response each time makes no sense.

The flow is short. A query arrives and the cache is checked. On a hit, the stored answer goes back to the user. On a miss, the model is called, the result is saved, and the response goes out. Repeated model calls drop sharply.

Semantic caching

Plain caching needs an exact string match, and "What is your refund policy?" and "Can I get a refund?" are different strings. A semantic cache compares meaning instead. Embeddings represent each request, and when similarity passes a chosen threshold, the system reuses an existing answer. That works well for:

• FAQs
• customer support
• internal knowledge assistants
• product documentation
• repetitive enterprise workflows

Set the threshold carefully. Too loose and different questions get the same answer, too tight and the cache rarely fires.

Route by difficulty

A common architectural error is sending every request to the strongest model available. Compare three requests: "What are your support hours?", "Summarize this customer complaint," and "Analyze these financial documents and identify potential risks." A premium reasoning model would handle all three, but paying for it on the first is waste.

Model routing fixes that. Easy tasks go to a small, efficient model, moderate ones to a mid-tier model, and hard ones to a high-capability model. A classifier at the front inspects each request and picks the destination. A production router may weigh query complexity, required accuracy, context length, user priority, task type, latency needs and cost limits. Spending drops noticeably, and nothing forces all traffic through a weak model.

Trim the prompt

Long prompts cost money, especially when one large system prompt rides along with every call. Imagine 3,000 tokens of instructions, 5,000 tokens of retrieved context and 500 tokens of user input on each request. Even for a tiny question, the model processes about 8,500 tokens.

Audit prompts for:

• repeated instructions
• examples that add nothing
• duplicated context
• heavy formatting rules
• irrelevant details
• stale instructions
• tool descriptions nobody uses

A shorter prompt often does as well as a longer one if it states the task clearly. The target isn't the shortest possible prompt. It's the smallest prompt that still gets the required result reliably.

Send fewer, better chunks in RAG

RAG pipelines get expensive when too many retrieved documents are handed to the model. Say retrieval returns 20 chunks of 1,000 tokens each. That's 20,000 tokens of context for one request, and the model probably doesn't need most of it.

A better path is query, retrieval, reranking, then only the top relevant chunks into the LLM. Token use, latency and noise all fall, and answers may even improve.

Compress what's left

Even relevant documents carry plenty of irrelevant text. Context compression extracts the useful parts before the model sees them. A 10,000-token document might shrink to 1,500 tokens of relevant material. That pays off with:

• legal documents
• financial reports
• technical documentation
• enterprise knowledge bases
• long customer histories

Cap the output

Models can write far more than a user wants. For a question like "What is the status of my order?", an answer of 1,000 tokens isn't needed. Set sensible output limits and response formats. For structured tasks, a compact format works better than prose:

Status: Shipped
Expected Delivery: Friday
Tracking Available: Yes

Fewer output tokens means a lower bill, and usually a faster reply too.

Use plain code where code will do

This may be the most important rule in AI engineering: some tasks don't need an LLM. Checking whether an order is above $1,000 is one line, if order_value > 1000. Ordinary software already handles calculations, sorting, filtering, validation, date math, database queries and deterministic business rules. Reserve the model for places where language understanding or reasoning adds real value.

Keep agents on a leash

Agent systems can rack up surprising inference costs. A single task might trigger a planning call, a tool-selection call, a retrieval call, an analysis call, a verification call and a final-response call. Run all six on a large model and the cost multiplies.

A better design can:

• cut needless agent loops
• use deterministic tools where possible
• cap the maximum number of iterations
• hand routine steps to smaller models
• cache intermediate results
• avoid repeated retrieval
• merge compatible operations

The goal isn't fewer useful actions. It's the removal of unnecessary ones.

Batch what isn't urgent

Not everything needs a real-time answer. Document classification, data enrichment, content processing, report generation, large-scale summarization and dataset labeling can usually run asynchronously or in batches. Depending on the provider and workload design, batching can improve utilization and lower the price of large jobs.

Choose the model like an engineer

Treat model selection as an engineering decision, judged on several questions:

• Quality: how accurate must the output be?
• Cost: what's an acceptable price per request?
• Latency: how fast must the reply arrive?
• Context: how much information must the model take in?
• Reasoning: how hard is the task?
• Reliability: how consistent must results be?

Swap "Which model is best?" for "Which model is sufficient for this task?" That small change can reshape production economics.

Prove nothing broke

Cost cuts shouldn't quietly erode quality, so measure every change. Build an evaluation set from real queries, then compare the baseline with the optimized system on accuracy, relevance, factuality, task completion, user satisfaction, latency and cost.

A made-up example: a baseline scoring 92% at $0.020 per request versus an optimized version at 91% and $0.009. If the business accepts a one-point quality dip, the second option is far cheaper to run.

Putting the layers together

A production application can chain these ideas in order: classify the request, check the cache, retrieve or process data, filter the context, route to a model, call the chosen LLM, validate the output, store it in a response cache, and return it to the user. The expensive model only runs when nothing earlier in the chain could answer.

What savings to expect

Don't read "60% off" as a promise. Results depend heavily on the workload. Stacking several optimizations can still bring big reductions. Take a hypothetical monthly bill of $10,000 and look at where it could shrink:

• cut repeated requests by 20%
• move suitable workloads to cheaper models
• reduce unnecessary input tokens
• retrieve and send only relevant context
• eliminate needless tool and model calls
• stop overlong responses

Together these can shave a large share off the total. The point is that savings usually come from many small architectural improvements, not from one magic trick.

A pre-launch checklist

Before putting an LLM application into heavy use, ask these questions.

• Necessity: does every step need an LLM? Could deterministic code do some of it? Are agent loops bounded?
• Prompts: are system instructions bloated? Are the examples needed? Is irrelevant context creeping in?
• Retrieval: are too many chunks retrieved? Is reranking in place? Can the context be compressed?
• Models: is the priciest model used where it isn't needed? Can requests be routed by complexity?
• Caching: are repeat requests cached? Would semantic caching fit?
• Output: are response lengths controlled? Do structured outputs make sense?
• Monitoring: is token usage tracked? Is cost per feature known? Are quality metrics watched next to cost?

Aim for sustainable, not cheapest

"Make the AI as cheap as possible" is the wrong goal. A better one is reaching the required business quality at the lowest sustainable cost. A support app can use a small model for simple FAQs and a stronger one for tangled complaints. A coding assistant may need heavy reasoning for difficult programming tasks. A document pipeline might prioritize throughput and consistency. No universal recipe exists, and the right architecture depends on the workload.

LLM spending doesn't have to rise in step with usage. Caching, routing, tighter prompts, leaner RAG, compression, efficient agents, output limits and continuous evaluation each help, and the biggest win is usually just avoiding calls that never needed to happen. If an AI application is getting costly to run, cutting usage may not be the answer. Engineering the system more carefully often is.