Where Our LLM API Bill Went and the Architecture Changes That Cut It

An LLM bill can run high even when the models themselves aren't unusually pricey. In this case the cause was a set of habits: the same 1,800-token system prompt went out with every call, failures were retried immediately with no backoff, and every workflow step defaulted to a frontier model whether it needed one or not. The estimate for this workload was that the headline price per token explained about 20% of the problem and the architecture explained the other 80%. That split belongs to this particular system and isn't a general benchmark. The lesson still travels well: the listed token price tells you the rate, not how efficiently your app uses it.

Before switching providers or haggling over prices, the spending got instrumented, which made every later decision rest on evidence and not on instinct. You can't optimize what you don't log.

Attribute tokens to workflow steps

The first requirement is attribution at the step level. For each API call, record the workflow step, the model, the input tokens and the output tokens, then aggregate nightly by step and model. With prices quoted per million tokens, a step's cost works out like this:

step cost =
number of calls × [
  (average input tokens ÷ 1,000,000 × input price per 1M tokens)
  +
  (average output tokens ÷ 1,000,000 × output price per 1M tokens)
]

One week of data gave a representative breakdown for a coding-agent workload, and the main finding was a surprise: the planning step made up only 8% of calls yet 41% of total spend. It was costly because it paired the longest contexts with the priciest model. Before measuring, the assumption had been that code generation was the expensive part, and the data showed that to be wrong. Request count alone wasn't the right metric. Total spend per step was.

A minimal implementation

It turned out to be simple, because API responses already carry token-usage data:

import logging
from openai import OpenAI

client = OpenAI(
    api_key="KEY",
    base_url="https://api.cometapi.com/v1",
)
def tracked_call(step_name, model, messages):
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0,
    )
    usage = response.usage
    logging.info(
        "llm_call step=%s model=%s prompt=%d completion=%d",
        step_name,
        model,
        usage.prompt_tokens,
        usage.completion_tokens,
    )
    return response

With those logs in place, the cost report is essentially a GROUP BY. The aim isn't an elaborate observability platform. It's making sure every token can be traced to a workflow step and a model.

Take the gateway discount, but treat it as the floor

Requests here go through CometAPI, and in this usage the resulting cost came in roughly 20% below paying the providers directly for the same model mix. That was a low-effort saving, since the client pattern and model choices stayed the same and the invoice dropped across models already in use. It was taken right away because it needed almost no architecture work.

The gateway wasn't the biggest lever, though. Later architecture changes saved more, at the price of real engineering work. Implementation order and financial impact don't always match, so grab cheap pricing savings immediately and spend engineering time on model routing, prompt overhead and retry behavior. There's no reason to delay an instant saving while building something fancier, but a lower rate can't make up for an inefficient request design. The billing details are in CometAPI's pricing docs.

Route each step to the right model tier

Tiering means giving different workflow steps different models, according to the quality each step needs. Planning really did need a frontier model. It's low in volume but high in impact, because a bad plan can damage every action after it. Bulk edits and lint fixes have different needs: high volume, tightly constrained and easy to retry or verify. The routing logic changed accordingly:

def choose_model(step):
    if step in ("plan", "hard_refactor"):
        return "claude-sonnet-4-6"  # High-value reasoning steps

if step in ("bulk_edit", "lint_fix"):
        return "gemini-2.5-flash"  # Lower-cost, fast, retry failures
    return "gpt-5-mini"  # Middle-tier default

That moved 46% of calls to a cheaper, faster model and kept the expensive one for planning and hard generation. Tiering gave the largest single line-item reduction on the bill, because it went after the high-volume steps and didn't chase small savings on low-volume ones. The principle is simple: use the strongest model where mistakes cost the most downstream, not everywhere by default.

Cut the repeated prompt overhead

The 1,800-token system prompt rode along with every request. Much of it was defensive boilerplate gathered over months, where each instruction seemed harmless when added but together they formed a big fixed cost per call. The fix was to drop redundant instructions, keep only the essential constraints and move the stable part into a cached prefix wherever the provider supports prompt caching.

It's easy to underestimate the scale. At one million calls, an unchanged 1,800-token system prompt alone amounts to:

1,000,000 calls × 1,800 tokens
= 1.8 billion input tokens

and that's before any task-specific context. Prompt tokens are tokens like any other. The bloated prompt raised API cost and also ate into the tokens-per-minute allowance on every request, so cleanup helped twice: it lowered recurring input cost and let more useful work fit inside the same rate limits. Trimming came before caching, since caching unnecessary text would only preserve the underlying design problem.

Retry with backoff and jitter

Retries made up 12% of calls. Many followed an HTTP 429 rate-limit response: the system retried at once under almost identical conditions, got another 429 and sometimes seemed to stretch out the limited period. Exponential backoff and jitter replaced the immediate retry. Backoff raises the delay between attempts progressively, and jitter adds a little randomness so concurrent requests don't all retry at the same instant. Retry volume dropped by more than half.

Rejected requests aren't all billed the same way, but blind retries still raise request volume and can trigger repeated billable work when a request reaches model execution. Here, retry control improved reliability and also controlled cost. The wider rule is that a retry should answer the cause of the failure, and repeating the same request instantly isn't recovery.

Which change mattered most

The four changes hit different parts of the cost equation:

• Model tiering gave the largest single line-item reduction, shifting 46% of calls to a cheaper model while the costly one kept the work that matters most. It took about a day to implement.
• Gateway routing was the easiest immediate saving, cutting the cost of the same model mix by about 20% with almost no engineering.
• Prompt trimming and caching took out recurring input overhead, because even the simplest task had been carrying the 1,800-token prompt.
• Backoff and jitter cut unnecessary request volume, with retries falling from 12% of calls to less than half that.

Together they achieved more than any single change. Don't add the savings up as though they were independent discounts. Gateway pricing changes the rate paid per token, tiering changes the blended rate across steps, prompt trimming changes the number of input tokens, and retry control changes the number of requests. Every change also alters the cost base the next one works on.

Working out a monthly cost

Suppose a system handles one million requests a month, averaging 2,000 input tokens and 500 output tokens each. The monthly token volume is:

Input:
1,000,000 × 2,000
= 2 billion tokens

Output:
1,000,000 × 500
= 500 million tokens

With prices quoted per million tokens, the baseline monthly cost is:

monthly cost =
2,000 × input price per 1M tokens
+
500 × output price per 1M tokens

Here, the gateway cut that baseline by about 20% before any architecture changes. Tiering then lowered the blended token rate, because 46% of requests moved to a model costing a fraction as much per token. Prompt trimming reduced input tokens on each affected request, and retry control removed needless requests. So a correct calculation needs four variables: request volume, the input and output token mix, model allocation per workflow step and current prices for each model. That's why there's no universal "you will save X%" number. The only useful estimate comes from your own workload.

Three ways cost work goes wrong

Optimizing low-volume steps because they're easy. It's tempting to shave pennies off a small step while a high-volume one eats most of the budget. Sort targets by total spend and not by how simple they are to change, which is why attribution has to come first.

Cutting cost without testing quality. A step moves to a cheaper model, the invoice drops and the change looks like a win. Three weeks later someone notices degraded output. Gate every model swap behind an evaluation set, a representative collection of tasks that catches regressions. A saving that fails the quality bar isn't an optimization.

Building a cache before measuring reuse. Caching helps only where content really repeats. An elaborate cache built before checking prompt reuse can burn engineering time without touching the main cost driver. Check how much repetition exists before anything else, and only then cache the prefixes or responses that really come back. Cost work done without measurement can improve the invoice while worsening quality, latency or reliability.

Leave the instrumentation on

Don't switch the instrumentation off after the first pass, because costs creep. The system prompt gets longer bit by bit, a newly added workflow step starts on the pricey model, an old retry bug returns, or traffic moves onto a step that used to be minor. The dashboard that revealed the original problem is also what stops it silently returning. A monthly review of the step-level cost table takes about ten minutes and has already caught two regressions.

Architecture, not pricing

A price per token is only a rate. What you actually spend depends on your architecture: the number of tokens bought, the models that handle them and the amount of repeated work. For this coding-agent workload, the most effective sequence was:

1. Instrument every call by workflow step and model.
2. Take the low-effort gateway saving.
3. Send each step to the cheapest model that meets its quality needs.
4. Remove repeated prompt overhead and cache stable prefixes where supported.
5. Swap blind retries for backoff and jitter.
6. Keep monitoring permanently so costs don't creep back.

Biggest win: tiering. Fastest win: the gateway, at close to 20%. Trimming the prompt took a fixed cost out of each request, and smarter retries halved the retry traffic and then some. Hunting for the cheapest advertised token price matters much less than knowing where your own tokens go. Measurement is what makes the architecture visible.