Cut LLM spend with a LiteLLM token router

LiteLLM is an open-source gateway that speaks the OpenAI request shape to many providers. It shines in several roles; this piece focuses on model routing — sending each call to a cheap-enough model so token spend stays under control.

What a model router does

A router sits between your app and the providers. The client still authenticates once against the gateway. The gateway inspects each request and, using a policy you define, picks which upstream model should answer. Treat it as a lightweight decision layer: trivial asks do not need the most expensive frontier model.

Token-based routing intuition

LiteLLM can count input tokens before it forwards the call. A practical heuristic:

• Short “read a page and summarize” work → Haiku-class models
• Dense Python that needs deep reasoning → Opus-class models
• Everything in the middle → Sonnet-class models

That pattern is token-based routing. Short prompts go cheap; long ones go premium. Production systems often add intent classification for finer accuracy, but even the token heuristic alone can cut spend by roughly thirty percent.

Without routing

Every request lands on a single expensive model. Costs track volume linearly — no intelligence about difficulty.

With routing

The same traffic is classified and spread across tiers. Engineers keep calling one gateway URL; the proxy chooses which billable model actually runs.

A simulated month of engineering traffic

I ran both setups through LiteLLM. First, everything hit Opus. About seven hundred requests produced a bill near $60.

Then I enabled the router. Opus only received prompts whose input tokens exceeded 300. The custom logger that implements that policy looks like this:

from litellm.integrations.custom_logger import CustomLogger
import litellm
import random

class TokenRouter(CustomLogger):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        # Skip routing for mock models - simulation handles model selection
        if "mock" in data.get("model", ""):
            return data

        # SMART ROUTING based on token complexity
        messages = data.get("messages", [])
        token_count = litellm.token_counter(
            model="gpt-3.5-turbo",
            messages=messages
        )

        if token_count < 50:
            data["model"] = "haiku"
            tier = "HAIKU  (5%)"
        elif token_count < 300:
            data["model"] = "sonnet"
            tier = "SONNET (30%)"
        else:
            data["model"] = "opus"
            tier = "OPUS   (65%)"

        print(f"[ROUTER] Tokens: {token_count} → {tier}")
        return data

proxy_handler_instance = TokenRouter()

Out of 741 requests, 494 (about 66%) still reached Opus. The rest split between Sonnet and Haiku, with Haiku taking the lightest share. The bill for a comparable workload dropped to $43 — no change to how developers write prompts.

Takeaways from the numbers

• 741 requests — a believable month of engineer usage
• $60 baseline — Opus for everything
• $43 with routing — same work, smarter selection
• ~28% saved — entirely at the proxy, invisible to clients

Savings grow as you add engineers, pricier models, and higher token volumes. Not every question needs the strongest model in the catalog.

LiteLLM’s router handles that quietly. Engineers keep the same OpenAI-shaped client; finance sees a smaller invoice.

Benchmarks here used LiteLLM v1.98 locally with Ollama stand-ins and Anthropic list prices for cost math. Real savings depend on how your request mix distributes across difficulty.