Compress every LLM call at the LiteLLM gateway with Headroom

Teams that already front models with LiteLLM can shrink inbound tokens for every upstream provider while leaving agent and app code alone. Headroom hooks the proxy as a pre-call guardrail, squeezes tool results and RAG chunks before the provider call, and still lets the model pull full originals through a retrieve tool when necessary.

Headroom, from earlier in this series, shrinks tool results, log blobs, and retrieval chunks before the model sees them. Compression itself was never the hard part—getting it onto the request path was.

As of June 2026 that friction drops for LiteLLM users: Headroom attaches as a native gateway guardrail. One switch compresses every call that flows through.

That combination removes the last excuse. Savings were always real; wiring was the tax. At the gateway you flip one control and leave agents and clients alone.

Why the gateway

Most multi-model production teams already sit behind a gateway.

LiteLLM is a common choice: one OpenAI-compatible front door over a hundred-plus providers, with routing, keys, budgets, and logging centralized.

That makes the gateway the natural home for compression. Every request already passes through, so one compressor covers every agent, client, and provider.

You configure once instead of instrumenting each app. Because LiteLLM speaks both OpenAI /v1/chat/completions and Anthropic /v1/messages, a gateway compressor reaches apps that speak OpenAI shapes, Anthropic-shaped tools such as Claude Code, retrieval services, and bespoke agents together.

Compression stops being an agent bolt-on and becomes a platform feature—the usual path for infrastructure: down the stack into the shared layer everyone already runs.

How the guardrail works

Headroom plugs in as a pre-call guardrail: inside LiteLLM’s lifecycle just before the payload goes upstream.

Sidecar flow:

Client (Claude Code, your app, any OpenAI/Anthropic client)
        |
        v
LiteLLM proxy  ── pre_call ──►  Headroom (sidecar): compress messages
        |                              |
        |   ◄── compressed messages ───┘
        v
LLM provider (OpenAI / Anthropic / Bedrock etc.)

Clients still hit LiteLLM normally. On pre-call, LiteLLM invokes Headroom, swaps compressed messages in, and forwards. Clients and providers never talk to Headroom directly—that invisibility is the point.

Two safety properties matter.

Nothing is permanently lost

Same Compress–Cache–Retrieve idea as the earlier Headroom write-up. Originals stay cached; LiteLLM exposes a retrieve_headroom tool. If the model needs full context, it calls the tool. Compression is reversible, not “lossy and hope.”

Fail open

Headroom is a sidecar, not inline critical path. If it is down or slow, calls proceed uncompressed. You keep savings when healthy and traffic when not—the right failure mode for a cost feature.

Turning it on

Needs LiteLLM v1.92.x or later (early adopters used v1.92.0-dev.1).

Rollout is small: attach headroom-compression and choose breadth:

• Per request — enable on individual calls
• Per virtual key — attach to a key; every call with that key compresses
• Globally — default_on: true for all proxy traffic

Because the guardrail sits at the gateway and handles both request formats, the same rollout covers OpenAI-compatible clients, Anthropic clients, home-grown agents, retrieval services, and IDE coding tools.

Per-key is the admin one-liner: attach the guardrail to a virtual key, hand the key out, and every provider/client behind it compresses automatically.

Claude Code example: set ANTHROPIC_BASE_URL to the LiteLLM proxy with the compression key—no client install, no code diff. Same idea for OpenAI-style clients via OPENAI_BASE_URL. The tool never knows Headroom exists.

Confirm application via x-litellm-applied-guardrails or the Guardrails panel in LiteLLM Logs. Authoritative setup lives in LiteLLM’s Headroom guardrail guide (see references)—follow that for exact config on a fresh release rather than stale snippets.

If you are not on the LiteLLM proxy

The gateway guardrail is the headline, not the only path.

Calling litellm.completion() in Python — use the callback:

import litellm
from headroom.integrations.litellm_callback import HeadroomCallback

litellm.callbacks = [HeadroomCallback()]
# Every call is now compressed before it reaches the provider
response = litellm.completion(model="gpt-4o", messages=[...])
response = litellm.completion(model="bedrock/claude-sonnet", messages=[...])

Or config:

# litellm_config.yaml
litellm_settings:
  callbacks: ["headroom.integrations.litellm_callback.HeadroomCallback"]

Under the hood it uses LiteLLM’s async_pre_call_hook, same lifecycle slot as the gateway guardrail. Local mode compresses in-process with Headroom’s compress(); cloud mode forwards to Headroom Cloud for managed retrieval/learning.

Your own ASGI proxy or web app — middleware:

from headroom.integrations.asgi import CompressionMiddleware

app.add_middleware(CompressionMiddleware)
# Intercepts /v1/messages, /v1/chat/completions, /v1/responses, /chat/completions

Responses can carry x-headroom-compressed: true and x-headroom-tokens-saved for per-request measurement.

Coding agent only, no Python app — standalone proxy:

pip install "headroom-ai[all]"
headroom proxy --port 8787

# then, for Claude Code:
ANTHROPIC_BASE_URL=http://localhost:8787 claude

Decision rule: LiteLLM proxy → guardrail. LiteLLM-in-code → callback. Own web app → middleware. Single coding agent → standalone proxy.

What to expect, honestly

Savings depend on content

Docs now quote a range: roughly 60–95% fewer tokens on JSON-heavy tool outputs and search results; closer to 15–20% on already-dense coding-agent traffic. Plan on your workload, not the ceiling. Leave a slice uncompressed as a control group so dashboards report measured savings.

Compression adds a little latency

Headroom cites ~15 ms for small JSON and 100–200 ms for large tool blobs. Token reduction usually returns more generation time than compression costs inbound; tiny payloads may not pay. That is why per-request and per-key controls matter.

The old tension remains

Aggressive compression fights complete context. retrieve_headroom is the valve, but it needs the model to notice and ask. It works well in practice without being a guarantee—watch quality on detail-sensitive tasks.

It is new

Gateway integration landed mid-2026 on a very recent LiteLLM line (initially a dev cut). Pilot behind a key, measure, then widen—do not flip global production on day one.

Where it sits in the stack

Useful production AI work keeps moving into layers around the model, consolidating into shared infra teams already run:

Clients / agents (Claude Code, RAG apps, custom agents)
        |
LiteLLM gateway  ── pre_call guardrail ──►  Headroom compression (+ CCR retrieve)
        |
LLM providers (OpenAI / Anthropic / Bedrock etc.)

The standalone compression idea is now a gateway feature. If LiteLLM is already the front door, this is a config change behind a key—not a re-architecture. That is why it is interesting even for teams that skipped a separate compression proxy before.

Takeaway

Headroom on LiteLLM is a small change with a wide blast radius. Enable it at the gateway and every agent/client behind it sends leaner requests across providers, with originals a tool call away.

Design stays conservative where it matters: reversible, fail-open, and toggleable per request, key, or globally so you can start narrow.

If tokens are a real line item and LiteLLM is already in path, trial cost is near zero. Attach one key, measure live traffic, decide from there.

References

1. LiteLLM blog, LiteLLM × Headroom — https://docs.litellm.ai/blog/headroom-integration
2. Headroom LiteLLM integration docs — https://headroom-docs.vercel.app/docs/litellm
3. Headroom on GitHub — https://github.com/chopratejas/headroom
4. LiteLLM documentation — https://docs.litellm.ai/

Rollout playbook

Week one: attach compression to a single virtual key used by a non-critical agent. Compare token counts and answer quality against an uncompressed twin on the same prompts. Confirm x-litellm-applied-guardrails appears and that retrieve calls succeed when you force a detail-heavy question.

Week two: expand to a coding-agent key if measured savings on JSON-heavy traffic looked healthy. Expect smaller percentage wins on dense code contexts; judge success by dollars and latency, not by chasing the 95% marketing ceiling.

Week three: decide global default only if fail-open behavior held during a deliberate Headroom outage drill and quality owners signed off on the most sensitive workflows.

Why this belongs in platform engineering

Application teams should not each invent compression. Gateways already own keys, budgets, and routing. Adding compression there matches how rate limits and auth matured: once in the shared path, every client inherits the control without a redeploy.

Keep the mental model: clients talk to LiteLLM; LiteLLM talks to providers; Headroom sits beside the pre-call hook. Nobody on either edge needs a new SDK. That invisible placement is what turns a research idea into a platform switch.

Decision card

• Already on LiteLLM proxy → native guardrail
• litellm.completion in process → callback
• Custom ASGI → middleware
• One coding agent → standalone Headroom proxy

Measure, then widen. Compression is a cost lever, not a religion.