Home / Articles / Phlox-GW: Open-Source LLM Gateway with Budgets, Guardrails, and HA

This article is published in English.

llm-gateway phlox openai anthropic observability self-hosted

Phlox-GW: Open-Source LLM Gateway with Budgets, Guardrails, and HA

Self-host an LLM gateway with chargebacks, rate limits, PII guardrails, audit logs, OpenAI and Anthropic endpoints, and Postgres-backed clustering.

2581 words

Open-source LLM gateways often ship “enterprise” controls—chargebacks, budgets, SSO, guardrails, audit trails, HA clustering, rate limits, routing—behind a paid license. Phlox-GW (Phlox Gateway) keeps those controls in a fully open-source gateway intended for self-hosted team infrastructure: one consistent front door across providers, with OpenAI and Anthropic endpoints and protocol translation so clients such as Claude Code can talk to models from mixed backends.

Phlox-GW is a single Go binary for macOS, Linux (including WSL), and Windows. Small deployments can run one instance with SQLite; larger ones scale to multi-node clusters on shared PostgreSQL. A sister project, the Phlox AI Platform, can sit beside the gateway when a full chat surface is needed; this guide focuses on the gateway itself.

Phlox-GW Interface Tour

Operations dashboard

Admins see high-level counts—users, providers, API keys, events, total spend—plus thirty-day charts for daily cost, tokens, requests, errors, and average latency.

Cost and budget monitoring

Monthly budgets attach to people and departments. Users carry a department tag so spend rolls up. Crossing a warning threshold notifies; hitting the hard limit blocks paid models until the next cycle or a limit increase. That chargeback loop is a primary reason teams reach for a gateway instead of raw provider keys.

Rate Limits

Limits apply at user, department, provider, or model scope, as requests per minute (RPM) and/or tokens per minute (TPM).

HA and Scaling with Clustering

A single Go process on PostgreSQL already goes far. For availability or thousands of concurrent sessions, add instances sharing one Postgres and put a network load balancer in front with health checks that drain unhealthy nodes.

Auditing

Audit entries capture logins and configuration actions: time, actor, action, target, details, and IP.

Privacy Preserving Request logging

Every gateway call logs metadata—time, request id, user, department, API key, provider, model, protocol, endpoint—without storing prompt or completion bodies, so content stays private while ops still has an incident trail.

Guardrails / PII Redaction and Blocking

Middleware can redact or block messages when sensitive patterns fire, on the way in (stop leaks to providers) and on the way out (stop leaks to clients), depending on configuration.

Self-Service API Keys

Signed-in users mint named keys with optional expiry, revoke them, and see last-used times. Admins get a fleet view to assign budgets/limits and revoke. Full key material appears once at creation.

Self-Service Usage Monitoring

Individuals see their request counts, input/output tokens, spend, and per-model cost breakdown without waiting on a finance export.

Installing Phlox-GW

For workstation evaluation the binary is self-contained and creates a SQLite database on first launch. Building from source needs a current Go toolchain and npm for the UI assets. A typical bootstrap:

curl \
  --proto '=https' \
  --tlsv1.2 \
  -fsSL \
  https://raw.githubusercontent.com/robert-mcdermott/phlox-gw/main/install.sh \
  | sh

Create a data directory and start the service:

  mkdir -p "/Users/<your-username>/.local/share/phlox-gw"
  cd "/Users/<your-username>/.local/share/phlox-gw"
  phlox-gw

Point a browser at the local UI, create the first admin, and continue configuration there. Production installs usually set environment variables for Postgres DSN, listen address, TLS termination at the load balancer, and session secrets—see the repository docs for the full variable list.

Configuring Phlox-GW

There is no mandatory config file: environment variables plus the web UI cover setup.

Adding/configuring a provider

Register each upstream (OpenAI-compatible, Anthropic, or others supported) with base URL and credentials stored by the gateway—not by every laptop.

Adding and configuring a model

Map provider model ids to gateway-facing names, attach pricing for chargebacks, and choose routing/failover peers when multiple backends can serve the same logical model.

Testing a provider & model in the Playground

The built-in playground sends a trial chat through the selected provider/model path so wiring issues show up before clients are pointed at the gateway.

Adding users

Create accounts (or connect SSO/OIDC when enabled), assign roles and departments, and attach budgets/limits.

Creating budgets

Define monthly ceilings and warning thresholds for people and departments; priced models honor those caps at request time.

Using Phlox-GW

Creating an API Key

From the self-service panel, mint a key, copy it once, and store it in the client’s secret store.

Testing the gateway endpoints

Export the key:

export PHLOX_API_KEY="pgw-sk-<rest-of-your-api-key>"

OpenAI-compatible chat completions against the local gateway:

curl -Ns http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $PHLOX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-ollama/glm-5.2:cloud",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "stream": false
  }'

Anthropic-style messages through the translated endpoint:

curl -sS http://127.0.0.1:8080/anthropic/v1/messages \
  -H "x-api-key: $PHLOX_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "local-ollama/glm-5.2:cloud",
    "max_tokens": 64,
    "messages": [{ "role": "user", "content": "What is the capital of Texas?" }]
  }'|jq

Using Claude Code with Phlox-GW

Point Claude Code (or similar) at the Anthropic-compatible base URL and gateway key:

env \
  ANTHROPIC_BASE_URL="http://127.0.0.1:8080/anthropic" \
  ANTHROPIC_API_KEY="$PHLOX_API_KEY" \
  ANTHROPIC_MODEL="azure/gpt-5.5" \
  claude

Protocol translation lets Anthropic-shaped clients reach whatever upstream the gateway routes to.

Checking your usage

Users refresh the usage panels for tokens and spend; spikes should match known batch jobs or runaway agents.

Monitoring usage and spend as an admin

Admins watch fleet dashboards, department rollups, and error/latency charts. Budget breaches and rate-limit hits are first-class operational events, not spreadsheet surprises.

Guardrails - sensitive information redaction

Enable patterns that match secrets, personal identifiers, or internal hostnames. Choose redact versus block per pattern family. Test with synthetic samples in the playground before enforcing on production traffic.

Logging and Auditing

Request Logging

Metadata-only request logs support incident response and chargeback disputes without retaining private prompt text.

Audit Logging

Configuration and auth events answer “who changed routing yesterday?” without spelunking application logs.

Conclusion

Phlox-GW packages chargebacks, budgets, rate limits, guardrails, audit trails, and clustering into a single open binary with OpenAI and Anthropic facades. Start on SQLite for evaluation, move to Postgres and an NLB when availability matters, and keep provider credentials and policy in one place instead of scattered laptop env files.

Operational checklist for a first production cut: (1) price every routed model so budgets mean something, (2) tag users with departments before the first invoice cycle, (3) turn on metadata request logs and audit from day one, (4) put synthetic PII samples through guardrails in the playground, (5) front a two-node Postgres-backed pair with health-checked load balancing before promising HA, and (6) document how Claude Code / SDK clients should set base URL and key so shadow IT does not bypass the gateway with raw provider credentials. Revisit RPM/TPM limits after a week of real agent traffic—the first limits are almost always too generous on bursty tool loops and too tight on interactive chat. Keep a runbook for rotating gateway keys and provider secrets on different calendars so a single leak does not force a dual outage. Finally, export spend by department on a fixed cadence even if nobody asked yet; finance will ask after the first surprising bill, and the gateway already has the numbers if tags were set correctly.

When expanding beyond a single team, treat the gateway as a product surface: version routing policies, review failover peers when a provider has a regional incident, and alert on rising 429s from upstream separately from gateway-enforced limits. Upstream throttling and local policy caps need different responses—buy capacity versus coach a noisy agent. Pair Phlox-GW metrics with provider status pages in the same on-call view so operators do not debug “gateway latency” that is actually a model-region brownout. With those habits, the gateway stays a control plane rather than another opaque proxy.

Document gateway SLOs the same way as any other edge service: availability of the /v1 and /anthropic surfaces, p95 latency excluding upstream model time when possible, and budget-block rates by department. Those three charts catch most “it feels broken” reports before they become Slack threads.

All articles Send the brief