vllm prometheus observability llmops gpu ttft

vLLM Inference Metrics: From TTFT and TPOT to KV Cache and Goodput

A production guide to vLLM Prometheus metrics—success rates, latency percentiles, token IO ratio, preemptions, chunked prefill, DPD, and speculative decoding acceptance.

Publisher: REACTAPP.TOPAdded to this site Sep 26, 202613 min read2973 words

This article is published in English.

Sources: A Deep Dive into vLLM Inference Metrics

Serving large language models is not the same observability problem as serving stateless microservices. CPU percent and average request latency are weak proxies when attention, continuous batching, and HBM capacity dominate behavior. Production focus has moved from raw token throughput toward goodput—successful completions inside strict SLOs.

This guide walks through the core the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the vLLM engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine engine metrics that belong on Grafana boards: reliability counters, latency histograms, token economics, KV-cache dynamics, scheduler pressure, and topology choices that show up only when telemetry is read correctly.

Summarized Statistics and System Health

LLM engines often fail quietly—degraded generations and hidden preemptions—rather than with clean HTTP errors. Top-line health therefore needs semantic finish reasons, not only status codes.

Success Rate and Finish Reason Distribution

Success rate is completed successes over all processed requests in a window. In vLLM, that means inspecting how the generation loop ended, not merely counting 200 OK. Healthy fleets usually stay at or above ~99.5% success; sustained drops under ~98% are emergency territory.

The counter is vllm:request_success_total, labeled by finished_reason:

Stop — natural EOS or configured stop string. Treat as true success.
Length — truncated at max context or max generation.
Abort — client disconnect or internal abort.

Chronic length finishes often mean the model rambles for the task, or max_model_len / generation caps are too tight for the prompt style. A typical PromQL framing is the rate of stop finishes over the rate of all finishes in a five-minute window.

Throughput vs. Success Convergence

Plot total request throughput beside successful completions. Under healthy load the curves should track. A widening gap means GPUs burn cycles on work that never lands as success—timeouts (aborts) or context breaches (length). Under load tests, large gaps flag attention degradation or non-terminating generations that race to length limits.

Python Garbage Collection and Process Memory

Much of the hot path lives in CUDA/C++, but orchestration is still Python. Track GC counts with process RSS.

Frequent GC injects jitter into inter-token timing. A steadily climbing RSS hints at leaks in bindings and eventually invites the OOM killer. Pods can look “fine” on CPU whether idle or busy; unmanaged memory pools are where silent death appears.

Quantifying Latency and User Experience

Generative latency is long-tailed. Averages hide pain; P50/P90/P99 (and friends) are mandatory.

End-to-End Request Latency

E2E spans arrival to final token and close. vLLM exposes vllm:e2e_request_latency_seconds as a histogram. Quantiles use histogram_quantile() with linear interpolation inside buckets. Erratic P99 spikes often mean the continuous batcher cannot interleave arrivals cleanly—time to hunt scheduler and cache pressure.

Time to First Token (TTFT) P99

TTFT is the interactivity metric. HCI guidance aims under about one second before frustration rises.

TTFT is dominated by prefill: read the prompt, build attention, materialize the initial KV cache. Cost scales roughly with sequence length squared times width—compute-bound TFLOPS work. Example scale: a 512-token prefill on a ~70B model is on the order of tens of TFLOPs.

PromQL for TTFT P99:

histogram_quantile(0.99, sum(rate(vllm:time_to_first_token_seconds_bucket[5m])) by (le))

For chat UIs, TTFT P99 is often the binding constraint. Huge hidden sizes on mid-tier GPUs (for example L40S-class) can make TTFT unacceptable regardless of batching—pushing teams toward smaller models or higher-TFLOPS silicon (H100-class).

Time Per Output Token (TPOT) and Inference Stage Breakdown

After the first token, decode takes over. TPOT (inter-token latency) measures subsequent token pace.

Decode is usually memory-bandwidth bound: each step reloads weights and KV history from HBM into compute with low arithmetic intensity (~1 FLOP/byte for dense non-MoE paths).

A useful decomposition:

E2E ≈ TTFT + (N_output − 1) × TPOT

Stage breakdown charts show prefill share versus decode share:

High prefill time → long prompts or weak TFLOPS.
High decode time → long outputs or saturated HBM bandwidth.

Comparing TTFT against TPOT under rising load is the fastest way to name the bottleneck.

Token Throughput and Workload Characterization

Latency without volume cannot price hardware.

Token Throughput and the IO Ratio

Split throughput into prompt tokens/sec and generation tokens/sec. Higher sustained totals amortize GPU rent better.

The token IO ratio (prompt tokens over generation tokens) characterizes work:

High (~20:1) — long inputs, short outputs: RAG, summarization, classification. Prefill-heavy.
Low — short prompts, long generations: open chat, code gen. Decode/HBM-heavy.

Mismatching topology to ratio wastes money: RAG on decode-optimized pools, or long-form chat on prefill-starved boards.

Request Length Heatmap (Prompt)

Prompt-length heatmaps show whether traffic is a calm unimodal distribution or a mix of tiny and huge prompts. Heavy tails in prompt length explain TTFT bursts even when QPS looks modest.

Engine Internals and Cache Dynamics

External latency is downstream of KV-cache and scheduler state.

GPU KV Cache Usage

Track how full the KV pool is. As occupancy climbs, continuous batching admits fewer sequences; past a threshold the engine preempts.

Prefix Cache Hit Rate and Savings

Prefix caching reuses KV for shared prompt prefixes (system prompts, RAG headers). Hit rate and saved prefill compute quantify whether prompt design and routing actually share prefixes—or accidentally uniquify every request.

Scheduler State and System Efficiency

Watch running versus waiting counts. Rising wait with flat running usually means KV or max_num_seqs caps, not “CPU busy.”

Scheduler State and Preemption Rate

When KV cannot store the next token’s vectors, vLLM preempts. vllm:num_preemptions_total is a red-line counter: nonzero sustained preemption demands more VRAM, more replicas, or a lower max_num_seqs ceiling.

System Efficiency and Request Queue Time

System efficiency compares processing pace to arrival pace. Near 1.0 means arrivals are handled promptly; values drifting under ~0.7 mean the wait queue grows faster than GPUs drain it.

That pain appears as vllm:request_queue_time_seconds—time stuck waiting before useful compute. Queue time inflates E2E with zero tokens produced.

Classic CPU-based HPA fails here. Wire vllm:num_requests_waiting and queue-time metrics through a Prometheus adapter so HPA scales on queue depth (for example more than a handful waiting) instead of CPU mirages.

Infrastructure, Load Balancing, and Advanced Topologies

Software signals must meet hardware truth.

Load Balancing and Hardware Utilization

LLM request durations vary wildly—minutes for long generations versus instant short replies. Round-robin therefore imbalances fleets. Prefer balancers aware of in-flight depth.

Cross-check with DCGM:

GPU utilization — high utilization with low queues is desired ROI; high utilization with deep queues is saturation.
VRAM used — confirms vLLM reserved the intended fraction at start.

Resolving the Prefill-Decode Conflict

A large compute-bound prefill inside a mixed batch can monopolize tensor cores and stall every ongoing decode—spiky TPOT and stuttering streams.

Two mitigations dominate: chunked prefill and disaggregated prefill/decode.

Chunked Prefill

Split huge prompts into fixed chunks (for example 512 tokens) and interleave chunks with other requests’ decode steps. TTFT for the newcomer rises slightly; everyone else’s TPOT flattens. Metric signature: smoother ITL histograms, modest P50 TTFT cost, large P99 stream stability gains.

Disaggregated Prefill and Decode (DPD)

At larger scale, isolate phases on different pools: prefill on TFLOPS-heavy GPUs, decode on HBM-heavy GPUs, moving KV across the network after prefill. Watch NIXL (or equivalent) transfer timers and bytes—for example vllm:nixl_xfer_time_seconds_sum and vllm:nixl_bytes_transferred_sum. DPD only wins when transfer time is less than the interference it removes; RDMA/NCCL quality decides that math.

Speculative Decoding Mathematics

Speculative decoding speeds memory-bound decode with a small draft model proposing tokens and a target model verifying many in one pass. Verification of several tokens can cost roughly one generate step of compute, beating HBM limits when drafts are good.

Key gauge: draft acceptance rate (vllm:spec_decode_draft_acceptance_rate). Expected accepted tokens per step follow a geometric series in acceptance probability α. If α falls under ~0.5, draft overhead can exceed savings and slow the system.

Evaluate drafts at production temperature. Greedy (T=0) benches can show α≈0.8 while creative T=1.0 traffic collapses the same pair toward α≈0.4—turning an optimization into a regression.

Prometheus Multi-process Telemetry Architecture

vLLM’s OpenAI-compatible server uses multiprocessing. Naive Prometheus clients break across workers (GIL and split memory).

The fix is prometheus_client.multiprocess with PROMETHEUS_MULTIPROC_DIR: each worker appends deltas to memory-mapped files; scrapes hit a MultiProcessCollector that merges OpenMetrics output. Heavy PromQL (histogram_quantile, rate) stays on Prometheus, not on the inference hot path—so measuring latency does not become the latency.

Taken together, these signals—success and finish reasons, TTFT/TPOT/E2E quantiles, token IO shape, KV occupancy, preemptions, queue time, plus topology-specific transfer and speculative stats—give a closed loop from user SLOs to GPU physics. That loop is what turns vLLM from a demo server into an operable inference platform.

Keep a written playbook that maps each red-line metric to a first action: rising preemptions → cut max_num_seqs or add VRAM; TTFT P99 only → shorten prompts, raise TFLOPS, or enable prefix cache; TPOT spikes with full KV → decode pressure or prefill interference; speculative α collapse → retune draft/temperature. Pair every capacity change with a before/after goodput chart on the same trace, so hardware spend ties to SLO wins rather than vanity tokens/sec. Revisit IO-ratio dashboards whenever product mix shifts—RAG launches and long-form features move bottlenecks between prefill and decode overnight, and yesterday’s autoscaling thresholds become today’s silent queue. Finally, treat multiprocess metric directories as part of pod lifecycle: empty or permission-broken PROMETHEUS_MULTIPROC_DIR paths look like “healthy” scrapes with zeros while users already feel the outage.

When onboarding a new model checkpoint, freeze a load profile (prompt-length mix, output-length mix, concurrency) and compare the same PromQL panels before promoting traffic. A checkpoint that wins on mean tokens/sec but loses on TTFT P99 or preemption rate is not a promotion candidate for interactive APIs.

← All articles Send the brief