LLM inference optimization: prefill, decode, and enterprise LLMOps

Designing high-performance, cost-efficient LLM serving means designing around real hardware limits—not wrapping an API and hoping GPUs stay busy.

Early enterprise generative AI budgets concentrated on training and fine-tuning. Once applications leave the lab, the bill shifts: ongoing GPU-bound inference and latency spikes that refuse to stay predictable.

Scale forces physical and financial trade-offs. Fast, thrifty systems look past thin wrappers and into how inference actually runs on silicon.

1. Two phases, two bottlenecks

Every inference request splits into prefill and decode. Each phase hits a different hardware wall.

Prefill (usually compute-bound)

Prefill ingests all prompt tokens at once and builds key-value (KV) activations in parallel. Long enough prompts make large matrix multiplies saturate compute, so the phase is compute-bound. Short prompts or tiny batches can flip the other way: not enough arithmetic to hide memory traffic, so prefill becomes memory-bound. Prefill time is the user’s first wait.

Decode (usually memory-bandwidth-bound)

After the prompt is in, tokens arrive one by one. Each step pulls full model weights and the growing KV history from High Bandwidth Memory (HBM) into GPU SRAM. At low batch sizes that transfer repeats so often the GPU waits on bandwidth instead of compute.

One nuance drives the rest of the design. Growing batch size amortizes the same weight fetch across many sequences, and decode drifts back toward compute-bound. That crossover is why continuous batching exists: it pushes decode into the regime where arithmetic units stay busy.

Latency equation

Total request latency splits into prefill and decode pieces:

T_total  =  TTFT  +  (N_tokens − 1) × TPOT

• TTFT (time to first token) covers the full prefill of the prompt plus the first output token—the user’s sense of initial responsiveness.
• TPOT (time per output token), or inter-token latency, is the steady cost of each later decode step.
• N_tokens is the completion length.

The (N − 1) factor is intentional: the first token already sits inside TTFT, so only the rest multiply by TPOT. Mixing raw prefill compute with TTFT is a common mistake. TTFT is user-facing and must include that first decode step.

2. Ingress before GPUs see traffic

Peak traffic will exhaust GPUs unless ingress sanitizes, evaluates, and filters work first. A typical path: API gateway → multi-layer semantic cache → on miss, an intelligent router that scores complexity and sends work to a commodity-model queue or a frontier-model queue.

Core pieces:

• Multi-layer semantic caching: exact-match key-value plus vector similarity with cos θ ≥ τ. Repeats never touch the model; cost and latency drop together.
• Intelligent model routing: a light classifier sends classification/formatting to small models and reserves frontier models for hard reasoning or multi-step tool use.

3. Execution engine: continuous batching and memory flow

After routing, work enters the execution engine. Static batching wastes capacity: the whole batch waits on the longest sequence. Production stacks use continuous batching (iteration-level scheduling) so slots stay packed and finished sequences leave as soon as they emit end-of-sequence.

Continuous batching is not only about throughput. It also walks decode out of the memory-bound regime into the compute-bound one, so the GPU does arithmetic instead of waiting on HBM.

4. Hitting the phase bottlenecks directly

Caching, routing, and batching manage traffic. The next levers reshape the phases themselves.

Prefill: FlashAttention

Prefill cost grows with the square of prompt length when classic attention materializes a full N×N score matrix in HBM. FlashAttention is an IO-aware tiled kernel that computes exact attention while streaming tiles through on-chip SRAM, cutting HBM traffic without approximating the math. Exact output, shorter TTFT on long prompts. It is default in most serving stacks and pairs with chunked prefill and disaggregation below.

Quantization

Decode pays for shipping weights HBM→SRAM. Quantization shrinks weights—FP16 down to FP8, INT8, or 4-bit (AWQ, GPTQ)—so each token moves fewer bytes. Effective bandwidth rises and more sequences fit in memory. Expect a modest accuracy trade that you must prove on your own evals.

Paged KV cache (PagedAttention)

Historical keys/values expand token by token and dominate live memory use. Contiguous allocation fragments and forces over-provisioning. PagedAttention stores KV in fixed-size blocks, like OS virtual memory, killing fragmentation and unlocking higher batch sizes on the same hardware. Pair it with continuous batching for high concurrency.

Speculative decoding

Sequential, memory-bound decode leaves compute idle between fetches. Speculative decoding spends that slack: a tiny proposer invents a short token run; the main network checks that run in one joint forward. Accepted tokens cost roughly one large-model step.

Correctness is preserved: matching prefixes stick; the first mismatch truncates and the target resamples from a corrected distribution. Under greedy decoding the tokens match the target model exactly; under sampling the distribution matches statistically. Speedup tracks draft acceptance rate, so draft quality matters.

5. Scheduling both phases: chunked prefill and disaggregation

Prefill and decode want opposite silicon. Sharing one GPU pool lets a long prefill monopolize compute and spike every other request’s inter-token latency. Two complementary fixes follow.

Chunked prefill

Instead of one giant prefill burst, split the prompt into segments and interleave (“piggyback”) them with decode steps in the same batch (Sarathi / Sarathi-Serve). That softens TTFT spikes for neighbors and mixes compute-bound and memory-bound work. Continuous batching picks which requests share a step; chunked prefill decides how a heavy prefill enters without starving decode.

Disaggregated serving

Chunked prefill reduces interference; disaggregation removes it by putting phases on different hardware (DistServe, Splitwise). Prefill lands on compute-optimized pools, decode on bandwidth-optimized pools, with KV shipped over the interconnect. Each phase scales on silicon matched to its bottleneck.

Trade-offs are real: KV transfer becomes a new bottleneck, and weights must live in both pools. Payoff shows at scale where phase-specific provisioning beats that overhead. Smaller sites often stop at chunked prefill alone.

6. Systemic impact and trade-offs

Production patterns trade performance for operational risk and infra cost. Numbers shift with hardware, model, traffic, and config—benchmark your own workload instead of copying generic percentages.

Multi-layer semantic caching — exact KV plus vector similarity. Hits avoid the model entirely. Cost: vector search (single-digit to low-tens of ms) and stale or near-miss answers if τ is too low.

Intelligent routing — complexity classifier steers simple work to small models. Lowers average token cost; misroutes hurt quality.

Continuous batching — iteration-level join mid-generation. Higher utilization and throughput under concurrency; individual requests may queue during assembly. Static batching still fits offline throughput jobs.

Quantization — lower-precision weights shrink HBM→SRAM transfers. More concurrency per GPU; validate accuracy; kernel support varies.

Paged KV — fixed blocks remove fragmentation. Higher batch sizes; needs block management and a compatible attention kernel.

Speculative decoding — small proposer suggests; large model checks jointly. Multiple tokens per large step; second model and acceptance-rate sensitivity.

When you need exact latency or cost claims, pull them from reproducible benchmarks of the target workload (published serving studies or internal load tests), not fixed marketing percentages.

7. Closing the loop

Prototype to production means designing for hardware. Decouple prefill from decode, watch how batch size moves decode between memory- and compute-bound regimes, cache predictable queries, route easy work to small models, and pack tokens with continuous batching. Then attack the decode wall with quantization, paged KV, and speculative decoding, and calm phase interference with chunked prefill and disaggregation. Together those pieces absorb peaks without exhausting GPUs.

Capacity planning checklist

When TTFT climbs, inspect prompt length distributions, semantic-cache hit rate, FlashAttention availability, and whether long prompts still monopolize the GPU as one prefill burst. When TPOT climbs under concurrency, inspect effective batch size, KV residency, quantization level, and whether decode slipped back into the memory-bound regime.

A practical weekly review asks three questions: Are we caching predictable queries? Are we routing trivial work away from frontier models? Are we packing decode so GPUs do arithmetic instead of waiting on HBM? Affirmative answers usually beat buying another rack before the serving stack is tuned.

Disaggregation belongs on the roadmap only after chunked prefill and continuous batching already earn their keep. Interconnect traffic and duplicated weights are real costs; pay them when phase-specific pools clearly outperform a shared fleet under your mix.

Document the measured crossover batch size where decode becomes compute-bound on your hardware. That single number anchors continuous-batching targets better than any generic blog chart.

Operator notes

Semantic caches need TTL and τ reviews; too-low τ serves near-miss answers. Routers need labeled complexity data or they mis-send hard prompts to tiny models. Continuous batching needs queue latency SLOs so “higher throughput” does not hide interactive pain. Speculative decoding needs draft acceptance dashboards—if acceptance collapses, you paid for a second model with no speedup.

Treat papers as mechanism proofs, not portable percentage promises. Reproduce on your GPUs, your prompt lengths, and your concurrency before you claim savings to finance.

References

Primary papers behind the mechanisms (numbers are theirs, not yours):

1. Orca distributed transformer serving — Yu et al., OSDI 2022 (USENIX).
2. PagedAttention memory management — Kwon et al., SOSP 2023 (arXiv:2309.06180).
3. GPTQ post-training quantization — Frantar et al., 2022 (arXiv:2210.17323).
4. AWQ activation-aware weight quantization — Lin et al., 2023 (arXiv:2306.00978).
5. FlashAttention IO-aware exact attention — Dao et al., NeurIPS 2022 (arXiv:2205.14135).
6. Speculative decoding for fast transformer inference — Leviathan, Kalman, Matias, ICML 2023 (arXiv:2211.17192).
7. Sarathi / Sarathi-Serve chunked prefills with decode piggybacking — Agrawal et al., 2023–2024 (arXiv:2308.16369).
8. DistServe prefill/decode disaggregation — Zhong et al., OSDI 2024 (USENIX).
9. Splitwise phase splitting for generative inference — Patel et al., ISCA 2024 (arXiv:2311.18677).