Why Round-Robin Fails LLM Fleets and How llm-d Routes by Cache and Queue

Serving a large language model on one box is a solved enough problem. Serving it to the world is where a different class of trouble starts. This piece covers what breaks when many GPU servers sit behind an ordinary load balancer, and what llm-d does about it.

The foundations, quickly

A few facts from the lower layers explain everything that follows.

Start with the model, which is just a large file of weights. GPUs are the right tool because the arithmetic parallelizes so well. VRAM exists because system RAM is roughly 52 times too slow to feed those cores. Each request splits into prefill, which is compute-bound and produces the initial pause (time to first token), and decode, which is bandwidth-bound and produces the stream (time per output token). In decode, every token costs a full re-read of the whole model.

On top of that hardware sits a software layer:

• The KV cache stores what prefill computed, so decode never has to redo it. That's the reason the initial pause occurs a single time.
• Its footprint is about 128 KB per token on an 8B model. Memory for the cache, not raw compute, therefore limits how many users fit at once. Roughly 60 GB on an H100 holds around 490,000 tokens' worth.
• PagedAttention treats that pool the way an operating system treats virtual memory. Fragmentation disappears, and a prefix shared by several requests is kept only once.
• Prefix caching lets blocks outlive the request that made them, keyed by a hash of their content. By turn ten of a chat, the work is about the same as it was at turn two.
• Continuous batching shares the cost of reading weights among everyone waiting. The KV read can't be shared, since it belongs to each sequence and recurs at every step. Throughput can climb tenfold or more, while per-user latency grows with context length: about 4.8 ms per token alone, and about 22 ms when the cache pool is full.
• Tensor, pipeline and expert parallelism divide one model among many GPUs when a single card can't hold it.

All of those optimisations share a catch: every one of them is per-server state.

Q: What goes wrong on a fleet?

A real service can't live on a single machine. Serving the whole world means many servers, each with its own GPUs and a complete copy of the model, with a load balancer in front:

[ your message ] ──▶ [ load balancer ] ──▶ [ Server 2 ]
                       round-robin      ──▶ [ Server 5 ]
                                        ──▶ [ Server 9 ]

That's the most standard, boring and well-understood piece of infrastructure around, and it has worked flawlessly for web services for twenty-five years. For LLM serving, it fails in two separate ways.

Failure 1: the cache gets scattered

Turn 1 of a conversation arrives and round-robin sends it to Server 2. That server prefills the thread, builds a KV cache, streams the answer, and, thanks to prefix caching, keeps the blocks.

Then turn 2 comes in, and round-robin routes it to Server 5. That machine has never seen this conversation, so the full thread gets prefilled again from zero. Server 2's usable cache stays untouched, paid for and idle until something evicts it.

Across a randomly balanced fleet, the prefix cache hit rate is close to zero. The most valuable optimisation in the stack gets thrown away at the routing layer.

Failure 2: every request looks equally expensive

Round-robin assumes that every request costs about the same, in time and in memory. That assumption comes from ordinary web serving, where it's close enough to true. Fetching a profile page, rendering a template and running an indexed query each finish within tens of milliseconds and hold a few kilobytes meanwhile. When all requests are that small and uniform, which server gets which one barely matters, and counting connections is a fine way to spread load.

An LLM breaks both halves of that assumption together:

• one user types a short question of about 40 tokens
• a second user drops in a 60-page PDF worth roughly 50,000 tokens

That's three orders of magnitude of difference in compute. Because the KV cache scales with tokens, memory differs by about the same factor, so one request holds a slice of GPU memory roughly 1,250 times larger than the other, for much longer.

Round-robin hands each of them a single slot. Worse, requests queue behind whichever arrived first. One machine can be buried under three giant prefills while its neighbour idles, and the balancer, which rotates blindly or at best counts open connections, sees a perfectly even spread.

The llm-d docs say generic load balancers ignore prompt structure, token count, queue depth and GPU load. At fleet scale that isn't a small inefficiency. It swamps whatever else was tuned.

Q: What would an LLM-aware router do?

It would stop asking whose turn it is and start asking which replica suits this particular request right now.

This is the problem llm-d targets. It's a distributed inference stack built natively for Kubernetes, hosted as a CNCF Sandbox project and started by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. vLLM serves as the engine, and llm-d coordinates many of them.

Requests enter through an Inference Gateway, made of a proxy and an Endpoint Picker (EPP), the part that decides where each request goes. The EPP rates every candidate replica using signals that a normal load balancer lacks:

• Prefix-cache overlap: the share of this prompt the replica already has cached
• Queue depth: the amount of work already waiting on it
• KV-cache utilisation: whether it's close to full and about to begin evicting
• Predicted latency: an optional XGBoost model in llm-d that estimates the time this request would take there

A three-replica example shows the trade-offs. Replica B is entirely idle yet holds nothing relevant, so choosing it forces a full cold prefill. Replica C matches 88% of the prefix, but 31 requests are ahead in line, which means more time lost waiting than saved. Replica A matches 92% with only 2 queued, and it wins. A round-robin balancer can't reach that verdict, because the verdict depends on what each server contains.

Q: How does the router know what's cached?

Two designs exist, and they trade accuracy against simplicity.

Approximate. The router chops each prompt into blocks of fixed size and chains rolling hashes over them. It remembers, in an in-memory LRU index, which prefix hashes it has sent to which pod. This design is very light, needing no tokenizer or extra service. The downside is that it guesses at server state from its own record, and the guess drifts whenever a replica evicts blocks without telling the router.

Precise. Here the router obtains exact tokenization from vLLM's render endpoint, while vLLM publishes KV-cache events over ZMQ each time a block is created or evicted. The router then holds a globally correct map of token blocks and their locations. Accuracy is exact, and complicated eviction patterns and disaggregated deployments are handled properly, though more components are involved.

The reported payoff for getting this right is about 3x higher output throughput and 2x faster time to first token. It doesn't come from new hardware. It comes from sending requests to the machine that already did the work.

Q: Is it a problem that prefill and decode share a GPU?

Yes, a serious one, and it follows from the basics above. Prefill craves compute and arrives in bursts, while decode craves bandwidth and runs for a long time. Put on the same GPU, the two compete.

Users feel the failure directly. Imagine fifty people in the middle of answers, all receiving tokens smoothly. Then a prompt of fifty thousand tokens arrives for prefill and monopolises the GPU for hundreds of milliseconds, and all fifty streams stall. One heavy request degrades everybody, and the batching scheduler can't help, because both jobs genuinely need the same silicon.

There's also a resource-shape mismatch. GPUs sized for prefill leave bandwidth over-provisioned, and GPUs sized for decode run short on FLOPs. A single pool can't be right for both.

The fix: two pools for two opposite jobs

Disaggregation splits the fleet in two:

• The prefill pool is sized for compute. It reads prompts, builds KV caches and passes them on.
• The decode pool is sized for memory bandwidth. It takes in caches and streams tokens in large batches, free of prefill interruptions.

The two pools have to exchange the KV cache, which means copying it from one GPU's memory into another's. NIXL, the NVIDIA Inference Xfer Library, handles that by moving cache tensors across RDMA or InfiniBand, NVLink, NVMe or TCP at full wire speed. Practicality hinges on this transport layer. A slower handoff would cost more than recomputing the cache.

The reward is that the two pools scale independently:

Workload	Typical examples	Add capacity to
Long prompts, short answers	Summarisation, RAG, code review	Prefill GPUs
Short prompts, long answers	Chat, agents	Decode GPUs

llm-d reports up to 70% more tokens per second from this split alone, and disaggregation is often described as an immediate ~25% gain in typical deployments.

Q: What else lives at the fleet layer?

With an inference-aware router and specialised pools, several more things become possible.

Tiered KV offload. VRAM is the scarcest asset, so cache blocks not currently in use can drop to CPU RAM and then to local SSD, returning when a matching request appears. That tier is far slower than VRAM, yet it beats repeating prefill by a wide margin. For cache-heavy workloads, llm-d reports as much as 13.9x throughput from hierarchical offload.

SLO-aware autoscaling. Kubernetes by default scales on GPU utilisation, which means almost nothing here, since a GPU can be 100% busy doing wasteful recomputation. llm-d scales on what was actually promised: targets for time to first token and inter-token latency. It can also scale the prefill and decode pools separately, which is possible only because they're disaggregated.

Wide expert parallelism. For Mixture-of-Experts models, experts spread over many GPUs so that only those a token routes to do any work. That's what makes 671B-parameter models servable, and llm-d cites about 50k tokens per second of cluster throughput on this path.

Flow control. Multi-tenant serving needs fairness so that a batch job from one customer can't starve another customer's interactive traffic. The gateway is where priorities have to be applied, since a request that has already reached a GPU is beyond help.

A caution on the numbers. All of these figures come from the project's own v0.7 benchmarks, run on "well-lit paths", which are tested reference setups. Treat them as upper bounds under good, well-tuned conditions and not as promises. Real results track the shape of the traffic. With no shared prefixes among prompts, cache-aware routing gains nothing, so it's worth measuring a real workload first.

Q: Is any of this needed?

Almost certainly not at the start. A sensible order exists, and jumping ahead costs more than it saves. llm-d itself describes its features as composable, meant to be adopted one by one as particular bottlenecks show up and not installed as a bundle.

That holds generally. Each layer discussed here was created to fix a measured problem in the one beneath it. Taking on a layer without having its problem only adds operational burden.

Measure first, because the bottleneck rarely sits where it's assumed to.

The whole stack as a chain of consequences

1. The GPU is there because a forward pass is billions of independent multiplications, and a CPU grinds through them serially.
2. VRAM is there because system RAM runs about 52x too slow to supply thousands of cores.
3. VRAM is also small, which turns memory into the scarce resource that limits all the layers above.
4. vLLM is there because a PyTorch script on its own isn't a service.
5. The KV cache is there because redoing prefill for each token would stall before every word.
6. PagedAttention is there because naive KV allocation wastes most of the memory.
7. Prefix caching is there because a chat resends the entire thread every turn.
8. Continuous batching is there because a single weight read ought to serve everyone waiting, not one person.
9. Parallelism is there because the model outgrew the card.
10. llm-d is there because round-robin balancing throws away item 7 and undermines item 8 once a second server joins.

What happens when you press enter in a chat app

The message is appended to the thread and sent to an inference gateway. A router hashes the prefix, looks up which replicas already hold that conversation, compares their queue depths and cache pressure, and selects one.

The chosen replica finds most of the thread cached from the last turn, so only the newest message needs prefill. If the fleet is split, that work lands on a GPU pool reserved for prefill, and the resulting KV cache then travels over RDMA to a decode pool.

At the decode pool, the request joins a batch with dozens of strangers. Every few milliseconds the whole model is pulled out of VRAM exactly once, and that one read yields the next token for all of them together. The cache grows by 128 KB, the batch gets rebuilt, and the cycle repeats.

A few hundred times over, and words appear on the screen.

Behind that answer sits a solution that's unreasonably hard to build.

Where to read more

• the llm-d repository, home of the reference paths and the v0.7 benchmark results
• the llm-d project site, covering architecture docs
• the vLLM documentation for the design notes on its prefix cache
• the NIXL repository, which hosts the KV transfer library