Continuous batching and chunked prefill in vLLM and SGLang

Classical request-level batching leaves a large share of GPU cycles on padding when language-model outputs vary wildly in length. Iteration-level continuous batching—popularized in production by vLLM and SGLang after the Orca research line—admits and evicts sequences on every token step. Chunked prefill then stops large prompt ingestion from freezing streams that are already decoding. Platform teams that still ship static Hugging Face pipelines feel both problems as soon as traffic mixes short chat turns with long generations.

1. The static batching trap

Vision and tabular servers batch fixed tensors—sixteen images of shape [16, 3, 224, 224], one forward pass, sixteen answers together. Copy that pattern onto large language models and throughput collapses for two structural reasons:

• Unpredictable output lengths. One request may finish in a few hundred milliseconds; another in the same batch may stream hundreds of tokens for many seconds.
• Head-of-line blocking. Static batches cannot return the short answer early. Its slot sits filled with pad tokens while the GPU waits on the longest peer.

In static batching, the slowest, wordiest request sets the pace for everyone else. Continuous batching treats the batch like a runway: finished sequences leave immediately and waiting ones take the free slot. The padding tax is not a minor inefficiency—it routinely wastes a majority of theoretical FLOPs on empty work when length distributions are heavy-tailed, which is the default for chat and coding assistants.

2. How continuous batching (iteration-level scheduling) works

Rather than schedule a whole request end to end, the engine schedules one token iteration:

1. Token forward step. One forward pass across every active sequence emits exactly one new token each.
2. Instant eviction. Sequences that produced <eos> are returned and freed from VRAM.
3. Instant admission. The freed capacity goes to a queued request without waiting for the rest of the old batch.

Operating at iteration granularity keeps cores on real work instead of pad zeros and commonly delivers multi-fold throughput gains at comparable median latency. The scheduler’s unit of work shrinks from “entire completion” to “one step for every live sequence,” which is why memory managers such as paged attention pair naturally with continuous batching: slots appear and disappear without reshaping a rigid static tensor.

3. The prefill hitch and chunked prefill

Continuous decode exposes a new conflict between prefill and decode:

• Prefill is compute-bound: the model ingests the full prompt to build the KV cache. A multi-thousand-token prompt can occupy hundreds of milliseconds of heavy matrix work.
• Decode is memory-bandwidth bound: one-token steps on the order of tens of milliseconds dominate interactive feel.

A naive scheduler that pauses thirty live decodes to ingest one huge prompt causes stream stutter—the classic prefill hitch. Chunked prefill slices that prompt (for example into 512-token pieces) and interleaves chunks with decode steps so prompt work spreads across iterations without sharp latency spikes. Users already streaming keep receiving tokens; newcomers still make progress on prompt ingestion. The same idea applies when RAG injects long contexts mid-session: without chunking, a single retrieval dump can freeze dozens of neighbors.

4. Performance and throughput comparison

These ranges are directional; exact numbers depend on model size, hardware, and prompt mix. The qualitative ranking holds across most public serving write-ups: static batching loses on utilization, continuous alone fixes decode waste but not prefill interference, and chunked prefill restores smooth streaming under mixed load.

5. Simulating an iteration-level scheduler in Python

The following sketch shows admission, chunked prefills, and eviction on every step. It is a teaching model, not a production runtime, but it makes the control loop explicit:

from dataclasses import dataclass
from typing import List


@dataclass
class LLMRequest:
    request_id: str
    remaining_prefill: int
    tokens_to_generate: int
    tokens_generated: int = 0

    @property
    def is_finished(self) -> bool:
        return self.remaining_prefill == 0 and self.tokens_generated >= self.tokens_to_generate


class ContinuousBatchEngine:
    """
    Simulates iteration-level scheduling with dynamic slot allocation
    and chunked prefill budgeting.
    """
    def __init__(self, max_batch_slots: int = 3, prefill_chunk_size: int = 256):
        self.max_slots = max_batch_slots
        self.chunk_size = prefill_chunk_size
        self.waiting_queue: List[LLMRequest] = []
        self.running_slots: List[LLMRequest] = []

    def submit_request(self, request: LLMRequest):
        self.waiting_queue.append(request)

    def step_iteration(self, step_number: int):
        print(f"\n--- [ITERATION STEP {step_number}] ---")

        # 1. Admit waiting requests if open slots exist
        while len(self.running_slots) < self.max_slots and self.waiting_queue:
            new_req = self.waiting_queue.pop(0)
            self.running_slots.append(new_req)
            print(f"  [ADMITTED] {new_req.request_id} entered running batch.")

        # 2. Execute one token step across running batch
        for req in self.running_slots:
            if req.remaining_prefill > 0:
                # Process a chunk of prefill
                chunk = min(req.remaining_prefill, self.chunk_size)
                req.remaining_prefill -= chunk
                print(f"  [PREFILL CHUNK] {req.request_id}: Processed {chunk} prompt tokens (Remaining: {req.remaining_prefill})")
            else:
                # Process single decode token
                req.tokens_generated += 1
                print(f"  [DECODE TOKEN] {req.request_id}: Emitted token #{req.tokens_generated}/{req.tokens_to_generate}")

        # 3. Evict finished requests immediately
        finished = [r for r in self.running_slots if r.is_finished]
        for req in finished:
            self.running_slots.remove(req)
            print(f"  [COMPLETED & EVICTED] {req.request_id} returned to client.")


if __name__ == "__main__":
    engine = ContinuousBatchEngine(max_batch_slots=2, prefill_chunk_size=200)

    # Simulate 3 incoming requests of varying prompt & response sizes
    engine.submit_request(LLMRequest("Req_A (Short)", remaining_prefill=0, tokens_to_generate=2))
    engine.submit_request(LLMRequest("Req_B (Long)", remaining_prefill=0, tokens_to_generate=4))
    engine.submit_request(LLMRequest("Req_C (Large Prompt)", remaining_prefill=400, tokens_to_generate=2))

    # Run 5 continuous iterations
    for step in range(1, 6):
        engine.step_iteration(step)

6. Practical takeaways for AI platform engineers

• Do not ship naive Hugging Face pipelines as production servers. Default transformers.pipeline static batching is a lab pattern. Wrap open-weight models in continuous-batching runtimes such as vLLM or SGLang.
• Enable chunked prefill on mixed chat loads. In vLLM, flags such as --enable-chunked-prefill cap tokens per iteration so long system prompts or RAG contexts do not hitch streaming users.
• Watch p95/p99, not only averages. Average tokens per second can look healthy while prefill scheduling still jitters individual sessions. Instrument time-to-first-token separately from inter-token latency so queue wait and decode health do not collapse into one misleading number.
• Budget KV memory with the scheduler in mind. Continuous batching only helps if unfinished sequences can leave cleanly and new ones can enter; oversubscribed VRAM turns admission into thrashing.

Taken together, continuous batching and chunked prefill are the default posture for modern open-weight serving—not optional tuning knobs after a static design fails in production.