LLM Inference End to End: Prefill, Decode, KV Cache, and Sampling

From keystroke to streamed token

This guide walks the full inference path — tokenizer, embeddings, attention, KV cache, sampling, speculative decoding — and answers the hard questions at each stage. The goal is a whiteboard-ready mental model, not a vendor pitch.

Part 1 — Two workloads, not one

Q1. What happens between Enter and the first word?

An honest end-to-end trace:

1. Input. A UTF-8 string such as "What is gravity?" arrives. The serving stack usually wraps it in a chat template with special tokens for system, user, and assistant turns.
2. Tokenization. The text is cut into subword pieces and mapped to integer IDs. That step is pure string processing — no neural network yet.
3. Embedding lookup. Each ID selects a row from the embedding matrix E ∈ ℝ^(V × d_model), yielding a (seq_len, d_model) tensor. Positional signal is added here or later inside attention (RoPE).
4. Prefill. Every prompt token runs through all N transformer layers together. Each layer stores its K and V projections into the KV cache. This phase is compute-bound.
5. LM head. Only the final position’s hidden state is projected to vocabulary size (~128K logits).
6. Sampling. Softmax turns logits into a distribution; a policy (greedy, top-k, top-p, temperature, …) picks one ID.
7. Decode loop. That ID is appended, embedded, and passed through the stack again with the KV cache — one new token per step — until a stop rule fires.
8. Detokenize and stream. IDs become text fragments; the server may flush partial strings to the client as soon as they are valid.

Time-to-first-token is almost always the prefill bill. Everything after that is decode.

Q2. Why call inference two different workloads?

Prefill and decode sit on opposite sides of the roofline model.

In prefill you load weights once and apply them to thousands of token vectors. In decode you reload the same weights for a single vector — gigabytes moved for millions of FLOPs. The GPU waits on HBM.

Nearly every serving trick — batching, KV cache, quantization, speculative decoding, FlashDecoding, disaggregated prefill/decode — exists to feed that starved compute unit.

Q3. Which latency metrics matter, and which knobs move them?

• TTFT — mostly prefill; grows with prompt length (and worse once attention dominates). Helped by prefix cache, chunked prefill, faster attention, shorter prompts.
• TPOT / ITL — mostly decode; roughly bytes_per_step / bandwidth. Helped by quantization, GQA/MQA, speculative decoding, smaller models.
• End-to-end ≈ TTFT + TPOT × output tokens.
• Throughput (tokens/sec/GPU) — a fleet metric; larger batches raise it while usually hurting per-user TPOT.

Throughput and latency fight each other. Serving policy is choosing a point on that curve.

Q4. Why can’t decode parallelize like prefill?

Generation is autoregressive: each new token conditions on all previous ones. Prefill already knows the entire prompt, so positions can be computed together under a causal mask. In decode the “future” does not exist yet.

Speculative decoding is the workaround: draft cheap guesses, then verify them in one parallel target pass — turning a sequential dependency into parallel verification (Part 8).

Part 2 — Stages one and two: feeding text in

Q5. Define a token — and why whole words or raw characters are the wrong extremes

A token is a subword atom in a fixed vocabulary. Pure extremes fail:

• Word vocabularies are unbounded; typos and rare names become <UNK> and destroy information while exploding the embedding table.
• Character vocabularies avoid OOV but stretch sequences 4–5×, punishing quadratic attention and forcing early layers to rediscover morphology.

Subwords split the difference: common words stay one piece; rare forms break into meaningful fragments; nothing is truly OOV.

Q6. How is BPE trained?

Byte-Pair Encoding starts from characters (or bytes) and repeatedly merges the most frequent adjacent pair until a target vocabulary size is reached. Merges are stored as a ranked list. At inference, the same merges apply greedily left-to-right. Training is count-based statistics, not gradient descent — but the resulting segmentation shapes everything downstream.

Q7. BPE vs WordPiece vs Unigram / SentencePiece

• BPE — merge by frequency; decode is deterministic given the merge table.
• WordPiece (BERT-era) — similar merges but scored by likelihood; often uses ## continuation markers.
• Unigram / SentencePiece — starts from a large candidate set and prunes using a unigram LM; can sample segmentations and handles raw bytes/Unicode cleanly without language-specific pretokenizers.

Production LLMs today skew toward byte-level BPE or SentencePiece variants.

Q8. Why byte-level BPE?

Working from raw bytes guarantees every string is encodable — no mystery Unicode failures. The vocabulary still learns common multi-byte pieces, so English stays efficient while rare scripts degrade gracefully into byte fallbacks.

Q9. Why do models stumble counting letters in “strawberry”?

Because the model never sees characters; it sees token IDs. If "strawberry" is one or two pieces, “count the r’s” is asking for structure that was discarded before the first layer. Character-level tasks need either a tokenizer that preserves characters or an external tool.

Q10. Do token IDs mean anything alone?

No. An ID is only an index into that model’s embedding table. ID 100 in one tokenizer is unrelated to ID 100 in another. Never mix tokenizers across models.

Q11. Production tokenizer gotchas

• Chat templates and special tokens must match training.
• Leading spaces often become their own tokens — " Hello" ≠ "Hello".
• Truncation can silently drop system instructions if you count characters instead of tokens.
• Multilingual byte fallbacks inflate length and cost.
• Image/audio multimodal special tokens change packing rules.

Q12. Vocabulary size tradeoffs

Larger V shortens sequences and helps multilingual coverage but grows embedding + LM-head parameters (V × d_model each) and decode softmax cost. Typical modern ranges sit around 32K–150K; beyond that you pay memory and bandwidth for diminishing returns.

Q6. How BPE is actually trained

Byte-Pair Encoding is compression reused as vocabulary learning:

1. Seed the vocab with base symbols (for byte-level BPE: all 256 bytes).
2. Express the corpus as sequences of those symbols.
3. Count adjacent pairs; pick the most frequent.
4. Merge that pair into a new symbol, append it to the vocab, and record the ranked merge rule.
5. Repeat until you hit the target size (for example ~100K–128K in modern stacks).

Artifacts: a symbol→ID map and an ordered merge list. Encoding at inference replays merges greedily in that order — deterministic, not a global shortest-segmentation search.

Subtlety: merges optimize frequency, not linguistics. ing becomes a token because it is common. Morphological elegance is incidental.

Q7. BPE vs WordPiece vs Unigram / SentencePiece

• BPE (GPT-line, Llama): frequency merges; deterministic encode.
• WordPiece (BERT-era): similar loop, but picks merges that maximize training-data likelihood under a unigram model — scoring how informative a pair is together versus apart.
• Unigram LM (often via SentencePiece; T5/Gemma-class): start large and prune tokens whose removal hurts likelihood least. Probabilistic segmentation enables subword regularization (sampling alternate cuts in training).
• SentencePiece is a library that can run BPE or Unigram. Its practical gift is treating raw characters with a visible space marker so tokenization is reversible and whitespace-pretokenization-free — crucial for Chinese, Japanese, Thai.

Q8. Byte-level BPE matters because…

Unicode-character initials would need ~150K starters and still need <UNK>. Starting from UTF-8 bytes means every string — emoji, Cyrillic, broken bytes — is encodable. No OOV.

Cost: emoji can burn multiple tokens; non-Latin scripts often cost 2–4× English for the same meaning, so non-English callers hit rate limits and context ceilings sooner. Larger vocabs (32K→128K-class jumps) exist partly to soften that fairness gap.

Q9. Why “how many r’s in strawberry?” fails

The network never sees letters; it sees IDs. If strawberry is a few subwords, counting characters requires recovering spelling from embeddings and aggregating — there is no native “walk characters” op. The same pathology hits anagrams, palindromes, reversals, and naive arithmetic when digit grouping is inconsistent across numbers. Modern tokenizers sometimes force digit splits specifically to make arithmetic learnable.

Q10. Token IDs in isolation

An ID is an index into that model’s tables. Crossing tokenizers is meaningless. Even “special” IDs differ across families.

Q11. Gotchas that bite in production

Matching the training chat template and special-token set is mandatory. Leading spaces are often real tokens. Counting characters when the API bills tokens silently truncates instructions. Multimodal sentinel tokens change packing. Cache keys must include tokenizer revision. Off-by-one BOS/EOS handling breaks stop conditions and logprob audits.

Q12. Vocabulary size tradeoffs

Bigger V shortens sequences and helps multilingual packing but inflates embedding + LM-head parameters and the decode softmax. Past ~100K–200K you usually pay memory/bandwidth for thin quality gains unless multilingual coverage demands it.

Part 3 — Embeddings

Q13. Embedding: gather vs matmul

Row lookup equals multiplying a one-hot by E. Kernels gather; math stays linear.

Q14. Contextual or not?

Input rows are non-contextual. Context is built by attention across layers. Retrieval embeddings are typically deeper states, not the input table row.

Q15. Weight tying

Share embedding weights with the LM head (transpose) to save parameters — common in smaller models. Large LMs often untie for quality. Tied heads are not an extra independent d × V matrix.

Q16. Positional information

Absolute embeddings add per-index vectors. RoPE rotates Q/K so relative offsets appear in scores and cooperate with KV caching. Relative bias schemes are alternatives; RoPE dominates new LLMs.

Q17. RoPE and long-context extension

RoPE’s relative geometry extrapolates better than naive absolute tables. Extensions (position interpolation, NTK-aware scaling, YaRN, etc.) rescale rotary frequencies so longer contexts remain stable without training from scratch.

Q18. The √d_model factor

When token and position vectors are added, scales can clash; multiplying embeddings by √d_model balances them. Rotation-based RoPE stacks often omit that multiply.

Part 4 — The transformer block

Q19–Q23. Attention mechanics (expanded)

x → MultiHeadSelfAttention → Add & Norm → FFN → Add & Norm → out

Attention(Q, K, V) = softmax(QKᵀ / √d_k) V

Queries seek, keys advertise, values deliver. Softmax over scaled dots yields mixing weights. Divide by √d_k so scores do not saturate as width grows. Many heads diversify subspaces. Causal masks write −∞ into forbidden futures so softmax mass is exactly zero — zeros in the mask would still leak. During pure decode with a cache, the key set has no future positions, so the mask is implicit; prefill still needs masking (or a causal kernel).

Q24–Q26. Shapes, cost, residuals

Projections are ~d × d each (modified under GQA). FFNs dominate parameters with ~4× expansion. Attention cost scales O(n² d); FFN O(n d²). Short prompts: weight/FFN bound. Long contexts: attention and KV memory dominate. Residuals keep deep stacks trainable by providing gradient highways and additive refinements.

Q27–Q30. Norms, FFN width, activations, MoE

RMSNorm skips mean centering and is cheaper than LayerNorm. Pre-norm stabilized very deep nets versus classic post-norm. Wide FFNs (~4×) buy token-wise capacity between attention mixers. Nonlinearities stop stacked linears from collapsing; SwiGLU won many LLM ablations. MoE routes tokens to a few expert FFNs so parameters scale faster than FLOPs per token — memory-heavy, compute-sparse.

Q31–Q32. Depth/width and attention variants

Depth repeats mixing; width enriches features. Labs choose under data and cluster limits. MHA stores K/V per head; MQA shares one K/V across queries; GQA shares per group; MLA compresses KV further. Serving cost pushed the industry along that sequence.

Part 5 — The KV cache

Q33. What is the KV cache, and why only K and V?

At decode step t a single new token arrives. Its query q_t must attend over keys and values for positions 1..t. Keys and values for 1..t−1 depend only on earlier tokens and frozen weights. Causality guarantees that introducing token t cannot rewrite those past K/V tensors — they are immutable — so they are computed once and stored.

Queries differ: you only need q_t for the current step. Older queries already produced older outputs and will never be read again, so caching Q is wasted memory.

Without a cache, emitting token t would re-run the whole prefix — work that grows harshly with length. With the cache, attention at each step is linear in current length and the FFN is effectively constant work per new token. Generation becomes linear in the number of decode steps instead of quadratic-in-passes.

Q34. Derive KV cache size with a real number

KV bytes = 2 (K and V)
         × n_layers
         × n_kv_heads × d_head
         × seq_len
         × batch_size
         × bytes_per_element

Plug in a Llama-3–70B-class GQA layout: eighty layers, eight key/value heads, one-hundred-twenty-eight dims per head, stored in two-byte floats.

Multiply those factors: two tensors (K and V) × eighty × eight × one-twenty-eight × two bytes lands near 320 KB for every token of every live sequence.

Stretch that to an 8k-token dialogue and you are carrying on the order of 2.6 GB for a single user. Push toward 128k context and the same formula points at tens of GB — enough that one conversation can occupy a large fraction of an H100-class device before weights are even counted.

Counterfactual without GQA (64 KV heads): the 8K figure jumps toward ~21 GB per sequence. GQA is the difference between tens of concurrent users and a handful.

Weights are a fixed tax paid once. The KV cache is a per-user, per-token, unbounded tax. It sets maximum batch size, which sets throughput, which sets dollars per million tokens. That is why diagrams label it the main memory bottleneck.

Q35. Why decode is memory-bound — the arithmetic

Consider a 70B fp16 model at batch 1 emitting one token:

• Bytes moved: ~140 GB of weights plus a few GB of KV ≈ 142 GB.
• FLOPs: ≈ 2 × 70e9 × 1 = 140 GFLOPs.
• Intensity: ≈ 1 FLOP/byte.

An H100 offers on the order of 3.35 TB/s HBM and ~1000 TFLOPS fp16. The ridge point where compute catches bandwidth is hundreds of FLOPs/byte. At 1 FLOP/byte you are using a tiny fraction of peak math units — the GPU is mostly a memory shuttle.

Wall time ≈ 142 GB / 3.35 TB/s ≈ 42 ms → ~24 tok/s from bandwidth alone. Real systems land within a factor of two of that estimate, which is how thoroughly bandwidth rules decode.

Corollary: raising batch size is nearly free for weight traffic until you become compute-bound. Moving from batch 1 to 32 reuses one weight load across 32 sequences and can multiply token throughput accordingly. That amortization is the central economic fact of LLM serving.

Q36. If batching is “free,” why not batch 512?

Because every sequence carries its own KV footprint, and cache memory scales with batch × seq_len:

GPU memory = model weights + KV cache + activations + fragmentation overhead

Example: 140 GB of weights on 2×H100 (160 GB) leaves ~20 GB for cache. At 320 KB/token that is ~65K tokens total across the batch. Sixteen users at 4K context already saturate the budget.

So KV size caps batch size, which caps throughput. GQA, MLA, cache quantization, paging, and eviction policies are throughput features wearing memory costumes.

Q37. PagedAttention and the fragmentation problem

Naive servers reserve a contiguous KV slab per request up to max_seq_len. That fails two ways:

• Internal fragmentation — a 100-token completion that reserved 4096 slots wastes most of the allocation.
• External fragmentation — freed regions of mixed sizes leave holes you cannot place new slabs into.

Reported waste in naive allocators often sits in the 60–80% range of KV memory.

PagedAttention (popularized by vLLM) copies the OS virtual-memory idea: fixed-size blocks (e.g. 16 tokens), plus a block table mapping logical positions to physical blocks anywhere in a pool. Blocks are allocated as generation lengthens.

Benefits:

• Fragmentation collapses → often 2–4× more concurrent sequences on the same GPUs.
• Prefix sharing / copy-on-write — many requests with one system prompt share physical blocks until their suffixes diverge; ideal for agents, RAG instructions, and few-shot stems.
• Beams and parallel samples share the common prefix instead of cloning it.

The price is indirection and an attention kernel that gathers from non-contiguous blocks.

Q38. Static vs continuous batching

Static batching waits to fill a batch, runs all sequences lockstep to the longest, and pads. Short requests wait for long ones; GPUs burn cycles on pad tokens.

Continuous (iteration-level) batching rebuilds the batch every decode step: finished sequences leave, new ones enter, and the GPU stays busy on live tokens only. Latency variance drops and goodput rises dramatically — this is table stakes in modern engines (vLLM, TensorRT-LLM, etc.).

Q39. Chunked prefill and prefill/decode disaggregation

A multi-thousand-token prefill can monopolize the GPU and stall decode tails for interactive users. Chunked prefill splits long prompts into slices interleaved with decode steps so TTFT rises a little while TPOT stays smoother.

Disaggregation goes further: run prefill on compute-optimized pools and decode on memory-optimized pools, shipping KV over the network. It adds orchestration complexity but matches each phase to the hardware it actually saturates.

Q40. Shrinking KV beyond GQA

Levers include:

• MQA / GQA / MLA — fewer or compressed KV heads.
• Quantized caches (INT8/FP8/NF4-style) — often near-lossless for K/V.
• Windowed / sliding attention — drop far tokens when the task allows.
• Eviction / streaming policies for extreme contexts.
• Cross-layer reuse research ideas that share projections across depth.

Each trades a bit of quality or generality for large memory wins.

Q41. Prefix caching — when architecture changes

If many calls share a long immutable prefix (system prompt, tool schemas, retrieved sticky context), caching that prefix’s KV avoids recomputing it. Architectures then favor stable prefixes and mutable suffixes: put volatile user text at the end, keep the shared stem byte-identical so cache keys hit. Product copy and prompt compilers become part of the performance surface.

Q42. When does the KV cache become invalid?

Anytime past tokens are edited, not merely appended: regenerating mid-conversation, rewriting a system prompt, swapping tool definitions, or applying non-causal edits. Prefix caches must be keyed by exact token identity (and template version). Sampling parameters do not invalidate K/V — they only change how logits become the next ID — but changing the model adapter, draft model, or RoPE scale mid-flight can.

Part 6 — LM head (linear + softmax)

Q43. What the LM head does and its shapes

The last block emits h ∈ ℝ^(seq_len × d_model). After a final norm, one linear map produces vocabulary logits:

logits = h @ W_lm      W_lm ∈ ℝ^(d_model × V)

For d_model = 4096 and V = 128,000, W_lm holds 524M parameters and yields a 128K-wide vector per position scored.

Q44. Logits for every position during decode?

No. Prefill may score many positions for training or specific APIs, but ordinary decode only needs the last position’s hidden state. Earlier positions already chose their tokens.

Q45. Cost of the LM head at decode

Each step multiplies a d-vector by d × V — on the order of 2 × d × V FLOPs and a full read of a potentially untied head. At large V this rivals an FFN slice and motivates adaptive softmax, vocabulary parallel sharding, and tied embeddings when acceptable.

Q46. Logits vs probabilities

Logits are unbounded scores. Softmax normalizes them:

p_i = exp(z_i) / Σⱼ exp(z_j)

Remember shift invariance: adding a constant to all logits does not change probabilities — the foundation of stable implementations and of temperature as a rescaling of differences.

Q47. Numerically stable softmax

Naive exp(z_i) overflows in fp16 for moderate logits and can underflow an entire row to zeros, producing 0/0. The fix subtracts the row max first:

p_i = exp(z_i − max(z)) / Σⱼ exp(z_j − max(z))

The largest term becomes exp(0) = 1; the rest lie in (0, 1]. Online algorithms (as in FlashAttention) fuse this with matmul to avoid materializing huge score matrices.

Q48. Perplexity

Perplexity is exp(average negative log-likelihood) over tokens — how “surprised” the model is. It is computed from the same logits/softmax stage. Lower perplexity on a domain suggests better next-token calibration there, but it is not a direct measure of helpfulness or safety.

Part 7 — Sampling strategies

After the LM head you hold a vector of logits. Sampling is how that vector becomes the next ID.

Q49. Greedy decoding — and why it disappoints

Greedy always takes argmax. It is deterministic and cheap, yet often bland or locally stubborn: early high-probability tokens can trap the sequence in repetitive, high-likelihood loops that a human rates as worse than slightly lower-likelihood alternatives. Use it for brittle extraction tasks; avoid it as a default for open-ended prose.

Q50. Why not beam search?

Beam search keeps the top-B partial hypotheses. It helps when the target is short and constrained (translation-style). For open chat it tends to produce generic, high-likelihood safe text, costs B× compute, and interacts poorly with nucleus-style diversity. Most LLM apps abandoned it for sampling.

Q51. Top-k

Keep only the k largest logits, renormalize, sample. Simple, but k is blunt: sometimes probability mass outside the top-k still matters; sometimes top-k includes junk when the distribution is sharp. Fixed k cannot adapt to sharpness.

Q52. Top-p (nucleus)

Sort by probability, keep the smallest set whose cumulative mass ≥ p, then sample. When the model is confident the nucleus shrinks; when uncertain it widens. That adaptivity is why top-p displaced fixed top-k as a default in many stacks. Typical p lives around 0.9–0.95, tuned per product.

Q53. Temperature — math and intuition

p_i = exp(z_i / T) / Σⱼ exp(z_j / T)

Divide logits by T before softmax.

• T → 0 — gaps explode; distribution collapses to argmax (greedy).
• T = 1 — unscaled model distribution.
• T > 1 — flatten; more surprise and risk of incoherence.

Temperature does not “add randomness” by itself in implementation — it reshapes the distribution that a sampler draws from.

Q54. Order: temperature vs truncation

Apply temperature to logits first, then top-k / top-p on the resulting distribution (or equivalently truncate on tempered probabilities). Truncating before temperature changes which tails exist and is usually the wrong composition. Document the order your server uses; silent swaps change behavior.

Q55. Repetition, frequency, and presence penalties

• Repetition / frequency — down-weight tokens already seen, often scaled by count.
• Presence — penalize a token once if it appeared at all, regardless of count.

They fight loops in creative decoding but can harm deliberate repetition (code, poetry, legal phrases). Prefer light penalties and strong stop rules over aggressive hobbling.

Q56. Beyond top-p

Min-p thresholds relative to the top probability (p_i ≥ min_p × p_max), behaving better when temperature is high and mass is diffuse. Mirostat and related controllers target a surprise level. Typical production still centers on temperature + top-p, with min-p gaining share.

Q57. Temperature 0 still varies — why?

Even “greedy” stacks differ across:

• Non-deterministic GPU reductions and fused kernels.
• Different batching / continuous-batch schedules.
• Speculative paths and numerically different softmax implementations.
• Server-side default penalties, stop strings, and templates.
• MoE routing noise if enabled.

Bit-stable greedy requires locked kernels, batch size 1, and identical software revisions — rare in multi-tenant fleets.

Q58. Forcing JSON or a schema

Options escalate:

1. Prompt hard for JSON (fragile).
2. Constrained decoding / grammar masks — zero out illegal tokens each step (regex, JSON schema, CFG).
3. Tool/function calling APIs that parse side channels instead of free text.
4. Post-hoc repair (last resort).

Constrained decoding is the reliable systems answer when validity is mandatory.

Q59. Evaluating sampling settings

Hold the model fixed and sweep temperature/top-p on a labeled set: task success, toxicity, diversity (distinct-n), repetition rate, latency. Prefer offline eval plus online A/B with clear product metrics. Do not tune on a single cherry-picked prompt.

Part 8 — Speculative decoding

Q60. Core idea

A cheap draft model proposes several future tokens; the target model verifies them in one parallel forward over the draft span. Accepted prefix becomes output; on the first rejection, sample a replacement from the target distribution and discard the rest. You spend most target capacity on parallel verification instead of one-token serial steps.

Q61. Why the output distribution is unchanged

Acceptance uses ratios between target p and draft q so that accepted tokens plus residual resampling reconstruct exactly p:

P(emit x) = q(x)·min(1, p(x)/q(x)) + P(reject)·p'(x)

Accepted path contributes min(q,p); residual sampling fills max(0, p−q). Summation yields p. Hence speculative decoding is lossless relative to ordinary target sampling — a systems speedup, not an approximation of the model.

Q62. Expected speedup

With acceptance rate α and draft length k, expected tokens per target forward are:

E[tokens] = (1 − α^(k+1)) / (1 − α)

Even if every draft token misses, you still obtain one token from the target’s verification pass. With draft cost c relative to a target step:

speedup ≈ E[tokens] / (1 + k·c)

Example: α=0.8, k=4 → ~3.36 tokens expectation; with c=0.05 → ~2.8×. Published wins often land in the 2–3× band. Large k sees diminishing returns because survival probability falls as α^k.

Q63. When speculation hurts

Low acceptance (hard prompts, domain shift, weak draft), draft too expensive, or kernels that cannot verify in parallel efficiently. Then overhead exceeds gain — gate on measured α.

Q64. KV cache on rejection

Rollback target (and draft) cache entries for rejected positions. Implementations checkpoint lengths and truncate; bugs here corrupt future attention. Accepted tokens keep their K/V.

Q65. Modern variants

Medusa / multi-head drafts, eagle-style feature-level drafters, prompt-lookup / n-gram drafts from the context itself, and cascaded drafters. Same verify-with-target principle.

Q66. Tree attention

Drafts can propose a tree of candidates, not only a linear chain. Tree attention verifies multiple branches in one target pass, raising expected accepts per call when branching is cheap.

Part 9 — Detokenizer and streaming

Q67. Detokenization is not a dumb reverse map

Detokenizers must undo space markers, handle byte fallbacks, compose multi-token graphemes, and avoid emitting incomplete UTF-8. Streaming detokenization buffers until a complete character/piece is safe to flush — otherwise clients see replacement characters.

Q68. How streaming works and what to measure

Servers emit tokens (or partial strings) as they are produced. Measure TTFT, TPOT, and time-to-first-readable character after detokenize buffering. Client UX cares about that last number more than raw token timestamps.

Q69. Stop sequences are tricky

Stops may span token boundaries (\n\n vs one token containing both newlines). You must check decoded text (or a careful token window), not only single-ID equality. Overlap with partially generated stops needs rollback so the stop text is not shown.

Q70. What ends a generation?

EOS / special stop IDs, matched stop strings, max-token limits, client cancel, or server watchdog. Prefer explicit EOS handling consistent with the chat template.

Part 10 — FlashAttention, quantization, roofline

Q71. FlashAttention — precise claims

FlashAttention computes attention without materializing the full n × n score matrix in HBM, using tiling and online softmax statistics. It reduces memory traffic and enables longer sequences / larger batches. It does not change the mathematical attention result (modulo numerics); it is an IO-aware algorithm, not a different model.

Q72. Does it help decode?

Decode with cache often attends with q length 1 to cached n keys — the classic FlashAttention win (huge square scores) is smaller. Variants (FlashDecoding, etc.) still help gather/KV bandwidth paths, but decode remains dominated by weight loads unless batch is large.

Q73. Quantization

Store/compute weights (and sometimes activations/KV) in narrower formats (FP8, INT8, INT4, …). Memory and bandwidth drop roughly with bit-width; quality loss depends on method (GPTQ, AWQ, smooth quant, …). Rule of thumb from practice notes: 8-bit often near-free; 4-bit costs a little, sometimes within bench noise — always measure on your tasks.

Q74. Roofline in one paragraph

Plot attainable FLOPs vs arithmetic intensity. Prefill sits toward the compute roof; batch-1 decode sits near the bandwidth roof at ~1 FLOP/byte. Optimizations that raise intensity (batching, fewer bits moved per useful FLOP) slide you right toward the ridge; pure kernel speedups cannot beat the roof you are under.

Q75. Splitting a large model across GPUs

Tensor parallel shards mats inside layers (needs fast interconnect). Pipeline parallel assigns layer ranges to ranks (bubble risk). Expert parallel shards MoE experts. Replica / data parallel copies the model for throughput. Real deployments combine them; KV must be partitioned or replicated consistently with the attention layout.

Part 11 — Whiteboard numbers

• 70B fp16 weights: 70e9 × 2 ≈ 140 GB.
• Forward FLOPs: ~2 × params × tokens (order-of-magnitude).
• KV per token (Llama-3-70B-class GQA): ~two · layers · kv_heads · head_dim · dtype_bytes on the order of a third of a megabyte per token.
• Batch-1 decode tok/s: ~bandwidth / weight_bytes (H100-class: tens of tok/s fp16 before other wins).
• Embedding + untied LM head at V=128K, d=4096: ~524M parameters each.
• Attention vs FFN params: roughly 1:2 in a standard block (4d² vs ~8d²).

Part 12 — Diagram caveats

Common teaching diagrams omit: chat templates before tokenize, GQA/MQA affecting KV width, sampling order details, detokenize buffering, paged KV, and speculative verify loops. Treat diagrams as maps, not contracts.

Closing

Inference is a pipeline from bytes to streamed text where prefill is compute-bound and decode is bandwidth-bound, the KV cache dominates multi-user memory, and every major optimization — GQA, paging, continuous batching, quantization, FlashAttention, speculation — exists to move work toward hardware sweet spots without changing the mathematical model more than you intend.

Deep dive A — Prefill vs decode economics in practice

When you size a fleet, start from the roofline split. Prefill jobs love tensor cores: long prompts on code-assist or RAG dumps can show healthy MFU if kernels are competent. Decode jobs at batch 1 look like accidental memory benchmarks — the same 70B weights steam through HBM for a single GEMV. That is why a product that streams 500-token answers feels slow even on flagship GPUs unless you batch, quantize, or speculate.

TTFT regressions usually mean prompts grew (history, RAG, tool schemas) without prefix caching. TPOT regressions usually mean batching collapsed, bandwidth fell, or KV pressure forced tiny batches. End-to-end latency is not a single knob: shortening answers helps TPOT times N; caching system prompts helps TTFT; speculative decoding helps TPOT when acceptance is high.

Throughput metrics posted on dashboards are easy to game with huge batches that make interactive users suffer. Publish both: tokens per sec per GPU and p50/p95 TPOT under a realistic mix. Continuous batching keeps the GPU fed without the pad tax of static batches; chunked prefill stops a single 32K prompt from freezing everyone else’s decode.

Disaggregated prefill/decode is the architectural admission that the two workloads want different machines. Prefill nodes can be denser compute; decode nodes need memory capacity and bandwidth for KV. The network must move KV efficiently or you lose the win. Not every team should build this on day one — but every team should know why the pattern exists.

Deep dive B — Tokenizer and multilingual cost

Byte-level BPE’s universality is a gift and a bill. English chat apps under-measure cost for Hindi, Thai, or emoji-heavy social text. If your margin model assumes English tokens per second, multilingual expansion can silently burn budget. Prefer tokenizers with larger multilingual coverage when that traffic matters; measure tokens per grapheme cluster on real corpora, not lorem ipsum.

Chat templates are part of the model identity. A single missed special end token can cause run-on generations or cutoffs. Store template version next to model revision in your registry. When debugging “the model ignores the system prompt,” tokenize both sides and diff IDs before blaming the weights.

Digit policies matter for spreadsheet agents and math tutors. If 1000 and 1,000 segment differently, column algorithms become lottery tickets. Prefer tokenizers with explicit digit rules when building calculators on top of LLMs, or route arithmetic to tools.

Deep dive C — Attention, GQA, and memory

Multi-head attention’s story is specialization under a budget. GQA’s story is KV bytes. Moving from MHA to GQA can cut cache size by roughly the ratio of query heads to KV groups — often 4–8× — with modest quality loss when trained for it. MQA is the extreme; MLA and related compressions chase the same bottleneck with different algebra.

Causal masking with negative infinity is not pedantry: any finite mask value leaks. Training bugs that accidentally allow future attention create models that look strong on teacher-forced metrics and then fail at true autoregressive use. Decode-time kernels that assume cache layout must match training geometry (head counts, RoPE base, scaling).

Residuals and pre-norm are why 80–100 layer stacks are routine. When a custom architecture removes them “to simplify,” expect training instability before cleverness. SwiGLU’s gating is a quality knob with memory implications — know whether your stack fuses it.

Deep dive D — KV paging and prefix strategy

PagedAttention reframes KV as a heap of blocks. Product consequences:

• Agent frameworks with huge shared system prompts should canonicalize that prefix so block hashes collide.
• Few-shot libraries should keep exemplars in a stable order; reshuffling kills cache hits.
• Personalization fields belong after the shared stem.
• Debug tools should show block-table hit rates the way CDNs show cache hit ratios.

Prefix cache invalidation is a correctness issue, not only a performance one. If a tool schema changes but the cache key ignores it, the model attends to stale KV for new instructions — subtle, nasty failures. Include tokenizer ID, template hash, LoRA ID, and RoPE scale in the key.

Eviction under memory pressure needs a policy: LRU on idle conversations, hard caps on context, or summarization compaction. Whatever you pick, surface it to product — silent truncation is a trust bug.

Deep dive E — Softmax, sampling, and determinism

Stable softmax is mandatory in fp16/bf16 paths. Fused kernels hide the max-subtract, but custom research code still trips overflow. Perplexity remains a useful regression canary when you change kernels or quant schemes: spikes mean the distribution moved.

Sampling composition should be unit-tested: temperature then truncate then penalties then draw. Swap order in a test fixture and watch golden transcripts diverge. Constrained decoding for JSON should be load-tested; mask generation can become CPU-bound if implemented naively.

Temperature zero equals deterministic is a marketing sentence, not a mathematical guarantee on real GPUs. If you need audit-grade replay, record the sampled IDs (or RNG seeds plus batch-1 locked kernels). For most apps, reproducibility within a release is enough — document that.

Deep dive F — Speculative decoding operations

Lossless speculation is powerful because product owners fear quality drift. Still, operations must track acceptance rate by prompt class. Code completion often accepts well; creative open-ended chat may not. Adaptive draft length based on recent acceptance beats a static guess.

Tree speculation helps when the draft can propose alternatives cheaply. KV rollback correctness tests should be in CI: deliberately reject mid-run and assert that subsequent attentions match a non-speculative baseline.

Draft models must share the tokenizer (or a compatible mapping) with the target. Mismatched tokenizers create silent garbage that looks like speculation made the model dumber.

Deep dive G — Streaming UX and stops

Users perceive fluidity from detokenized characters, not raw IDs. Buffer until Unicode is valid; flush on whitespace when possible for chat UX. Stop sequences overlapping tokens need careful undo so the client never sees the stop text. Max-token cutoffs should append a clear continuation signal if the product allows continue.

Deep dive H — FlashAttention and quantization in serving

FlashAttention unlocks longer prefill and denser batches by saving HBM. Celebrate that without claiming it rewrites decode physics. Quantization is the decode lever: fewer bytes per weight load raise effective tokens per second. Validate on your eval suite — especially tool calling and long-context recall — before declaring 4-bit free.

Roofline reasoning keeps arguments honest: if you are bandwidth-bound, buying more TFLOPS without more bandwidth (or fewer bytes) disappoints. If you are compute-bound in prefill, better GEMM and chunk scheduling matter more than shaving KV.

Deep dive I — Multi-GPU realities

Tensor parallel needs fast interconnect or it stalls on all-reduces. Pipeline parallel needs microbatches to hide bubbles. Mixing them with continuous batching and paged KV is why inference engines are large codebases. When someone proposes just shard the model, ask which dimension and what happens to the KV layout.

Deep dive J — Putting the pipeline together

A request path worth memorizing:

1. Template plus tokenize on CPU
2. Embed and prefill with causal attention, writing KV (GPU, compute-bound)
3. LM head on last position then sample (GPU with CPU policy)
4. Decode loop with cache, continuous batching, optional speculation (GPU, bandwidth-bound)
5. Detokenize, stop checks, stream on CPU
6. Free or pool KV blocks in the allocator

Every optimization maps onto one arrow in that list. If you cannot point to the arrow, you are not reasoning about inference yet — you are collecting slogans.

Extra FAQ-style clarifications

Does increasing temperature increase creativity in a measurable way? It increases entropy of the next-token distribution. Whether humans call the result creative depends on the task; measure with diversity and quality metrics, not vibes alone.

Is greedy ever correct? Yes — classification-as-generation, structured extraction with constrained decoding, and some eval harnesses. Open-ended dialogue is usually not that case.

Can you delete the KV cache and recompute? Always, at quadratic cost. Edge devices sometimes do short windows without cache; servers almost never should.

Why do diagrams show softmax inside attention and again at the LM head? Different softaxes: one mixes values inside a layer; one turns vocabulary logits into a categorical distribution for sampling. Same function family, different jobs.

What should a new engineer memorize first? Prefill versus decode bottlenecks, KV bytes per token formula, TTFT versus TPOT, and that batching trades latency for throughput. Everything else hangs off those four ideas.

Extended walkthrough — Prefill kernels and TTFT levers

Prefill quality of service is mostly about how fast you can turn a prompt into a warm KV cache. Attention kernels, fused MLP paths, and whether you chunk the prompt decide the shape of the TTFT curve. When prompts share a long stem, prefix caching turns repeated prefill cost into a lookup; when they do not, every new session pays full price.

Chunked prefill is a fairness tool as much as a latency tool. Without it, one 40k-token ingest can block decode for everyone else on the GPU. With it, interactive streams keep ticking while the long job advances in slices. The trade is a slightly higher TTFT for the long job itself — usually acceptable.

Prefix placement is a product decision. Put volatile user text after the frozen stem so cache keys stay stable. If marketing rewrites the system prompt daily, treat that as a cache-bust event and expect TTFT to jump until the new stem is hot again.

Extended walkthrough — Decode bandwidth math in words

Imagine reading the entire weight set from HBM once per generated token at batch size one. The arithmetic intensity collapses toward one useful multiply-accumulate per byte moved. Hardware that shines on dense GEMM looks idle because it is waiting on memory controllers. Batching reuses each weight load across many sequences; quantization shrinks the load itself; speculative verification turns several serial steps into one parallel check. Those are different ways to climb the same hill.

When someone claims a kernel is “2× faster” for decode, ask which roof they were under. If the run was bandwidth-bound, a pure ALU optimization cannot deliver 2×. If they also cut bytes moved, believe them after a profiler screenshot.

Extended walkthrough — Serving policies and SLOs

Write SLOs separately for TTFT and TPOT. A release that improves throughput while p95 TPOT doubles is a regression for chat UX even if the capacity dashboard celebrates. Continuous batching parameters (max batch tokens, max waiting queue, priority for decode vs prefill chunks) are the practical knobs behind those SLOs.

Admission control belongs next to KV accounting: reject or queue new sessions when projected cache memory exceeds a watermark. Otherwise OOM kills look like random model failures to clients.

Extended walkthrough — Tokenizer ops checklist

Before promoting a model:

1. Freeze tokenizer files and chat template text in the same artifact store as the weights.
2. Golden-test a handful of multilingual strings and emoji for token-length surprises.
3. Golden-test digit-heavy strings if the product does math.
4. Confirm stop-token IDs match the template’s end markers.
5. Confirm logprob tooling uses the same tokenizer as inference.

Skip any of those and you will debug “model quality” that is really packaging drift.

Extended walkthrough — Attention variants as a timeline

Early LMs used full MHA and paid for it in cache. Production pressure invented MQA, then GQA as the compromise that most frontier checkpoints adopted. MLA and related low-rank KV ideas continue the same saga: keep query richness, shrink what you store per past token. When reading a paper, translate claims into “bytes per token of KV” and “quality delta on our eval” — marketing language about “efficient attention” is otherwise meaningless.

Extended walkthrough — Paged KV and multi-tenant behavior

Fragmentation math is brutal with contiguous reservations. Paging converts that into an allocator problem with fixed blocks. Multi-tenant clusters should track allocation failures, steal latency from block tables, and prefix hit rate. A sudden hit-rate drop often means a template whitespace change or a silent tokenizer bump.

Copy-on-write sharing across beams and parallel samples is not optional for search-like decoding; without it you multiply memory by beam width for free.

Extended walkthrough — Sampling for products

Ship defaults that match the job:

• Extraction / tool args: low temperature, constrained decoding, greedy or tiny nucleus.
• Support chat: moderate temperature, nucleus around the common 0.9 band, light repetition penalty if loops appear.
• Creative brainstorm: higher temperature, consider min-p, stronger user-visible warnings about variance.

Log the sampling tuple with each response for support tickets. “The model said X yesterday” is unanswerable without it.

Extended walkthrough — Speculative decoding rollout

Roll out behind a flag. Measure acceptance by endpoint. Disable when α falls below a break-even derived from your draft cost c and draft length k. Keep a non-speculative canary traffic slice forever — it is your quality ground truth.

Ensure rejection paths scrub KV. Add a chaos test that forces mid-sequence rejects under load.

Extended walkthrough — Streaming protocol details

Decide whether the wire format streams tokens, detokenized strings, or both. Clients that re-tokenizeize differently will disagree on boundaries. Prefer server-side detokenize with explicit “flush” events when a complete grapheme cluster is ready. Document how stop sequences are stripped so UI code does not double-handle them.

Extended walkthrough — Quantization acceptance tests

Before enabling 4-bit weights in production:

• Run the standard eval suite.
• Run tool-calling fixtures.
• Run long-context needle tests.
• Compare refusal / safety probes if applicable.
• Compare p95 TPOT and VRAM.

If quality is within agreed epsilon and latency wins are real, ship. If not, stay on 8-bit for that model.

Extended walkthrough — Whiteboard drill answers in prose

Memory for a seventy-billion-parameter fp16 checkpoint is about one hundred forty gigabytes for weights alone. A forward pass costs on the order of two FLOPs per parameter per token. KV growth is linear in live tokens and in the number of key/value heads you actually store. Batch-one decode speed tracks memory bandwidth divided by bytes touched per step. Embedding tables and untied LM heads can each consume hundreds of millions of parameters at large vocabularies — they are not rounding errors. Inside a block, the feed-forward path often owns roughly twice the projection parameters of attention.

Extended walkthrough — Teaching diagram honesty

When you draw the pipeline for newcomers, annotate the omitted pieces: templates, paging, continuous batching, speculation, and detokenize buffers. Otherwise engineers implement the diagram and are surprised by production.

Closing addendum — What “done” looks like for an inference owner

You can explain, without slides, why a given incident hurt TTFT vs TPOT. You can estimate KV memory for a proposed concurrency × context target. You can name which lever — batch, quant, GQA, speculation, prefix cache — applies to the symptom. You treat tokenizers and templates as versioned dependencies. That is operational literacy for LLM inference; everything else is detail you can look up.

Case study narrative — A 70B interactive service

Suppose you expose a 70B fp16 chat model on a pair of high-memory GPUs. Weights alone claim on the order of one hundred forty gigabytes, so the pair is nearly full before any conversation starts. The remaining headroom must cover KV for all live sessions, activations, and allocator fragmentation.

You set a product target: sixteen concurrent users, four-thousand-token contexts, streaming answers. Rough KV math at a few hundred kilobytes per token says sixteen times four thousand times that rate is tens of gigabytes — too much for the leftover memory. Options appear immediately: switch to a GQA checkpoint, quantize the cache, lower concurrency, shorten context, or add GPUs. That single arithmetic conversation is the difference between a vague “we need more GPUs” ask and a precise capacity plan.

You also notice TTFT spikes at the top of each hour when many clients reconnect with identical system prompts. Prefix caching collapses those spikes once the stem is pinned. Failure to pin the stem after a prompt edit looks like a model regression in the metrics dashboard.

Case study narrative — Agent loops and cache keys

An agent sends the same tool schema on every turn. If the schema JSON is serialized with unstable key order, prefix hashes miss and you re-prefill thousands of tokens per step. Canonical JSON encoding becomes an inference optimization. Likewise, inserting the user profile before the schema breaks sharing across users; inserting it after preserves the shared tool stem.

Case study narrative — Speculative decoding on code vs chat

On a code-completion endpoint, a small draft model trained on similar data accepts most tokens; end-to-end latency drops by about 2×. On an open-ended creative endpoint, acceptance collapses and the same draft wastes capacity. Feature flags per route beat a global speculation switch.

Case study narrative — Stop sequence bugs

A team sets stop to a markdown fence. Half the time the fence is one token; half the time it is split. Users see trailing backticks. The fix checks decoded text windows and rolls back the stop span. Token-ID-only stops were the root bug.

Case study narrative — Quantization surprise

Four-bit weights improved tok/s by ~40% on a memory-bound decode path. Tool-calling evals dropped three points. The compromise: keep 8-bit for tool routes, 4-bit for long-form blog drafts. One model artifact, two serving profiles — allowed by routing on the gateway.

Practical checklist — Launch readiness

• [ ] Roofline story documented for prefill and decode
• [ ] KV bytes/token measured on the real checkpoint
• [ ] Concurrency × context budget fits GPU memory with margin
• [ ] Continuous batching enabled with decode priority policy
• [ ] Prefix cache keys include template, tokenizer, LoRA, RoPE scale
• [ ] Sampling tuple logged per response
• [ ] Constrained decoding for any JSON contract
• [ ] Streaming detokenize buffers incomplete UTF-8
• [ ] Stop sequences tested across token boundaries
• [ ] Speculative decoding gated on measured acceptance
• [ ] Quantization validated on task-specific evals
• [ ] Tensor/pipeline parallel plan matches interconnect reality

Practical checklist — Incident triage

1. Is TTFT or TPOT broken?
2. Did prompt length or batch composition change?
3. Is KV memory near the watermark?
4. Did tokenizer/template versions drift?
5. Did a kernel/quant flag flip?
6. Is speculation accepting poorly?
7. Are we bandwidth-bound or compute-bound on a profile?

Work the list in order; most “model is dumb today” pages resolve by step four.

Glossary in one place

TTFT — time until the first emitted token after request start. TPOT / ITL — average time between subsequent tokens. Prefill — parallel processing of the prompt that fills KV. Decode — autoregressive steps that emit new tokens. KV cache — stored keys/values for past tokens. GQA — grouped query attention reducing KV heads. PagedAttention — block-wise KV allocation. Nucleus (top-p) — adaptive truncation of the probability tail. Speculative decoding — draft-then-verify acceleration. Roofline — model relating intensity to peak FLOPs vs bandwidth.

Final synthesis paragraph

Treat inference as a manufacturing line. Tokenizers define the parts bins. Embeddings and blocks are the machines. The KV cache is work-in-progress inventory that consumes warehouse space per open order. Sampling is quality control on each unit. Streaming is the shipping dock. Speculative decoding is a parallel inspection lane. If you optimize a station without watching inventory and bandwidth, you create piles of unfinished work and angry customers — even though every station “looks faster” in isolation.

Appendix — Comparing mental models you will hear in interviews

Interviewers often ask you to reconcile three slogans: “transformers are parallel,” “decoding is sequential,” and “attention is quadratic.” They are all true in different scopes. Training and prefill exploit parallelism across positions that already exist. Decode cannot invent future tokens early without speculation. Attention’s quadratic term matters most when sequence length is large relative to width-driven matmuls. A strong answer states the scope before the slogan.

Another common trap is equating parameter count with decode speed. Parameters drive bytes moved per step; layout (GQA), dtype, and batching decide how those bytes translate to tok/s. Two 70B models can differ by 2× in interactive latency after those levers.

A third trap is treating sampling as “just RNG.” Temperature, truncation, penalties, and constraints each reshape the categorical distribution; RNG only chooses among what remains. Debugging weird outputs starts with logging that tuple.

Appendix — Minimal equations worth memorizing

KV bytes ≈ 2 × layers × kv_heads × head_dim × bytes × seq_len × batch. Intensity for batch-1 decode ≈ 2 × params / weight_bytes ≈ O(1) FLOP/byte for dense fp16. Expected speculative tokens per verify ≈ (1 − α^(k+1)) / (1 − α). Softmax stability: subtract max before exp. Causal mask: write −∞, not zero, into forbidden scores.

If you can re-derive those on a whiteboard and explain each symbol, you already outrank most hand-wavy explanations.

Appendix — Where to go deeper next

After this pipeline tour, natural sequels are: reading a PagedAttention-style allocator path in an open engine, profiling one prefill and one decode step with a vendor profiler, and implementing a tiny greedy decode loop with an explicit KV cache in NumPy or PyTorch to feel the shapes. Those exercises turn the vocabulary above into muscle memory without requiring a full training stack.

One more pass — Linking metrics to user pain

When a support ticket says “the AI is slow,” translate it. If the spinner sits before any text, chase TTFT: prompt length, cold prefixes, prefill congestion, or a missing cache. If text appears quickly but then dribbles, chase TPOT: batch bloat, KV memory forcing tiny batches, unquantized weights on a bandwidth-limited card, or speculation disabled on a route that used to win.

When the ticket says “answers got worse after Tuesday,” compare sampling tuples, template hashes, and quantization flags before you retrain anything. Serving regressions masquerade as intelligence regressions constantly.

When the ticket says “it invents JSON,” stop tuning temperature and ship constrained decoding. Sampling cannot enforce a grammar; masking can.

Capacity planning meetings go better when someone writes three numbers on the board: bytes per KV token, target concurrency, target context. Multiply, add weight footprint, compare to GPU memory. Arguments about “AI magic” end; procurement math begins.

That is the point of this entire pipeline tour: replace mystery with meters, and meters with levers you are allowed to pull.

Closing note on scope

This rewrite stays inside the inference path: what happens after weights exist, when a client sends text, and before an operations team starts talking about fine-tuning. Training-time concerns — data mixtures, optimizers, parallel training schedules — matter enormously, but they are a different course. If you can hold the prefill/decode split, the KV tax, and the sampling composition in your head under pressure, you already have the scaffolding on which those other courses attach.

Reader exercise

Take one production request from your logs. Write down its prompt token count, output token count, and observed TTFT/TPOT if you have them. Estimate KV memory with the per-token formula for your model’s layer and head layout. Ask whether the request was closer to compute-bound prefill or bandwidth-bound decode. Name one lever — prefix cache, batch size, quantization, GQA, or speculation — that would have moved the metric you care about most. If you can finish that exercise without opening this article again, the rewrite did its job.

Keep a short personal cheat sheet: prefill versus decode, KV bytes formula, TTFT versus TPOT, and sampling order. Review it before capacity reviews and before blaming the model for serving bugs. Small habits like that compound into reliable intuition. Practice explaining the pipeline aloud in under two minutes; fluency under time pressure is the real test of understanding. Practice explaining the pipeline aloud in under two minutes; fluency under time pressure is the real test of understanding. Practice explaining the pipeline aloud in under two minutes; fluency under time pressure is the real test of understanding.