vLLM Serve Flags Explained, With Sensible Starting Values

vLLM is a go-to engine for serving large language models, mostly because of its throughput and the way PagedAttention handles memory. You'll usually drive it from the command line, with either vllm serve or the older python -m vllm.entrypoints.openai.api_server form. Here's one concrete command and then a tour of the flags you'll touch most, each with a sensible starting value.

A starting command

This one serves a mid-sized model on a multi-GPU machine. It's written for PowerShell, so the line continuations are backticks:

vllm serve meta-llama/Llama-3.1-8B-Instruct `
  --host 0.0.0.0 `
  --port 8000 `
  --tensor-parallel-size 2 `
  --gpu-memory-utilization 0.90 `
  --max-model-len 8192 `
  --max-num-seqs 256 `
  --dtype bfloat16 `
  --served-model-name llama3-8b `
  --api-key $env:VLLM_API_KEY

Every flag in it is explained below, along with a few extra ones that deserve a mention.

Picking the model and precision

model is the positional argument: a Hugging Face model ID or a local path to weights. It's the only thing you must supply. The other options have defaults, but they're rarely what you'd want in production.

dtype sets the numeric precision for weights and activations. The usual choices are auto, float16, bfloat16 and float32. On Ampere-class GPUs or newer, go with bfloat16. It's steadier numerically than float16, costs essentially nothing in throughput, and sidesteps the overflow problems some architectures hit in float16.

trust-remote-code lets vLLM run custom modeling code that lives in the model's Hugging Face repo. A few architectures need it to load at all. Keep it off unless the model card says otherwise, because turning it on means executing arbitrary code from that repo.

quantization picks a compression scheme such as awq, gptq or fp8, shrinking memory use at some quality cost. Set it only when you're loading weights that are already quantized, or when your GPU memory leaves no other option. With enough headroom, bfloat16 or full precision is the better call.

Budgeting GPU memory and context

gpu-memory-utilization is the share of GPU memory vLLM may claim for weights, activations and the KV cache. It affects both stability and throughput more than almost any other setting. Start at 0.90. Push much higher and other processes on the card can trigger out-of-memory failures. Drop lower and you waste KV cache space, which caps how many requests you can handle at once.

max-model-len caps prompt plus generated tokens for each request. The model's trained context length is the ceiling, but you can set it lower to save cache memory. Match it to what your app really needs. If a model supports 128K tokens and you only run short chat turns, there's no point reserving all that. A smaller value frees room for more simultaneous sequences.

max-num-seqs limits how many requests get batched together, so it governs concurrency. 256 suits many workloads. Raise it when there's spare GPU memory and heavy traffic, and lower it if long contexts are causing memory pressure.

block-size is the token count per block in the PagedAttention KV cache. Bigger blocks mean less bookkeeping but more wasted space for short sequences. The default of 16 is fine for most setups, and you'll rarely need to touch it.

swap-space reserves CPU RAM, in gigabytes per GPU, where KV cache blocks can be parked under memory pressure rather than turning requests away. Begin with 4. If bursts of traffic cause failures instead of orderly queuing, raise it.

Spreading the model across GPUs

tensor-parallel-size is how many GPUs each layer gets split across. It's the answer when a model won't fit on one card, and also a way to trade a bit of latency for throughput on multi-GPU hosts. Set it to the number of GPUs in the node, favoring powers of two (1, 2, 4, 8), since attention head counts usually divide cleanly by those.

pipeline-parallel-size sets the number of pipeline stages, typically combined with tensor parallelism when a model must span several nodes. Leave it at 1 unless the model is too big for tensor parallelism alone inside one node.

Exposing the server

host and port decide where the API server listens. Use 0.0.0.0 when other machines need to reach it, or 127.0.0.1 for local-only access. 8000 is the conventional port.

served-model-name is the name clients put in API requests, and it can differ from the Hugging Face ID. It's handy for giving a deployment a clean, versioned label. Choose something short and stable so applications downstream don't get tied to a specific weights path.

api-key forces clients to send a bearer token. Without it, anyone who can reach the port can send requests. Set it for anything past local experiments, and read it from an environment variable instead of hardcoding it in scripts or leaving it in shell history.

enforce-eager turns off CUDA graph capture and runs PyTorch in eager mode. You give up some throughput in return for quicker startup and easier debugging. In production, leave it off, since CUDA graphs usually give a sizable speedup. Flip it on temporarily when chasing a crash, because eager mode produces clearer stack traces.

How to tune

Most of the work is a three-way balance against available GPU memory: context length through max-model-len, concurrency through max-num-seqs, and precision through dtype and quantization. Begin cautiously with gpu-memory-utilization, measure real throughput and latency under realistic load, and adjust from there instead of guessing numbers up front.