This article is published in English.
220 tok/s Qwen reports on dual 3090s — what to verify
Reproduce community throughput claims with careful batching, quantization, and measurement notes.
Use this as an operator-facing rebuild of the ideas in “Community reports 220 tokens/second Qwen3.8–27B on two RTX 3090s. I tried it — and figured out why most people can’t reach it”: clear stages, ordered code slots, and recovery notes that survive a handoff. Overview works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.
Getting SGLang running (a much bigger deal than llama.cpp)
For Getting SGLang running (a much bigger deal than llama.cpp), define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.
git clone https://github.com/sgl-project/sglang.git
cd sglang
python3.12 -m venv .venv
source .venv/bin/activate
RuntimeError: cargo is required to discover the Rust extension modules
SGLANG_BUILD_RUST_EXTS=none pip install -e "./python[all]"
CUDA_VISIBLE_DEVICES=1,2 python -m sglang.launch_server \
--model-path /mnt/models/Qwen3.8-27B-AWQ-INT4 \
--tp 2 \
--speculative-algorithm DFLASH \
--speculative-draft-model-path /mnt/models/Qwen3.8-27B-DFlash2 \
--speculative-num-draft-tokens 8 \
--mamba-radix-cache-strategy extra_buffer \
--disable-prefill-cuda-graph \
--cuda-graph-max-bs-decode 16 \
--mem-fraction-static 0.85 \
--context-length 65536 \
--reasoning-parser qwen3 \
--host 0.0.0.0 --port 30000
The results: technically alive, spiritually disappointed
For The results: technically alive, spiritually disappointed, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.
| Metric | Min | Max | Average |
|-------------------------|----------|-----------|-----------|
| Decode speed | 47 t/s | ~100 t/s | ~55 t/s |
| Prompt prefill | 342 t/s | 466 t/s | ~428 t/s |
| Accept length (DFlash) | 3.2 | 6.8 | ~4 |
| Capability | Advertised limit | Reality |
|-------------------------|---------------------------|-----------------------|
| Max context per request | 65,536 (I set it) | fits, one at a time |
| Total KV pool | ~131K tokens max | 103K at default flags |
| Parallel slots | 48 concurrent requests | 3–6 with real prompts |
Why reality left the theory in the dust
For Why reality left the theory in the dust, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call. For Why reality left the theory in the dust, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.
Setup Custom allreduce failed with CUDART error: peer access is not
supported between these two devices.
/sys/bus/pci/devices/05:00.0/current_link_width → 4 (max 16)
/sys/bus/pci/devices/09:00.0/current_link_width → 4 (max 16)
Life at full tilt: context limits and one dark reboot
When working through Life at full tilt: context limits and one dark reboot, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.
Verdict: SGLang is picky, and it’s not your software
When working through Verdict: SGLang is picky, and it’s not your software, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.
Operational checklist
When working through Operational checklist, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest.
Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.
Pin dependency versions and record the image digest that ran the demo. Reproducibility beats tribal knowledge.
Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.
Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.
Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.
Batch note for 0dcc1bedc39e: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.
For hardening note 0, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
Hardening detail 0/1002: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
When working through hardening note 1, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
Hardening detail 1/1002: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
hardening note 2 works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.
Hardening detail 2/1002: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
For hardening note 3, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.
Hardening detail 3/1002: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.