Why Dual RTX 3090s Hit 55 tok/s Instead of 220 with SGLang and DFlash2
A hands-on look at running Qwen3.8-27B on two 3090s with SGLang, and how PCIe x4 links, missing P2P, tight KV cache and power spikes crushed the advertised speed.
This article is published in English.
A short video has circulated among self-hosting fans. It shows Qwen3.8-27B writing code across a pair of RTX 3090 cards, with generation speed blowing by 220 tokens per second. The setup uses SGLang plus DFlash2, a speculative drafter. That small model proposes seven upcoming tokens at once, and the large model accepts or rejects them in a single pass, so correct guesses cost nothing.
The attraction runs deeper than a demo. At that speed, an agent harness could point at a locally hosted model, running several coding agents side by side on one box with no API keys, no metered billing and no data leaving the building. A home server with two spare 3090-class GPUs seems ideal: twice the memory bandwidth, so surely 220 tokens a second.
It doesn't work out that way. The reasons are more instructive than the clip.
Installing SGLang
Anyone used to llama.cpp, where one binary and one model file give a chat in ten minutes, will find SGLang humbling. Here's the sequence on an Ubuntu server.
Step 1, the Python environment. SGLang has a big dependency tree, so it gets its own virtual environment on Python 3.12 after cloning the repository.
git clone https://github.com/sgl-project/sglang.git
cd sglang
python3.12 -m venv .venv
source .venv/bin/activate
Step 2, a stubborn install. The first genuine install attempt churned for roughly 39 minutes, filling a wheel cache with most of PyPI, then aborted with an error.
RuntimeError: cargo is required to discover the Rust extension modules
A Rust compiler, for a GPU inference server. Newer SGLang releases include optional Rust extensions, and the build tooling refuses to proceed without cargo although the engine never uses them. Setting one environment variable skips them.
SGLANG_BUILD_RUST_EXTS=none pip install -e "./python[all]"
That second run needed another 30 minutes or so, then completed without trouble. The result was sglang 0.5.20, torch built for CUDA 13 and triton automatically fixed at 3.7.1, which happens to be what Ampere GPUs need to dodge a known kernel bug.
Step 3: the models. Two downloads follow the community recipe:
cyankiwi/Qwen3.8-27B-AWQ-INT4is the primary model. INT4 quantization shrinks it to around 20 GB in 5 shards, small enough for two 24 GB GPUs.z-lab/Qwen3.8-27B-DFlash2is the speculative drafter, about 3.9 GB.
Step 4: the launch. One command does it, with TP=2 meaning tensor parallelism across both 3090s.
CUDA_VISIBLE_DEVICES=1,2 python -m sglang.launch_server \
--model-path /mnt/models/Qwen3.8-27B-AWQ-INT4 \
--tp 2 \
--speculative-algorithm DFLASH \
--speculative-draft-model-path /mnt/models/Qwen3.8-27B-DFlash2 \
--speculative-num-draft-tokens 8 \
--mamba-radix-cache-strategy extra_buffer \
--disable-prefill-cuda-graph \
--cuda-graph-max-bs-decode 16 \
--mem-fraction-static 0.85 \
--context-length 65536 \
--reasoning-parser qwen3 \
--host 0.0.0.0 --port 30000
Watch for two traps. A flag named --cuda-graph-max-bs prevents startup, as recent builds renamed it to separate decode and prefill variants. Also, leaving out --reasoning-parser qwen3 makes the API return the model's raw thinking, with nested think tags, inside the normal content field. Adding the parser splits output into tidy reasoning_content and content fields, the same layout other OpenAI-compatible servers use, which anyone writing a client will appreciate.
Step 5, waiting it out. Startup on the first run lasted about six minutes until the ready message showed up. Nothing had failed: SGLang compiles GPU kernels just in time through Triton and records CUDA graphs on each card. Subsequent starts reuse the cache and finish in roughly 35 seconds. A progress bar that seems stuck during that first boot is just the compile in progress.
What the measurements showed
The server worked. It answered prompts, streamed reasoning_content and accepted agent connections. These are the numbers measured on the machine itself:
| Metric | Min | Max | Average |
|-------------------------|----------|-----------|-----------|
| Decode speed | 47 t/s | ~100 t/s | ~55 t/s |
| Prompt prefill | 342 t/s | 466 t/s | ~428 t/s |
| Accept length (DFlash) | 3.2 | 6.8 | ~4 |
And this is the capacity picture:
| Capability | Advertised limit | Reality |
|-------------------------|---------------------------|-----------------------|
| Max context per request | 65,536 (I set it) | fits, one at a time |
| Total KV pool | ~131K tokens max | 103K at default flags |
| Parallel slots | 48 concurrent requests | 3–6 with real prompts |
That's about 55 tokens per second on average, and not 220. The sting is that a single RTX 3090 Ti running llama.cpp reaches about 75 tokens per second on the same model family. Two cards with tensor parallelism and a fancy drafter lost to one card using the older approach.
Acceptance rates explain some of the swing. When DFlash guesses land, with 6.8 of 8 tokens accepted, batches briefly touch about 100 t/s. When they miss, speed falls to 47. Even the best batch fell far short of 220, so the drafter wasn't the main culprit. Something else was eating the speed.
The clue in the log
The hint sat in the startup log all along:
Setup Custom allreduce failed with CUDART error: peer access is not
supported between these two devices.
Under tensor parallelism, every weight matrix is divided between the two GPUs, and partial outputs must be merged after each attention block and each MLP block. In a 27B model that adds up to around 100 round trips for every token produced. Merging at that rate needs a fast link between cards, and this pair lacked one. Two separate faults compounded each other.
No peer-to-peer. GeForce cards no longer officially support GPU-direct P2P, so each stitch gets staged through system RAM: out of GPU 1, into memory, across to GPU 2 and back. That doubles the copies and triples the latency.
A narrow link, the real killer. Checking the physical link state tells the story:
/sys/bus/pci/devices/05:00.0/current_link_width → 4 (max 16)
/sys/bus/pci/devices/09:00.0/current_link_width → 4 (max 16)
Both GPUs were negotiating PCIe x4, not x16. Four lanes apiece, routed through the host bridge, deliver only 1 to 2 GB/s between the cards, yet the workload crosses that link about 100 times for each token. The GPUs idled while waiting for copies. This explains prefill limping along at 428 t/s against an expected 1,200 or more, and decode stalling near 55.
GPUs are fast. The wire between them is slow. Under tensor parallelism, the wire effectively becomes the model.
How anyone reaches 220
Nobody shows their motherboard in a demo. A well-wired x16 pair with functioning P2P offers around ten times the inter-card bandwidth of this build. The headline number was most likely a short burst of predictable code with warm caches and an ideal drafter run, not a sustained rate. The recipe's software side is sound, but it quietly presumes hardware that few home machines have.
Two more daily-driver findings
Context room is smaller than expected. Two 24 GB cards appear generous. Then subtract the INT4 weights (about 10 GB on each card), the drafter, CUDA graphs and, because Qwen3.8 uses hybrid attention, the recurrent state memory. What remained for KV cache was about 103K tokens overall, or 131K with riskier settings. One 65K-token request fits fine. The advertised 48 concurrent slots does not, since every live conversation also claims part of the mamba state pool, and realistically only three to six sessions fit. By contrast, llama.cpp hands nearly all VRAM to a single model and handles large contexts without fuss.
A power-related reboot. With both 3090s kicking in together, running prefill and drafting under full tensor-parallel load, combined draw leaps from idle to about 700 watts within milliseconds. Power supplies handle that badly. One such spike triggered protection and shut the server off mid-generation. It restarted unaided, but the lesson holds: a pair of cards on this workload calls for a quality PSU with real transient headroom, and a wattage figure on the box isn't enough.
Should it be tried at home?
SGLang plus DFlash2 is impressive engineering when its assumptions hold, and those assumptions amount to a hardware checklist:
- Both GPUs in real x16 slots. Consumer gaming boards typically offer one, perhaps with a second wired through the chipset at x4. Read the manual before buying anything, since this line decides between 220 and 55.
- Working GPU-to-GPU communication. That means Resizable BAR and ACS tweaks in the BIOS, community driver forks on GeForce or NVLink outside the consumer world. Realistically, it's server-motherboard territory.
- A high-quality PSU with transient headroom. Spikes from two 350 W cards will catch out a weak supply eventually.
- Four RTX 3090s for Qwen3.8-27B at full context. That budget makes the KV cache stop being a cage, compared with a compromised 103K pool.
Software like this doesn't forgive cheap hardware. It runs slower somewhere nobody thought to look.
Lacking both halves, the software and the hardware it presumes, llama.cpp is still the sensible pick. It means one card, one binary, about 75 tokens a second over ordinary PCIe, no Rust toolchain and no unplanned reboots. The pair of GPUs can return to running independently at x4.
The 220 figure was real. It just doesn't apply to this box, and likely not to many others until the wire between the cards has been checked.