Serving Qwen3.8-27B on One RTX 3090 With a Patched vLLM and DFlash2

A 27-billion-parameter model on a single consumer GPU usually means a painful choice between context length, concurrency and speed. A community fork of vLLM tuned for Qwen3.8-27B changes that equation on a 24 GB RTX 3090: in an agent-harness workload it reaches roughly 177 tokens per second of decode on a plain (non-Ti) card that is power-limited to 300 W. This guide walks through a native, container-free installation, explains which optimizations produce that speed, and shows exactly where the approach stops paying off so you can decide whether it fits your workload.

The two launch profiles covered below trade context length against parallelism and speed. The figures come from an agent workload on one box, so treat them as an indication rather than a guarantee.

| Config     | Context   | Parallel requests | My measured decode speed |
|------------|-----------|-------------------|--------------------------|
| CTX=fast   | 65,536    | 8                 | ~177 tok/s               |
| CTX=long   | 131,072   | 4                 | ~122 tok/s               |

Because the server exposes an OpenAI-compatible API, plugging it into an agent harness such as DeepSeek Harness is a matter of pointing the client at a base URL of the form http://<host>:18020/v1 and supplying a bearer key. Subagents can then issue requests in parallel against a local model that answers at speeds that previously required datacenter hardware.

What the qwen38-27b-rtx3090 project actually is

The project lives in the syv-ai/qwen38-27b-rtx3090 repository. It is neither a new model nor a new inference engine. It is a fork pinned to vLLM 0.28.0, combined with a set of patches, requantization scripts and launch profiles whose single purpose is fitting this particular hybrid model onto a 24 GB card and running it fast.

A Docker image is provided, and docker compose --profile single up -d is the entire installation if you are happy with containers. The repository also supports running the same steps by hand in a Python virtual environment, which is the route taken here. It keeps logs visible, avoids another large image on disk and makes it easier to see what each step changes.

Why a single tok/s figure means little

Throughput on this stack depends heavily on the task. Generating code is fast, open-ended prose is slower, and reproducing text that already sits in the prompt is dramatically faster (the lookup-drafting section below explains why). The project documentation stresses the same point: a throughput number for this stack says nothing unless you also know which prompt generated it. The 177 tok/s figure was measured with agent workloads; your own results will depend on what your prompts look like far more than on chance.

Installing natively, without Docker

Before you start, make sure the machine has:

• Linux or WSL2 with an RTX 3090 of any variant (the 24 GB of VRAM is what matters)
• A current NVIDIA driver, so that nvidia-smi runs without errors
• Python 3.12 or later, including its development headers
• A CUDA 13 toolkit, or at least its headers (see the curand pitfall in step 2)
• About 60 GB of free disk space for models and caches

A practical shortcut for the system-level prerequisites is to let a coding agent with shell access (Claude Code, Codex, OpenCode or similar) drive the installation. Point it at the repository and ask it to get the stack running on your 3090. When something fails, the agent reads the error and resolves it in place, be it installing python3.12-dev or libssl through apt, upgrading the driver or switching CUDA toolkits. That turns hunting down missing packages from an afternoon of forum searches into a routine step. Review what the agent runs, as you would for any tool with shell access.

Step 1: clone the repository

Start by fetching the repository and moving into it.

git clone https://github.com/syv-ai/qwen38-27b-rtx3090
cd qwen38-27b-rtx3090

Step 2: create a virtual environment and install the pinned vLLM

The fork targets vLLM 0.28.0 specifically, because that release is where DFlash2 speculative decoding became part of upstream vLLM (merged as PR #52816). The commands below create a fresh venv and install that exact version together with FlashInfer, the Hugging Face download tooling, ninja for kernel builds and pandas.

python3.12 -m venv venv
venv/bin/pip install -U pip
venv/bin/pip install vllm==0.28.0 huggingface_hub hf_transfer ninja \
  flashinfer-python flashinfer-cubin==0.6.13 pandas

Two details here are easy to get wrong:

• Do not allow pip to move flashinfer-python to another version. The launcher sets FLASHINFER_DISABLE_VERSION_CHECK=1, and the patches were validated against exactly this pinned pair.
• The DFlash2 sampling path compiles a FlashInfer kernel at runtime, and that kernel includes curand.h. If your CUDA installation lacks the curand headers, the compilation fails on first start and the server quietly switches to a slower fallback. Output stays correct, throughput drops by around 5%, and nothing in the logs tells you. On Ubuntu with CUDA 13, installing libcurand-dev-13-0 via apt fixes it.

The second point is a good example of a failure that no health check will catch. If your numbers look a few percent low, check for those headers first.

Step 3: download the base checkpoint

The starting point is the dbirks/Qwen3.8-27B-W4A16-AutoRound checkpoint, about 19.5 GB. Enabling the high-performance Xet transfer mode speeds up the download considerably.

HF_XET_HIGH_PERFORMANCE=1 venv/bin/hf download \
  dbirks/Qwen3.8-27B-W4A16-AutoRound \
  --local-dir models/Qwen3.8-27B-W4A16-AutoRound

Step 4: prepare the model

The scripts in prepare/ modify the checkpoint. They requantize the input and output embedding matrices, which public quantized checkpoints leave at full precision, rebuild the MTP draft head around a better-suited vocabulary, and then download the fast variant plus the 1.2 GB DFlash2 drafter. Everything runs on the CPU and each script finishes within a few minutes.

M=models/Qwen3.8-27B-W4A16-AutoRound
venv/bin/python prepare/quant_lm_head.py     $M
venv/bin/python prepare/quant_embed.py       $M
venv/bin/python prepare/quant_mtp.py         $M
venv/bin/python prepare/build_draft_vocab.py $M --ids prepare/draft_vocab_ids.json
venv/bin/python prepare/fetch_fast_variant.py
venv/bin/python prepare/fetch_dflash2.py

Step 5: apply the patch stack

The patches/ directory contains roughly fifteen .patch files covering embedding-quantization wiring, split-KV attention for draft verification, fixes to the Marlin int8 kernels, DFlash2 lookup drafting and more. Their order is defined in patches/series. The loop below strips comments and blank lines from that file and applies each patch to the vLLM package inside the venv. It skips dflash2-backport.patch, which only exists for vLLM versions older than 0.28.0 where DFlash2 was not yet native.

sed -e 's/#.*//' -e 's/^[[:space:]]*//;s/[[:space:]]*$//' -e '/^$/d' patches/series |
while IFS= read -r name; do
  [ "$name" = "dflash2-backport.patch" ] && continue
  patch -p1 -d venv/lib/python3.12/site-packages/vllm < "patches/$name"
done

A patch that fails to apply nearly always points to a version mismatch. Run venv/bin/pip show vllm and confirm that it reports exactly 0.28.0.

Step 6: verify the installation

Before launching anything, generate an API key and run the verification script in offline mode. It checks that every patch landed and that the model has the expected shape.

openssl rand -hex 24 > api_key.txt
bash verify.sh --no-server    # checks all patches applied + model shape correct

A clean result means your installation matches, byte for byte, the setup the maintainers benchmark against. For self-hosted inference, where subtle version drift is a constant source of confusion, that reproducibility is unusually valuable.

Step 7: launch the server

All configuration happens through environment variables. The default command enables DFlash2 speculation and prefix caching on the selected GPU; the comment shows how to switch to the long-context profile, or to a 245k profile after running kvarn/install.sh.

CUDA_VISIBLE_DEVICES=0 SPEC=dflash2 PREFIX_CACHE=1 bash single-user/start_qwen.sh
# add CTX=long for ~131k context (4 slots), or CTX=huge after kvarn/install.sh for 245k

Always set CUDA_VISIBLE_DEVICES explicitly. vLLM otherwise tends to choose the GPU with the most free memory, which on a multi-GPU machine may not be the card you intended. The bearer key is read from api_key.txt or from VLLM_API_KEY; set it before exposing the port to anything beyond localhost, because without a key the server accepts unauthenticated requests. Remaining settings come from .env. A couple of minutes after launch, you have an OpenAI-compatible endpoint serving a 27B model from a single card that costs less than a used bicycle.

Where the speed comes from

Running stock vLLM on off-the-shelf quantized checkpoints wastes performance in several non-obvious places. The project's optimizations document (docs/optimizations.md in the repository) describes nine changes. Four of them account for most of the gain.

Quantizing the embeddings everyone skips

Qwen3.8-27B uses untied embeddings, meaning separate input and output tables of about 2.5 GB each. Public "quantized" checkpoints keep both in bf16, largely because quantizing them is awkward. The prepare scripts convert both to int8, recovering 2.6 GB of VRAM with no measurable quality loss. On a 24 GB card, that is roughly enough memory for one additional user's worth of context.

Exploiting the hybrid architecture

Only 16 of the model's 64 layers are conventional attention layers whose memory grows with context. The remaining 48 are Gated DeltaNet layers, a linear-attention design whose per-conversation state has a fixed size regardless of whether the conversation is 100 or 100,000 tokens long. Stock vLLM allocated that state in fp32, about 150 MB per request, and could not get past 37 simultaneous requests despite a configured limit of 64. The fork stores it in fp16, perplexity stays identical to three decimal places, and all configured slots become usable.

DFlash2: drafting seven tokens in one pass

This change matters most for single-request latency. Speculative decoding pairs a small drafter with the large target model. The drafter proposes several upcoming tokens, and the target model verifies all of them in one forward pass, keeping the correct prefix and regenerating from the first mistake. Verification is much cheaper than generation on a memory-bound GPU, because the weights are read once for several tokens, so each step can yield more than one token.

Qwen's built-in MTP head chains four guesses one after another. The fork replaces it with DFlash2, a five-layer drafter that predicts an entire seven-token block in a single non-autoregressive pass, using hidden states taken from five different depths of the target model. The drafter itself was GPTQ-quantized from 3.85 GB to 1.19 GB so it shares the card without competing heavily for memory bandwidth. The result is roughly three tokens per step instead of one. Because standard speculative decoding accepts or rejects draft tokens so that the final output distribution matches the target model alone, the speed-up is lossless; GSM8K accuracy stays at 96 to 96.5% across all configurations the repository reports.

A draft vocabulary built from the model's own output

The drafter can only propose tokens that exist in its reduced output vocabulary; anything outside it is rejected by definition. The original vocabulary was derived from web text and covered 92% of the tokens this model actually generates. The maintainers collected 5.4 million tokens of the model's own output and rebuilt the vocabulary from those counts, reaching 97.5% coverage. That change alone added about 10% throughput, with no new hardware and no change to the model.

Lookup drafting for text the model is quoting

One further technique explains the most extreme numbers. When the output repeats material already present in the prompt, such as source code under edit or a pasted document, the fork bypasses the drafter and proposes tokens directly from the context. Reproducing a 25k-token document reaches 381 tok/s on the reference hardware. Coding agents spend much of their time reproducing code that appeared moments earlier in the prompt, the ideal case for this technique, and it is one reason agent-harness measurements come out high.

Why doubling the context does not run out of VRAM

The single-user launcher defaults to a 65k context. Switching to CTX=long roughly doubles that to 131k, and it seems reasonable to expect an out-of-memory error on a 24 GB card. Instead the server starts normally and runs only somewhat slower. Several design decisions explain this.

• Weights have a fixed footprint. With W4A16 quantization the model body takes about 15 GB, independent of context length.
• The KV pool is reserved once, in bytes. Before serving any traffic the fork reserves a fixed budget, about 5.2 GiB here, and vLLM logs the result, for example "GPU KV cache size: 136,429 tokens." Because the pool is allocated at startup, it either fits then or the server refuses to start. There is no per-request allocation that can fail halfway through an agent task.
• The long profile uses cheaper tokens, not more memory. The default profile keeps attention caches in bf16, while CTX=long stores them in int8, roughly halving the cost per token. The same 5.2 GiB then holds 136,429 tokens instead of 68,605, doubling maximum context to 131k. Teacher-forced perplexity on an identical passage differs by only 0.2% between the two profiles.
• Most layers do not grow with context. Only the 16 attention layers scale with sequence length; the 48 DeltaNet layers keep their fixed state. Long context is therefore much cheaper here than on a pure transformer, which is a large part of why the model fits on one card at all.

Why the cap is 131,072 and not 136,429

Not all of the pool can go to one request's attention cache. Each resident request also needs its fixed-size DeltaNet state, and DFlash2 adds eight speculative state slots per request on top of that. With four seats available in the long profile, the launcher sets the maximum context at 131,072, about 4% below pool capacity, keeping the remainder for those state pages and for block alignment. This headroom is what backs the no-OOM promise: startup checks the most demanding scenario, a single maximum-length request while all remaining seats are also in use.

What you give up instead

Longer context is paid for in speed, not in crashes. Tokens are stored in int8, and each resident request reserves state pages from a pool now divided among fewer seats. Parallel slots drop from 8 to 4 and decode falls from about 177 to about 122 tok/s. The card does more with the same bytes and charges for it in throughput rather than in exceptions.

Where it falls short: deep single-request context

The stack is at its best with short and medium contexts, which is what agent harnesses mostly send. A single request approaching 100k tokens behaves differently. Measured on one 3090, a single request decoded at roughly 107 tok/s with a short prompt, 78 tok/s around 10k tokens and 38 tok/s around 43k, sagging to roughly 31 tok/s near 100k. The cause is the drafter's acceptance rate, which decays with depth. The repository observes the same effect, with acceptance settling around 0.29 regardless of cache dtype, so this is a property of the model and the context depth rather than an unfixed bug.

For comparison, a llama.cpp-based fork on the same class of card holds a steady 65 to 80 tok/s at 150k context. It has no drafter whose guesses degrade and no verification block that stops paying off, so it is never especially fast but also never slow. The table below sets the two side by side; note that the vLLM short-context range mixes a single-request figure with the multi-slot agent measurement.

| Context depth   | vLLM fork (DFlash2) | llama.cpp ATX fork |
|-----------------|---------------------|--------------------|
| short (<4k)     | 107–177 tok/s       | 65–80 tok/s        |
| ~10k            | ~78 tok/s           | 65–80 tok/s        |
| ~43k            | ~38 tok/s           | 65–80 tok/s        |
| ~100k           | ~31 tok/s           | 65–80 tok/s        |

The practical rule follows directly. If your work is dominated by one very long document read slowly, llama.cpp is the steadier option; see our guide on serving a Qwen3.8 MoE model on RTX 3090s with llama.cpp tensor offloading for that side of the trade-off. If your work is many medium-length coding conversations, the vLLM fork is far ahead.

Why this suits agent harnesses

Harnesses such as Pi, Hermes or DeepSeek Harness do not hold a single conversation. With subagents they generate many parallel threads at once, typically five to twelve, each of medium length, with most of them reusing an identical system prompt and the same codebase context. That is precisely the load this fork was tuned for:

• Aggregate throughput scales with concurrency. Four concurrent streams delivered more than 300 tok/s combined on a single 3090. The repository's reference benchmarks show 279 to 335 tok/s at four concurrent requests and up to about 400 tok/s aggregate at eight with MTP.
• Prefix caching makes shared context nearly free. With PREFIX_CACHE=1, 64 requests sharing a 5.8k-token system prompt completed in 17 seconds instead of 222 in the repository's benchmark, with median latency falling from 95 s to 8 s. After the first request, subagents pay almost nothing for shared instructions. On this hybrid model the cache also stores recurrent DeltaNet state, not only attention KV, which is why a follow-up turn on a 24k-token document takes about 1 second rather than 23.
• Lookup drafting matches agent output. Refactors, rewrites and edits repeatedly quote their input, which is where the stack reaches its 380+ tok/s peaks.

There is a ceiling. Beyond about four resident long-context streams you are limited by how the pool is divided, not by compute. The repository states plainly that eight concurrent long-context requests perform strictly worse than four, describing it as "not a tradeoff, a loss." Designing a harness around roughly four parallel workers at medium depth gets the best out of the card. For an example of what a local Qwen3.8-27B setup can build in practice, see building a local game clone with Qwen3.8-27B and Pi.

Key takeaways

• The speed comes from software: requantized embeddings, fp16 DeltaNet state, a seven-token DFlash2 drafter and a vocabulary fitted to the model's real output. The GPU is unchanged.
• Pin every version the repository pins, install the curand headers, and run verify.sh before trusting any benchmark you take.
• A KV pool reserved at startup turns out-of-memory errors into a boot-time decision, and the long profile buys context by switching to int8 caches at the cost of slots and speed.
• Speculative decoding degrades with context depth, so for single requests far beyond 40k tokens a llama.cpp setup may serve you better.
• Size agent concurrency around four medium-depth workers with prefix caching enabled, and always report throughput together with the workload that produced it.

For more detail, the repository's docs/reproductions folder includes notes on reproducing the non-Docker path on a 3090 step by step.