Qwen3.8-27B on One RTX 3090: 262K Context at 65-80 Tokens per Second

A 27-billion-parameter dense model, the full 262K context window, one aging consumer GPU, and a generation speed that feels like talking to a person. Until recently that sentence sounded like marketing. With Qwen3.8-27B and a specialized llama.cpp build, it's now a reproducible setup on a single RTX 3090.

Some background on why it matters. Qwen3.8-27B is a strong daily-use model, clearly ahead of the previous Qwen3.6 generation. Running it at good quality used to be expensive, though. The Q8 version, which is nearly lossless, wants roughly 28 GB of VRAM, so no single consumer card can hold it. Many home setups paired two RTX 3090s just to serve one such model. The recipe below gets a much lighter configuration working on a plain 24 GB card released in 2020.

The numbers first

Measured on separate machines, one model and one GPU each:

| GPU            | Decode Speed    | Prompt Prefill   |
|----------------|-----------------|------------------|
| RTX 3090       | 65–80 tok/s     | 1,000–1,300 t/s  |
| RTX 3090 Ti    | 70–100 tok/s    | 1,200–1,700 t/s  |
| RTX 5090       | 150–180 tok/s   | 3,000–4,000 t/s  |

Look at the top row again: 65 to 80 tokens per second from a single RTX 3090. That's quicker than most people read and well above the roughly 30 tok/s where a chat starts to feel natural, and it comes from a card NVIDIA shipped in 2020 while running a dense 27B model at full context. The 3090 Ti reaches about 100, and the 5090 lands between 150 and 180, which would have counted as datacenter throughput a few years back.

That's far beyond what earlier local setups delivered. So what changed?

Three steps to reproduce it

Step 1: build the specialized llama.cpp fork

A fork of llama.cpp called llamAmpere targets Ampere GPUs and newer ones, and tunes them aggressively:

git clone -b main https://github.com/JakeATX/llamAmpere.git
cd llamAmpere
cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DGGML_CUDA_FA=ON \
      -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_NATIVE=ON
cmake --build build-sm86 -j8 --target llama-server

The 86 there is the CUDA architecture code for RTX 30-series cards. On a newer generation, replace it: Ada (40-series) uses 89 and Blackwell (50-series) uses 120. The fork itself works on all of them, and only this compile flag differs.

Step 2: fetch the custom quantized model

Speed depends on a purpose-made quantization named ATX-4-XS. It's published as jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF on Hugging Face and weighs 14.5 GB. You can download it two ways. With the Hugging Face CLI:

huggingface-cli download jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF \
    --local-dir ~/models/qwen38-27b-atx

Or with plain wget into a folder of your choice:

mkdir -p ~/models/qwen38-27b-atx
wget -O ~/models/qwen38-27b-atx/Qwen3.8-ATX-4-XS.gguf \
  "https://huggingface.co/jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF/resolve/main/Qwen3.8-ATX-4-XS.gguf"

Change the paths if your models live elsewhere, for instance on a dedicated storage drive.

Step 3: start the server

Launch with this command, adjusted for your own directories:

CUDA_VISIBLE_DEVICES=0 GGML_Q8_TURBO3_MMA_FUSED=1 \
/home/andrewzhu/storage_1t_1/github_repos/llamAmpere/build-sm86/bin/llama-server \
    -m /mnt/data_2t_3/ai_models_all/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF/Qwen3.8-ATX-4-XS.gguf \
    --alias Qwen3.8-27B-GGUF \
    -c 262144 -b 4096 -ub 1024 -t 8 -tb 8 -ngl 99 -fa on -ctk q8_0 -ctv turbo3 \
    --parallel 1 --jinja --fit off \
    --cache-prompt --cache-ram 8192 --ctx-checkpoints 24 --checkpoint-min-step 10240 \
    --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0 \
    --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \
    --host 0.0.0.0 --port 8082 \
    --spec-draft-vocab-map docs/mtp-vocab/atx_65536.txt

Two details cost people speed when skipped. One is the GGML_Q8_TURBO3_MMA_FUSED=1 environment variable, which the fork's own notes describe as worth 13% and the line everyone forgets. The other is the vocab-map file at the end. That path, docs/mtp-vocab/atx_65536.txt, sits inside the fork's repository, which you've already cloned.

Why it works

Four separate ideas stack here. None is magic, and together they show careful engineering.

1. A build rewritten for one chip family

Stock llama.cpp writes its CUDA kernels to be broadly correct across many GPU generations. This fork goes the opposite way and rewrites the hot paths for one architecture, then tunes them until nothing more can be squeezed out.

Two examples. Qwen3.8-27B isn't a pure transformer: 48 of its layers hold Gated DeltaNet recurrent state instead of attention. The fork reworked those kernels so each GPU warp owns four state columns, not one, which keeps the hardware pipeline fuller. That gave roughly 5 to 6% on both prefill and decode.

The remaining 16 layers do real attention. Their key data is stored compressed, and the earlier decompression path used only about 10% of the memory bandwidth, like drinking through a narrow straw. The rewrite stages data with asynchronous copies and fixes the read pattern, which makes long-context decoding up to 11% faster.

Less glamorous fixes mattered just as much. Two memory-allocation corrections freed 2.8 GB of VRAM: one buffer was allocated twice, and the speculative drafter reserved its own compute buffers when it could share the main model's. Those 2.8 GB decide whether 262K context fits on the card.

2. A quantization that spends bits wisely

A quantization scheme sets how many bits each weight receives. A naive one gives every tensor the same budget, say 4 bits. But tensors differ in how badly rounding hurts them.

The ATX-4-XS recipe came from measuring which tensor roles do real damage when compressed, such as attention projections and output heads. It uses IQ4_XS as the base format and upgrades chosen roles to Q5_0, Q6_K or Q8_0. The outcome is a 14.5 GB file, smaller than a standard Q4_K_M build of the same model, which pulls fewer bytes from memory on each pass while staying close to full quality for everyday work.

Smaller means faster because generating one token at a time is bound by memory bandwidth: each token requires reading the weights once from GPU memory. Fewer bytes per token yields more tokens per second. It took careful measurement of which bits to keep for this to come without a quality bill.

3. Turbo3: compact short-term memory

While generating, a model keeps a KV cache, a record of everything it has read and written so each new token can look back. At 262K tokens that cache is huge. With standard formats it would swallow most of your VRAM before the weights even load.

This stack, inherited from the TurboQuant+ line of work, stores attention keys at q8_0 (nearly lossless) and stores values rotated and compressed to about 3 bits. That's the Turbo3 label. The asymmetry is intentional: keys work like addresses, so errors misfile everything, while values are content that tolerates shorthand.

A fair caveat applies. This isn't Q8 fidelity throughout. For coding, writing and agent-style daily use no quality wall shows up, but if your workload demands bit-exact reproduction at temperature 0, stay with Q8 on two cards.

4. MTP and the vocab map: cheap guessing ahead

The largest gain comes from speculative decoding using MTP (multi-token prediction). The model carries a small draft head of its own that suggests upcoming tokens, while the big model validates those suggestions in parallel. In this fork the depth is 3, meaning three tokens get guessed, all three are verified in one pass, and whichever guesses were correct are kept.

What changed is the verification. Older llama.cpp accepted a draft token only when it matched the target's top choice exactly, which is strict and wasteful. This stack uses proper rejection sampling instead. Each draft is accepted with probability min(1, p/q), where p is the main model's probability for the token and q is the drafter's. In plain terms, it's fair dice with a provably identical final distribution to normal generation, so output arrives faster at the same quality.

That leaves the map file. Why does the launch command need a vocabulary map at all?

Qwen3.8-27B has a vocabulary near 248,000 tokens. Each draft step multiplies the hidden state by a huge projection matrix covering all of them, about 1,152 × 248,320, which is some 286 million multiply-accumulates per drafted token. Yet real text hardly touches most of that vocabulary. Measured on real output, a shortlist of the top 65,536 tokens accounts for 99.59% of the drafter's proposals.

So the map trims only the drafter's projection, and the main model still chooses from the entire vocabulary:

Full vocab draft head:    1,152 × 248,320 ≈ 286M MACs per draft token
Shortlisted (65K):        1,152 ×  65,536 ≈  75M MACs per draft token

That's roughly a 75% reduction in the drafter's costliest matrix. Because drafting happens several times for each generated token, it was the single biggest speedup in the v0.2 release, the jump from about 65 to about 75 tok/s. Without the map, every guess is billed in full.

Combine all four layers: rewritten kernels, smarter bit allocation, compressed memory shorthand and cheap lookahead. A 2020 GPU then beats what specialized hardware offered in 2023. The RTX 3090 didn't improve. The community simply got very good at squeezing it.

The bigger picture

Consider what happened. A single engineer, with frontier models helping out, took a few days to get a consumer GPU from 2020 serving a leading open model roughly triple the speed of the official stack, then released the kernels, the quantization and the recipe for free.

That's the state of local AI in 2026. Each month, something that needed two cards last quarter fits on one. It also changes the argument about who controls AI. Hardware already sitting in people's homes can run modern models offline, with no API key, no per-token bill and no outside permission. Rules that drive up inference costs or restrict who can buy compute would fall hardest on exactly this kind of independent work.

Whatever you think about those policy debates, the practical lesson is simple. The weights live on your own drive, and efficiency work in the open-source community, delivered as kernels and commits instead of announcements, keeps lowering the price of running your own model. The code is in the llamAmpere repository, and the model is jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF.