Doubling Qwen3.8-Flash-Next Decode Speed with MTP on RTX GPUs

Plenty of people say Qwen3.8-Flash-Next doesn't live up to its benchmark scores. Used daily next to Qwen3.8-27B, it behaves differently from that reputation. It doesn't burn a lot of thinking tokens for its size, it does what you ask without arguing, it holds the thread of a long session, and it recalls earlier history correctly. Compared with DeepSeek V4 Flash 0731 it's the nicer everyday assistant, and it's easy to end up making it your default.

The one complaint is speed. On three RTX 3090s plus an RTX 3070 Ti, decode sits at about 30 tokens per second. That's a slow, even drip, and a fast typist outruns it.

There's a known fix, and it's called MTP (multi-token prediction). It's the DeepSeek flavor of speculative decoding: a lightweight draft head bolted onto the model proposes several upcoming tokens, and the full network just verifies them. Checking costs far less than generating, so five tokens can land in the time one normally takes.

The snag is that llama.cpp hasn't merged MTP for this model. After a lot of trial and error, one combination of weights, fork and flags roughly doubled throughput with the same model files, the same server and the same four GPUs. The recipe is below, and after it comes the part about why shipping this feature has been so messy.

Get the right weights

For the main model, grab the 4.27bpw Q4_K_M build from the AtomicChat GGUF repository.

That quant was made with MacBooks in mind, since it's sized for Apple silicon's unified memory. It still runs fine on RTX cards, and it needs less VRAM than the comparable Unsloth quant. Sometimes a build packaged for one audience turns out to suit another.

The repo also has a smaller 3.84bpw IQ4_XS variant. Both got tested, and the smaller one is clearly worse. Saving 0.4 bits per weight isn't worth what you lose.

The MTP head ships as its own sidecar file, which you'll find in Unsloth's sidecar folder. The Q8_0 head was used here. If VRAM is tight, the Q4_K_M head works too.

Build the fork

The working code is on the qwen4exp/mtp branch of danielhanchen's llama.cpp fork. Unsloth's own documentation sends you to that same branch, which tells you where this feature really lives right now.

It's an ordinary CMake build: install the packages, clone the branch, configure.

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch qwen4exp/mtp https://github.com/danielhanchen/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split

Launch the server

Here's a launch command that works as-is. It uses four GPUs, a 200K context window, and an MTP draft that can propose up to 5 tokens per step.

CUDA_VISIBLE_DEVICES=0,1,2,3 llama-server \
  -m .../Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
  --alias Qwen3.8-Flash-Next \
  --jinja \
  -ngl 99 \
  -c 200000 \
  --split-mode layer \
  --tensor-split 0.9,0.22,1.0,0.75 \
  -fit off \
  --host 0.0.0.0 --port 8082 \
  --metrics \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 \
  --reasoning-format none \
  --reasoning-effort low \
  --parallel 1 \
  -md .../mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type draft-mtp --spec-draft-n-max 5

Most of the speedup comes from a handful of flags:

• --spec-type draft-mtp turns on the MTP draft head. Until upstream merges the work, only the fork knows this flag.
• --spec-draft-n-max 5 lets the draft guess up to five tokens ahead. With this model the acceptance rate was high enough to make five worthwhile.
• -md ... points to the sidecar file holding the MTP head, because the main quant doesn't include it.
• --parallel 1 isn't optional. There's an open bug where MTP combined with multiple parallel slots lets one conversation's content bleed into another. A single slot keeps the output clean.
• --split-mode layer with --tensor-split 0.9,0.22,1.0,0.75 spreads layers over four unequal cards: three 24GB 3090s and one 8GB 3070 Ti. Those ratios were tuned to that exact mix, so expect to tweak them for your hardware.
• -fit off makes llama.cpp load exactly what you specify instead of making its own memory decisions.

What you get

On that machine the before and after looked like this:

Before MTP    After MTP
Decode speed          ~30 t/s       60–80 t/s
Model weights         same          same
Server hardware       same          same
Quality               same          same
Total VRAM used       76GB          ~80 GB

You feel the difference more than you measure it. At 30 tokens per second you're watching a model type. At 60 to 80 you're having a conversation with it. Same weights, same box, same morning, and output quality didn't budge.

Anything with more than 80 GB of VRAM can run this exact setup and should see the same gain.

Why isn't this merged yet?

Because open source sometimes works out badly, even when the code exists.

MTP for Qwen3.8-Flash-Next isn't an unsolved research problem. vLLM and SGLang supported it from day one. Forks of llama.cpp ship it, and Unsloth bundles it into its desktop app. Those demo videos running at 170 tokens per second are real.

Upstream llama.cpp has had five separate pull requests try to land it, and none got in. One sat as a draft with merge conflicts and not a single review. Another was closed after a contributor submitted someone else's code without that person's permission. A maintainer explained that crediting somebody isn't the same as getting their consent, and the best working implementation has been orphaned ever since. A third, the cleanest of the bunch, was closed by its own author last week with the message that it wasn't being pursued upstream at the moment. You can read the stalled PR #27836 and the abandoned PR #28610 yourself.

Three hard problems surround those PRs. The first is a GGUF format dispute: three competing layouts for where the MTP head is stored. The second is the correctness bug with parallel slots, tracked in issue #28286. The third is backend regressions, since the same feature that doubles speed on CUDA is a net loss on Vulkan.

None of that makes the maintainers the bad guys. Their rule is that mainline must never break anyone's production server, and speculative decoding on a model with hybrid state is probably the scariest diff you could send them. Still, a feature that works and a feature that ships are two separate engineering artifacts, and the gap between them is measured in weeks of waiting.

Should you try it?

An afternoon of building made a daily-use model twice as fast, and the commands above are the whole recipe. If you've got 80+ GB of VRAM and a Qwen3.8-Flash-Next that dribbles out tokens, it's worth the effort. Unsloth's own MTP guide covers the vendor side.

Whether mainline catches up is anyone's guess. The prerequisites are landing and the format is settling, so someone will probably carry the right implementation to the right branch with its authors' blessing. Until then the frontier lives in forks, so clone accordingly.