Serving Qwen3.8-Flash-Next on RTX 3090s with llama.cpp Tensor Offloading

Qwen3.8-Flash-Next is a roughly 180-billion-parameter open model whose quantized files add up to 88GB, yet people report running it on a single RTX 3090 with 24GB of VRAM. On paper that is off by an order of magnitude. It works because of the model's architecture, not because of a clever quantization or an unusually powerful GPU: large parts of the model never need to live on the graphics card at all. This article explains the four design choices that make that possible, then walks through complete llama.cpp configurations for a three-GPU box and a single-GPU machine, including what each important flag does and what throughput you can realistically expect.

Why Flash-Next is worth running locally

Alibaba released Flash-Next in August. At the time of writing, its published results place it ahead of Qwen3.8-27B, DeepSeek V4 Flash (0731) and, on several agentic benchmarks, Opus 4.6, while activating only 6 billion parameters per token. Benchmark standings change quickly, so check current evaluations before treating those comparisons as settled. The more durable story is how the model is structured, because that structure is what decides which hardware can serve it.

Four architectural choices that move work off the GPU

Flash-Next combines four ideas. Each one shifts some costly work from expensive hardware to cheap hardware, or removes it entirely.

Ultra-sparse mixture of experts

The model has 48 layers, and each layer holds 512 expert sub-networks. Any single token passes through only 11 of them: 10 chosen by a router that inspects the token and picks the most suitable specialists, plus 1 shared expert that every token uses without routing. In other words, about 2% of the experts do the work for a given token.

A useful picture is a hospital with 512 specialists on call and one general practitioner who is always in the room. Each patient gets ten consultations matched to their symptoms, and the other 501 specialists are not involved. The key insight for local inference is that "on call" only means "reachable". A specialist does not have to be sitting in the GPU to be available.

The shared expert exists because purely routed designs tend to lose general knowledge that every token needs. Putting that knowledge in an always-on generalist lets the routed experts specialize more aggressively. Most current MoE designs include one, and its parameters are part of the 6B active figure.

Gated DeltaNet plus Qwen Sparse Attention

A standard transformer keeps a key/value entry for every token it has processed. That KV cache grows linearly with context and becomes enormous at long lengths. Flash-Next largely avoids this. Three out of every four layers use Gated DeltaNet, which compresses history into a small state of fixed size. The remaining layer in each group uses Qwen Sparse Attention, which reads from a fixed budget of 512 blocks, 2,048 tokens, whether the context holds 32K or a million tokens.

The practical effect is dramatic: at 178K context, the KV cache in the configurations below takes about 6GB, where a comparable dense model would need around ten times as much. That is the reason a 24GB card can hold a long conversation at all.

A gated, multi-lane residual stream

Classic transformers route everything through one residual stream from the first layer to the last, and in deep networks early features tend to get diluted along the way. Flash-Next widens that path into four parallel lanes, with learned gates controlling what enters and leaves each lane. During training, one lane ended up specializing as a long-range channel that carries early information deep into the network. It is a small architectural change that adds capability without meaningful cost.

An n-gram embedding table

The fourth piece is a table of 51 billion parameters, more than the rest of the model combined, that performs no matrix multiplications at all. It is a lookup structure, and it is the main reason a single gaming GPU is viable.

The n-gram table: knowledge that needs no GPU

In a normal language model, each token is mapped to an ID and the model looks up that ID in an embedding table: one row per token. A single-token row carries very little information about context. Given a phrase such as "dostoevsky wrote the brothers", the embedding for "the" says nothing about Karamazov; the model has to infer the continuation with its expensive layers, and it has to do so again every time that pattern appears.

Flash-Next adds a second table whose rows are keyed by short phrases instead of single tokens. Conceptually, the entries look like the sketch below: a hashed phrase on the left and the kind of hint its learned vector encodes on the right.

Drawer "born on october"     → hint: 1985, Honolulu, singer
Drawer "dostoevsky brothers" → hint: Karamazov, novel, 1880
Drawer "def main("           → hint: python entry point

During reading, the model hashes the most recent few tokens, fetches the matching row and gets a precomputed hint essentially for free. Nobody wrote these rows by hand. Throughout training, whenever a phrase was followed by a particular continuation, its row was adjusted slightly in that direction, so after trillions of tokens each row approximates what usually comes next. The table holds about 20 million bigram and trigram entries, and its output is injected once, early, at layer 2. It is, in effect, memorization of common phrasing, and a great deal of real text is common phrasing.

The difference between the two kinds of knowledge in the model is what makes the hardware split possible. The comparison below summarizes it.

|                  | Neural network          | N-gram table            |
|------------------|-------------------------|-------------------------|
| Work per token   | Matrix math (expensive) | Drawer lookup (no math) |
| Needs the GPU?   | Yes, every millisecond  | No — CPU can fetch it   |
| Lives in         | VRAM                    | RAM. Even SSD.          |

The decisive property is that the row to fetch depends only on the input text, not on any hidden state inside the network. As soon as a prompt is tokenized, the CPU knows exactly which rows will be required, so it can prefetch them while the GPU is still busy with earlier layers.

That lets the model be spread across the memory hierarchy according to what each tier does well:

• VRAM (fast, costly): the roughly 6B parameters that are actually computed for each token.
• System RAM (moderate, cheap): idle expert weights and the 29GB n-gram table.
• NVMe storage (slow, cheapest): overflow, paged in when touched.

The GPU stops being the place where the whole model is stored and becomes the place where the active computation happens.

A three-GPU configuration for daily use

Consider a server built from three RTX 3090s (72GB of VRAM in total), 48GB of DDR4 system memory and a PCIe Gen3 NVMe drive holding the weights. Nothing about it is workstation-class; it is the kind of machine many developers assemble from older gaming cards.

The model file used here is the UD-IQ4_XS build from unsloth's GGUF repository for this model on Hugging Face, about 88GB split across three shards: roughly 59GB of backbone weights plus the 29GB n-gram table. Support for this architecture is recent, so pull the latest llama.cpp source and rebuild before trying it.

The command below launches llama-server across three of the machine's GPUs. Most flags are routine (host, port, sampling parameters, batch sizes, context length), so pay attention to the offload flags, the split flags and the load mode, which are discussed right after.

CUDA_VISIBLE_DEVICES=0,2,3 CUDA_SCALE_LAUNCH_QUEUES=4x llama-server \
  -m /mnt/data_2t/ai_models_all/llm_hf_models/unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  --alias Qwen3.8-Flash-Next \
  --jinja \
  --metrics \
  --host 0.0.0.0 \
  -ngl 99 \
  --batch-size 4096 \
  --ubatch-size 512 \
  --flash-attn on \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 \
  --split-mode layer \
  --tensor-split 0.9,0.95,1.0 \
  --fit off \
  --main-gpu 0 \
  --ctx-size 178000 \
  --parallel 1 \
  --image-min-tokens 1024 \
  --reasoning-format none \
  --timeout 1200 \
  --ctx-checkpoints 8 \
  --port 8082 \
  --load-mode mmap \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ot '^per_layer_token_embd\.weight$=CPU' \
  --reasoning-effort low \
  -t 14

Pinning the n-gram table to system RAM

The -ot '^per_layer_token_embd\.weight$=CPU' override tells llama.cpp that any tensor whose name matches the regular expression should be placed in CPU memory rather than VRAM. The n-gram table is stored under the tensor name per_layer_token_embd.weight, so this single line keeps all 29GB of it out of the GPUs. When a token needs a row, the CPU fetches it and passes the result on, which is exactly the prefetching arrangement described above. The anchors ^ and $ matter: they ensure the pattern matches only that tensor and does not accidentally move other weights.

Letting the OS manage residency with mmap

--load-mode mmap memory-maps the model file instead of reading it into RAM up front. The operating system then keeps frequently touched pages in whatever page cache is available and leaves rarely touched pages on the SSD. With 48GB of RAM and a 29GB table, that is the right trade: the kernel decides what deserves to stay resident. On a machine with plenty of memory, such as 128GB, switching to none makes more sense, because the whole file is read once and there are no page faults afterward.

Splitting layers across cards

--split-mode layer combined with --tensor-split 0.9,0.95,1.0 divides the 59GB backbone into contiguous groups of layers, one group per GPU, with slightly uneven proportions so the first card keeps headroom for its extra duties as --main-gpu. -ngl 99 puts all 48 layers on GPUs, around 20GB per card. The KV cache, quantized to q8_0 via --cache-type-k and --cache-type-v, fits next to the weights and covers 178K tokens in about 6GB.

Measured throughput on three cards

On this setup, the reported numbers are shown below.

Decode:  30-50 tokens/second
Prefill: 400-700 tokens/second
Context: 178,000 tokens

That is faster than reading speed for a model in the frontier-adjacent class, on used consumer cards in a mid-tower case.

A single-GPU configuration and its limits

One 3090 is enough if you meet one condition: at least 64GB of system RAM. The n-gram table and the offloaded experts need somewhere to live, and system memory is usually the cheapest upgrade for a local AI machine.

The command uses the same GGUF file. The notable differences are the new -ncmoe flag, a single visible GPU, --load-mode none instead of mmap, and fewer CPU threads.

CUDA_VISIBLE_DEVICES=0 llama-server \
  -m /mnt/data_2t_3/ai_models_all/unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  --alias Qwen3.8-Flash-Next \
  --jinja \
  --metrics \
  --host 0.0.0.0 \
  -ngl 99 \
  -ncmoe 38 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --flash-attn on \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 \
  --ctx-size 178000 \
  --parallel 1 \
  --image-min-tokens 1024 \
  --reasoning-format none \
  --timeout 1200 \
  --ctx-checkpoints 8 \
  --port 8083 \
  --load-mode none \
  -ot '^per_layer_token_embd\.weight$=CPU' \
  --reasoning-effort low \
  -t 8

What -ncmoe 38 does

-ncmoe 38 keeps the expert weights of the first 38 layers in CPU RAM, while the last 10 layers keep their experts in VRAM. That places about 43GB of expert weights in system memory. The attention and shared components of every layer still run on the GPU because of -ngl 99.

Parking so much in RAM remains workable because of sparsity. Each token activates 10 of the 512 experts per layer, so it only touches around 1.3GB of expert weights in total. Experts that live in RAM are computed on the CPU where they sit, without copying them to the GPU, and experts in VRAM run natively on the card. The CPU is much slower than a 3090, so there is a real penalty, but it scales with the roughly 2% of weights each token touches rather than with everything stored.

Because RAM holds so much here, --load-mode none reads the file fully at startup instead of relying on page faults, which is another reason the 64GB minimum matters.

Tuning for other cards

The same command works on an RTX 4090 or 5090; only -ncmoe needs adjusting to the available VRAM. The suggested starting points are listed below.

| GPU        | VRAM | Suggested -ncmoe | Experts in RAM |
|------------|------|------------------|----------------|
| RTX 3090   | 24GB | 38               | ~43GB          |
| RTX 4090   | 24GB | 38               | ~43GB          |
| RTX 5090   | 32GB | 28-30            | ~32GB          |

Each step you lower -ncmoe moves one layer's experts, about 1.1GB, back onto the GPU. After the model loads, check nvidia-smi and aim to leave around 500MB of VRAM free; running the card completely full invites out-of-memory errors when context grows.

Setting realistic expectations

On a single 3090, decode runs at 15 to 20 tokens per second. That is usable, but only just: fast enough to read along, slow enough that long generations feel sluggish, because part of the computation genuinely happens on the CPU in system RAM. As a proof of concept, or to get value from a machine you already own, running a 180B-class model at reading speed on one gaming card is remarkable.

For an assistant you keep open all day for coding and everyday work, three or four 3090s are what make the experience comfortable. The gap between roughly 18 and 40 tokens per second is the gap between a demo and a daily tool. If you are exploring smaller local Qwen models for agentic coding instead, the walkthrough on building a local game with Qwen3.8-27B shows a lighter-weight setup.

Multi-token prediction: the next speedup to watch

Flash-Next ships with a trained multi-token prediction (MTP) head, a small extra module that proposes several upcoming tokens at once so the main model can verify them in parallel. It is speculative decoding with a draft component trained end to end for this exact model. On vLLM, it has been reported to deliver around 2.5x decode speedups on real prompts.

At the time of writing, llama.cpp supports the Flash-Next architecture itself but its MTP draft path does not yet cover this model's head. Support exists for sibling models, so the change should be modest, but check the current llama.cpp release notes before assuming it is available. Once it lands, the three-GPU configuration's 30 to 50 tokens per second could plausibly rise to somewhere around 60 to 100 on the same hardware, purely through a software update. Treat that as an estimate until you can measure it.

Key takeaways

• Flash-Next fits consumer hardware because of design, not brute force: ultra-sparse MoE, fixed-size attention state and a lookup-only n-gram table all reduce what must sit in VRAM.
• Anything whose access pattern depends only on the input text, like the n-gram table, can live in system RAM and be prefetched by the CPU; -ot with a tight regex is how you place it there.
• -ncmoe is the main dial for single-GPU setups: lower it until VRAM is nearly full, leaving a small safety margin.
• Choose mmap when RAM is tight and you want the OS to manage residency; choose none when RAM is plentiful and you want predictable latency after load.
• System RAM matters as much as the GPU for this class of model; 64GB is the practical floor for one card.
• Expect roughly 15 to 20 tokens per second on one 3090 and 30 to 50 on three, with MTP support in llama.cpp as the most likely next gain.