Run Qwen3.8-Flash-Next on a 64GB MacBook: Files, Quants and llama.cpp Flags

Qwen3.8-Flash-Next has files that add up to 88 gigabytes, and it runs well on a stack of three RTX 3090s. After days of regular use, the impression is consistent: it thinks less than Qwen3.8-27B yet handles nearly every task thrown at it. You get less rambling, fewer loops and more actual answers. For a model with 6 billion active parameters, that mix of brevity and competence is unusual.

Gaming GPUs in a rack are one thing, but the architecture change behind this model raises a different question. Could it run on a MacBook? That means one laptop with 64 GB of unified memory, the kind of machine you carry on a plane and wouldn't normally trust with a model of 180 billion parameters.

Two days of tests on an M4 Max and an M1 Max, both with 64 GB, answered it: yes. Expect 25 to 30 tokens per second for decoding and around 400 tokens per second for prefill, with no CPU offload at all.

"No offload" is the phrase that matters. A three-3090 server has to stream experts through system RAM, while the laptop just loads the model and runs it. The M1 Max is a six-year-old chip from the first generation of Apple silicon, and it still reaches about two-thirds of the M4 Max's speed. That's completely usable.

What follows covers how to set it up.

Which file to download: not the famous one

There's a trap here. The most popular quantization of this model is Unsloth's GGUF. It's the right file for a Linux server and the wrong one for a Mac, and Unsloth's own README concedes as much by asking for 75 to 96 GB of memory on macOS.

The cause is a quirk in how Apple's graphics driver treats memory. It's worth understanding, since it explains everything that comes after.

llama.cpp loads models by memory-mapping the file. The operating system hands over a pointer, and pages get read on demand. On Linux that's perfectly lazy: pages you touch are loaded, and pages you don't touch can stay on disk or be evicted when something else needs the space.

Metal, Apple's graphics API, adds a complication. When llama.cpp gives Metal a memory-mapped shard, the entire shard gets wired. That means it's pinned in RAM, guaranteed to be present, and can't be evicted. Unsloth's shards mix the model weights with the n-gram table inside the same files. So when Metal asks for a single weight tensor, it also pins gigabytes of table. On a 64 GB Mac the result is an immediate out-of-memory error, before one token has been produced.

Recall what that n-gram table is. It's a 29 GB "second filing cabinet" that does no math at all and only gets consulted, roughly 2.7 kilobytes per generated token. At 30 tokens per second that comes to about 3 MB/s of reads, a load a USB stick from 2012 could handle. The table was never meant for fast memory. It only has to be reachable.

Unsloth's layout forces it into fast memory by accident. On Linux you can override placement per tensor, but Metal offers no such control. The fix therefore has to be structural: put the table into a shard that holds no GPU tensors, so nothing forces it to be pinned.

That's exactly what the AtomicChat M64 builds do. Each of their quants is repacked so that shard 2 contains only the n-gram table. The table streams from the SSD page by page through the OS cache, the way the architecture intended, and only the real compute weights are wired.

This also explains the "M64" label. It names a layout family inspired by 64 GB Macs, a split-shard repack, and it doesn't promise a size. All three of their builds carry the suffix, including the 110 GB one meant for 128 GB machines. What decides whether a build fits your computer is the bits-per-weight number, not the suffix.

Quality gives you another reason to prefer them. The builder started from the BF16 originals, used its own published importance matrix, and then measured every build against the reference model's real outputs. The raw KLD logs and reference logits are public, which is more transparency than most publishers offer.

Which quant: 3.84 or 4.27?

Two builds matter on a 64 GB Mac. Both were used as the main model for full days. Measured on the M4 Max with 64 GB, the results were:

| Build                     | Decode  | Prefill | Max context | Resident RAM |
|---------------------------|---------|---------|-------------|--------------|
| AD-3.84bpw-IQ4_XS-M64     | 28-32/s | ~400/s  | 220,000     | 45.8 GB      |
| AD-4.27bpw-Q4_K_M-M64     | ~25/s   | ~350/s  | 144,000     | 54.5 GB      |

The trade-off is unusually clean. The 4.27 gives clearly better quality, with a divergence from the full-precision model of less than half that of the 3.84. In exchange it costs speed and 8.7 GB of precious resident memory, which is why its context tops out at 144K and not 220K.

For daily use the 3.84 is the better pick, for one reason: context. A 220,000-token window at 30 tokens per second is a nicer companion than a 144,000-token window at 25. Loading whole codebases, entire collections of notes and complete project histories is what makes the window noticeable all day long, while the quality gap only appears in edge cases that come up once a week.

Choose by workload. If you feed it long documents all the time, take the 3.84. If you do precise reasoning over shorter inputs and the quality gap bothers you, take the 4.27. It fits in 64 GB, but the Mac becomes a single-model machine, completely.

The names are lies. Good lies, but lies.

This part caused a stir on Reddit, so here's the plain version: neither "IQ4_XS" nor "Q4_K_M" in those filenames describes what's inside.

AtomicChat's builds are mixed recipes. Different parts of the model receive different quantization types, and the filename carries only the closest standard type to the file's average bits per weight, which the model card says outright. Here's what each build really contains.

AD-3.84bpw-IQ4_XS-M64

| AD-3.84bpw-IQ4_XS-M64  | Real type     | Share | Why                     |
|------------------------|---------------|-------|-------------------------|
| n-gram table           | ~6-bit        | 46%   | memorized hints, cheap  |
| MoE expert body        | IQ1_M (1.75b) | ~40%  | 512 experts, 2% active  |
| dense layers, heads    | Q6_K/Q8_0     | ~14%  | the parts that think    |

AD-4.27bpw-Q4_K_M-M64

| AD-4.27bpw-Q4_K_M-M64  | Real type      | Share | bpw contribution |
|------------------------|----------------|-------|------------------|
| n-gram table           | Q5_1           | 41%   | 1.74             |
| ffn_gate/up experts    | IQ2_S (IQ3_S*) | 29%   | 1.24             |
| ffn_down experts       | IQ4_NL         | 24%   | 1.03             |
| everything else        | Q8_0           | 5%    | 0.23             |

On top of that, layers 0 to 3 and 40 to 47 get the higher-precision IQ3_S. The builder's importance-matrix statistics showed activation energy piling up in the last layers, and spending bits evenly would waste them on layers 4 through 39.

Does that count as cheating? Reddit commenters said yes. The numbers point the other way, because they show a calibrated recipe doing exactly what it was designed to do. Compared with the full BF16 model, the 4.27 build picks the same top token 89.5% of the time, and its perplexity is only 2.6% higher (a ratio of 1.026). Bits weren't thrown away, just relocated. An expert that a token touches roughly 2% of the time can be squeezed hard, whereas the dense layers that every token passes through stay large. The architecture itself makes the same bet, and here it's applied one level down, to the quantization.

The command

First pull and rebuild llama.cpp. Support for the qwen4exp architecture reached mainline on August 27, and things are moving quickly:

cd ~/github_repos/llama.cpp
git pull
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

Next, raise the GPU memory ceiling. macOS limits it to about 48 GB on a 64 GB machine, and the setting resets at every reboot:

sudo sysctl iogpu.wired_limit_mb=57344

Then launch the server:

llama-server \
  -m /Users/andrewzhu/ai_models/Qwen3.8-Flash-Next-AD-3.84bpw-IQ4_XS-M64/Qwen3.8-Flash-Next-AD-3.84bpw-IQ4_XS-M64-00001-of-00028.gguf \
  --alias Qwen3.8-Flash-Next \
  --jinja \
  -ngl 99 \
  -c 168000 \
  -fit off \
  --host 0.0.0.0 --port 8083 \
  --metrics \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --reasoning-format none \
  --parallel 1

Four details in that command matter more than they appear to.

-fit off is required. llama.cpp's automatic memory fitter misjudges this architecture and can't allocate. Without the flag the server dies at startup on a machine with plenty of memory, and the failure looks exactly like "not enough RAM," which sends people on a frustrating hunt. The flag wasn't needed on the Linux rigs, but the Mac needs it.

There's no -ot override. A Linux server needs the n-gram table pinned to the CPU, done with the regex -ot '^per_layer_token_embd\.weight$=CPU'. The Mac can skip that. With the split-shard layout the table goes to the host on its own, because nothing in its shard makes Metal pin it. The shard design already solves the problem the regex was solving.

There's no --load-mode flag either, so the default mmap stays. On a 128 GB Linux machine you might disable mmap and read everything once. On the Mac, mmap is the feature. It lets the OS page the 39 GB table in and out of the SSD as tokens consume it. Turning it off would try to make all 85 GB resident, and the laptop only has 64.

Finally, -ncmoe is absent. A single-3090 setup needs about 43 GB of experts parked in system RAM. On the Mac, the whole in-memory portion, 45.8 GB, fits under the 56 GB wired limit, so nothing needs offloading. Unified memory simply holds it all. This is the first machine where the answer to "what goes where?" is "everything that matters, wherever it matters, at once."

The old laptop does it too

The same setup also works on an M1 Max with 64 GB. That's Apple silicon's first serious push, released six years ago, when 64 GB in a laptop was a spec-sheet flex and not an inference requirement.

It reaches about two-thirds of the M4 Max's speed with the same model, the same file and the same flags. One workable arrangement is to move daily work onto the M1 Max and leave the M4 Max with a single job: this model, always loaded, answering at 30 tokens per second on the next desk.

Two caveats apply, and they're worth stating without sugarcoating.

The machine is taken. With the model loaded, memory pressure stays high, so close the browser tabs and chat apps, because this Mac now does one thing. That's the reason to own two.

The SSD is working. The table streams constantly. Bandwidth is modest, but it adds real wear over months, and the fans will spin on long sessions. Putting the model files on an external USB4 NVMe enclosure is a kind gesture toward the internal drive.

Zoom out for a moment, though. A laptop bought for writing code and editing video now runs a model that trades blows with frontier AI. The model has 180 billion parameters, and it lives on hardware whose cooling was never tuned for tensor workloads. There's no datacenter involved, no cluster and no API invoice. The architecture came to the hardware, not the other way around.

That's the real story of Qwen3.8-Flash-Next. Big models didn't get smaller. Clever models got welcome.