How Much Memory Do You Need for Local AI? Speeds from 16 GB to 512 GB
Measured local LLM speed and quality across 16, 32, 128, 256 and 512 GB machines, what each class can run, and which GPUs, Macs and mini PCs fit where.
Артыкул апублікаваны на англійскай мове.
The first question in any local AI build is how much memory you need, and the honest answer is that memory decides which models load while the hardware behind it decides how fast they run. Below are measured results across six memory classes, from a 16 GB gaming card to a four-box GB10 cluster with 512 GB, using the same test harness everywhere.
Each class was measured on one reference machine. Other hardware with the same amount of memory will load the same models, but its speed will differ, so the lists of matching hardware further down are about what fits, not about how fast it goes.
Six classes, six reference machines
| Memory class | Measured on | Setups | Fully scored | Fastest decode | Best quality |
|---|---|---|---|---|---|
| 16 GB | RTX 5060 Ti 16 GB | 5 | 1 | Flint 40B, 115.9 tok/s | Qwen3.8-27B, 66.1 % |
| 32 GB | RTX 5090 32 GB | 12 | 9 | Qwen3.8-27B, 253.4 tok/s | DeepSeek-V4.1-Flash-1017, 68.0 % |
| 128 GB | 1× DGX Spark (GB10) | 15 | 9 | Nemotron 3.5 Lightning 30B-A3B, 141.9 tok/s | Ornith-1.5-122B, 70.8 % |
| 256 GB | 2× DGX Spark (GB10) | 9 | 5 | DeepSeek-V4.1-Flash-1017, 70.6 tok/s | DeepSeek-V4.1-Flash-1017, 69.1 % |
| 256 GB Mac | Mac Studio M5 Ultra 256 GB | 8 | 4 | Nemotron 3.5 Lightning 30B-A3B, 306.9 tok/s | DeepSeek-V4.1-Flash-1017, 69.7 % |
| 512 GB | 4× DGX Spark (GB10) | 5 | 4 | DeepSeek-V4.1-Flash-1017, 81.8 tok/s | DeepSeek-V4.1-Flash-1017, 66.7 % |
The fastest result per class goes 115.9 tok/s on 16 GB, 253.4 on 32 GB, 141.9 on one Spark, 70.6 on two Sparks, 306.9 on the Mac and 81.8 on four Sparks. The big classes look slow at the top because their runs are mostly far larger models, and larger models decode slower.
Quality barely follows memory
Plot every fully scored run and the quality band is narrow: all 31 land between roughly 57 and 71 %, whether they run on a 16 GB card or a 512 GB cluster. Qwen3.8-27B on one 5060 Ti scores 66.1 %; DeepSeek-V4.1-Flash on four Sparks scores 66.6 %. What bigger memory buys you is mostly context length, parallel users and the option to run very large MoE models at all.
One big model at four budgets
| Setup | Quant | Decode, tok/s | Prefill 8K, tok/s | Quality mean | Peak memory |
|---|---|---|---|---|---|
| RTX 5090 + RAM/NVMe offload | Maya-S IQ2_XXS | 20.9 | 828 | 68.0 % | 30.9 GiB |
| Mac Studio M5 Ultra | MLX 2/3-bit (e2e3-p4-h8) | 74.2 | 1,763 | 67.4 % | 206.3 GiB |
| 2× DGX Spark | EXL3 2.5 bpw | 70.6 | 2,157 | 66.1 % | 98.7 GiB |
| 4× DGX Spark | EXL3 4.0 bpw | 81.8 | 2,016 | 66.6 % | 103.2 GiB |
DeepSeek-V4.1-Flash is a 340 GB model, and it was measured on four very different budgets. On a single RTX 5090 with 96 GB of RAM and an NVMe drive holding most of the experts it decodes at 20.9 tok/s. The Mac does 74.2, two Sparks 70.6 and four Sparks 81.8. All four score between 66.1 and 68.0 %. Offloading turns a model you can't run into one you can, at a quarter of the speed.
Context costs memory too
Model weights are only part of the budget. Qwen3.8-27B EXL3 4.0 bpw on the RTX 5090 peaks at 18.9 GiB with a 128K context. The same model split across two 5060 Ti cards reaches a 256K context at 22.4 GiB. Qwen3.8-Flash-Next on two Sparks loads a 1M-token context at 87.9 GiB. On a 16 GB card, the gap between 43,008 and 117,760 tokens of context on the same model comes down to the quant and the engine's cache handling.
Where your hardware lands
16 to 24 GB
RTX 5080, RTX 5070 Ti, RTX 5060 Ti 16 GB, RTX 4090, RTX 4080 Super, RTX 4080, RTX 4070 Ti Super, RTX 4060 Ti 16 GB, RTX 3090 Ti, RTX 3090, RTX PRO 4000 Blackwell, RTX 4500 Ada, RTX 4000 Ada, Radeon RX 9070 XT, Radeon RX 9070, Radeon RX 9060 XT 16 GB, Radeon RX 9060 XT LP, Radeon RX 7900 XTX, Radeon RX 7900 XT, Radeon RX 7900 GRE, Radeon RX 7800 XT, Radeon RX 7600 XT, Radeon RX 6950 XT, Radeon RX 6900 XT, Radeon RX 6800 XT, Radeon RX 6800. These load the same 27B-class quants and MoE-with-offload setups as the RTX 5060 Ti 16 GB reference. Cards with 24 GB, such as the RTX 4090, 3090 and RX 7900 XTX, have room for longer context or a higher-bit quant but not a different model tier.
32 to 96 GB
RTX 5090, RTX PRO 6000 Blackwell, RTX PRO 5000 Blackwell 72 GB, RTX PRO 5000 Blackwell 48 GB, RTX PRO 4500 Blackwell, RTX 6000 Ada, RTX 5000 Ada, RTX A6000, Radeon AI PRO R9700, Radeon AI PRO R9600D, Radeon PRO W7900, Radeon PRO W7800 32 GB, Radeon PRO W7800 48 GB. 32 GB is the RTX 5090 tier: 27B models at 4 to 6 bits with 128K to 256K context, plus large MoE models with RAM offload.
128 GB of unified memory
Framework Desktop, GMKtec EVO-X2, HP Z2 Mini G1a, ASUS ROG Flow Z13, Mac Studio M5 Max, MacBook Pro M5 Max, NVIDIA DGX Spark, ASUS Ascent GX10, MSI EdgeXpert, Gigabyte AI TOP ATOM, Lenovo ThinkStation PGX, Acer Veriton GN100, HP ZGX Nano G1n, plus every GB10 box: NVIDIA DGX Spark, ASUS Ascent GX10, Dell Pro Max with GB10, MSI EdgeXpert, Gigabyte AI TOP ATOM, Lenovo ThinkStation PGX, Acer Veriton GN100 and HP ZGX Nano G1n. Anything in this group loads the 120B-class models measured on one Spark; Macs and AMD Strix Halo machines will run them at their own speed.
256 GB and up
A Mac Studio M5 Ultra with 256 or 512 GB, or two and four GB10 boxes linked over ConnectX-7. This is the tier for the 340 GB DeepSeek-V4.1-Flash and models of that size without offload.
Picking a class
- Chat, coding help and transcription for one person: 16 GB works, 32 GB is comfortable and much faster.
- Long-context coding agents and several users on one box: 32 GB on an RTX 5090 with a batching engine.
- Models above 100B without offload: 128 GB of unified memory; prefer the Mac if speed matters, a Spark if your stack is CUDA.
- The largest open MoE models at usable speed: a 256 GB Mac Studio, or two to four GB10 boxes.
- Before buying more memory, check whether a better engine or quant on the hardware you have closes the gap. On the same 5090 the software stack alone changed speed by almost 4×.