Local LLM Workstation: From 20 to 60 Tokens per Second with Two GPUs

A year of local inference experiments teaches a lot, and mostly it teaches disappointment. Even with plenty of optimism, the moment real work needs doing, the cloud subscription wins. Lately, though, there's a push in the other direction. Subscriptions buy less work every month. The days of using GitHub Copilot or Cursor without watching a budget feel distant, and the money that once covered a month can vanish in a day. That's reason enough to take a local setup seriously. What follows is how one buggy stack running around 20 tokens per second turned into a dependable workstation doing 60 tokens per second on average, with peaks well above that.

Hardware that makes sense

The internet is full of people stacking multiple Mac Pros with hundreds of gigabytes of unified memory, running models on a Raspberry Pi (which has its place, but that place is edge AI), or boasting about one token per minute achieved by hammering an SSD. That's a circus. The actual rules are plainer:

• VRAM capacity comes first. Model and context must fit in VRAM, and 32 GB or 48 GB are the affordable sweet spots today.
• Extra GPUs are mainly for fitting more, not for speed. A second card adds VRAM for bigger models and longer context, but splitting work adds communication overhead, so expect a slight penalty rather than a speedup.
• Topology matters once you have several cards. Look for high-bandwidth, low-latency links between GPUs, ideally behind the same PCIe switch or at least via CPU-connected lanes. Routing through the chipset is much worse, particularly with tensor parallelism.
• Power and cooling are part of the design. Two or three GPUs draw a lot and dump the same amount of heat into the case, so PSU capacity, airflow, slot spacing and GPU TDP all count.

The second point is hard to satisfy on consumer boards. Having two PCIe slots on paper doesn't guarantee a good path between the cards, and boards that do offer one aren't cheap.

Used business workstations are the alternative. Plenty come with Xeon CPUs, lots of PCIe lanes and factory PSUs rated 1 kW or more, since they were built for multiple professional GPUs. Fitting two or three cards is routine there and often much cheaper than a comparable modern consumer platform. There are trade-offs: older DDR4, PCIe Gen4 or even Gen3, and CPUs several generations behind.

Inference differs a lot from gaming, yet published PCIe scaling benchmarks for the RTX 5090 show that even very fast GPUs lose surprisingly little speed when bandwidth is cut, and that was enough encouragement to try it. Economics decides most things, after all.

The pick was two NVIDIA RTX 5060 Ti 16 GB cards, the cheapest route to 32 GB of VRAM with excellent power efficiency, in a Xeon workstation for the reasons above. One card didn't physically fit even though the slots and lane layout were right, so a PCIe extension cable and a zip tie held it to the case. Call it Polish zip-tie engineering.

Models that fit

This hardware handles a short list of models, each with a 160k to 256k context window once the KV cache is quantized. Two are from Google, namely gemma-4-26b-a4b and its QAT variant. Two are from Qwen, namely qwen3.6-35b-a3b and qwen3.6-27b at q4_k_s.

The 27B Qwen became the daily workhorse, and not out of mere preference. Rankings and benchmarks can mislead, but skim a few and two things stand out. Few models run on such modest hardware, and Qwen leads that category by a clear margin. It's also one of the few that can compete with small frontier models, the kind already proven at carrying out small, well-defined tasks reliably.

Given the context size and capability, the best use is inside an agentic orchestration setup, paired with a larger cloud model that plans the work and hands out individual tasks. Fully or even semi-autonomous coding is still beyond what local inference can do on its own. A hybrid arrangement, however, gives really good results.

Software, and why the first try was slow

LM Studio came first because it's simple. It's great for experimenting and learning the various llama.cpp and model options. It also helps debug the integration layer between tools that consume the exposed models. It isn't always obvious whether settings passed through something like OpenCode reach the requests, and LM Studio lets you inspect them, spot the gaps and adjust.

With Qwen3.6-27B, though, it averaged around 20 tokens per second even with flash attention on, and stability was shaky, with odd timeouts every so often.

Two changes that mattered

Tensor parallelism

CUDA tensor parallelism has since landed in LM Studio, per its 0.4.15 changelog, though that wasn't always true, and the prebuilt runtime tested here still left performance unclaimed. First, what is it?

The traditional multi-GPU method puts some layers on GPU 0 and some on GPU 1 and passes activations from one to the other:

Model
                   │
          ┌────────┴────────┐
          │                 │
       GPU 0             GPU 1
      layers 0–N        layers N–end
          │                 │
          └───────→─────────┘

That lets you run a model too big for one card, and that's all. For a single stream, the two GPUs never cooperate on the same layer. Work moves along: GPU 0 handles some layers, activations travel, GPU 1 finishes the rest.

Tensor parallelism splits the computation of each layer across the GPUs instead:

One layer
                     │
             ┌───────┴───────┐
             │               │
          GPU 0           GPU 1
       part of tensor   part of tensor
             │               │
             └───────┬───────┘
                     │
                 next layer

The contrast is laid out in the multi-GPU guide inside the llama.cpp repository. Pipeline parallelism, the layer mode, places different layers on different GPUs and sends tokens through in sequence. It minimizes traffic between cards but needs many tokens in flight to scale. Tensor parallelism, the tensor mode, cuts every layer across GPUs and does several cross-GPU reductions per layer. It can parallelize any workload but leans heavily on interconnect speed. Pipeline mode maximizes batch throughput, and tensor mode minimizes latency.

With both GPUs working on the same layer, Qwen3.6-27B reached about 50 tokens per second. That creates a fresh problem, though. If two GPUs cooperate on every layer, they have to talk to each other constantly.

NCCL

NVIDIA's NCCL, the Collective Communications Library, is built for exactly this kind of GPU-to-GPU communication. Operations such as AllReduce sit at the heart of distributed GPU work, and without an efficient collective layer, tensor parallelism can burn too much time moving data around instead of computing.

The surprise was that the prebuilt llama.cpp runtime in the LM Studio version tested had NCCL support switched off. Tensor parallelism doesn't require NCCL, since llama.cpp can run the needed reductions with its own code, but NCCL supplies an optimized backend for that traffic.

Here's what is going on. Tensor parallelism has GPUs swap intermediate results, for example after part of a matrix multiplication:

Matrix operation
                       │
             ┌─────────┴─────────┐
             │                   │
          GPU 0               GPU 1
       computes part        computes part
             │                   │
             └─────────┬─────────┘
                       │
                   AllReduce
                       │
                 combined result

An AllReduce merges the partial results from every GPU and gives the combined answer to all of them. llama.cpp has its own implementation, so the internal version works. Here, NCCL's AllReduce turned out to be faster.

A few hours later

The painful part (missing libraries on Fedora, compiling, juggling llama and model parameters) isn't worth writing up in detail, partly because this corner of the software changes so fast. The outcome, as averages:

• Layer split: about 20 tokens per second
• Tensor parallelism with internal AllReduce: about 50
• Tensor parallelism with NCCL: about 60

Peaks run higher. A llama.cpp log from one run shows prompt processing near 950 to 970 tokens per second and decoding at about 78 to 84 tokens per second:

113.07.960.883 I slot print_timing: id  0 | task 972 | prompt processing, n_tokens =   4810, progress = 0.79, t =   4.96 s / 969.87 tokens per second
113.08.772.504 I slot print_timing: id  0 | task 972 | prompt processing, n_tokens =   5580, progress = 0.92, t =   5.77 s / 966.90 tokens per second
113.09.409.053 I slot print_timing: id  0 | task 972 | prompt processing, n_tokens =   6092, progress = 1.00, t =   6.41 s / 950.75 tokens per second
...
114.19.200.672 I slot print_timing: id  0 | task 972 | n_decoded =   5384, tg =  77.34 t/s, tg_3s =  83.57 t/s
114.22.229.460 I slot print_timing: id  0 | task 972 | n_decoded =   5639, tg =  77.63 t/s, tg_3s =  84.19 t/s
114.25.259.925 I slot print_timing: id  0 | task 972 | n_decoded =   5894, tg =  77.89 t/s, tg_3s =  84.15 t/s
114.28.274.967 I slot print_timing: id  0 | task 972 | n_decoded =   6149, tg =  78.15 t/s, tg_3s =  84.58 t/s
114.31.308.233 I slot print_timing: id  0 | task 972 | n_decoded =   6405, tg =  78.38 t/s, tg_3s =  84.40 t/s
114.34.343.320 I slot print_timing: id  0 | task 972 | n_decoded =   6661, tg =  78.59 t/s, tg_3s =  84.35 t/s
114.37.376.671 I slot print_timing: id  0 | task 972 | n_decoded =   6913, tg =  78.75 t/s, tg_3s =  83.08 t/s
114.40.405.502 I slot print_timing: id  0 | task 972 | n_decoded =   7169, tg =  78.94 t/s, tg_3s =  84.52 t/s
114.43.413.731 I slot print_timing: id  0 | task 972 | n_decoded =   7421, tg =  79.09 t/s, tg_3s =  83.77 t/s
114.46.451.228 I slot print_timing: id  0 | task 972 | n_decoded =   7673, tg =  79.21 t/s, tg_3s =  82.96 t/s

All of that is with Qwen3.6-27B. Gemma-4-26B-QAT can run twice as fast, and while it may be less accurate than Qwen, as an assistant it feels almost instant.

The absolute numbers matter less than the ratio. This is old second-hand hardware that cost roughly $1,100 in total, and these changes roughly tripled the starting performance.

Does it pay off?

Take these assumptions:

• Hardware cost: $1,100.
• API pricing: $0.36 input and $2.80 output per million tokens for hosted Qwen 27B on OpenRouter. That isn't a true like-for-like with a quantized local model, but some assumption is needed.
• Coding workload: about 15 input tokens per output token, since coding reads more than it writes.
• Prompt processing at 700 tokens per second.
• Output generation at 60 tokens per second.

Prompt processing and generation run one after the other. For 15 input tokens plus 1 output token, the time is 15 / 700 + 1 / 60, about 0.0381 seconds. That works out to roughly 393.75 input tokens and 26.25 output tokens per second, or around 1.42M input and 94.5K output tokens in an hour.

Pushing the same workload through the API would cost about $0.51 for input (1.4175 times $0.36) and $0.26 for output (0.0945 times $2.80). So the machine replaces about $0.775 of API spend per hour of active inference, and the break-even point is $1,100 divided by $0.775, roughly 1,420 hours.

Note that those are hours of real inference. Nobody runs a model flat out for eight hours straight, so calendar time to break even is much longer. And electricity, maintenance, failures and replacement parts haven't been counted at all.

So does it pay? Everybody has to run that calculation themselves. As a plain AI assistant, the economics aren't convincing, and the break-even is long enough that API usage may simply be more practical. Add agent orchestration, unattended task runs and pipelines, though, and the picture flips: local inference suddenly looks a lot more sensible. Here, semi-autonomous pipelines with occasional decision gates keep the machine busy about 12 hours a day, and that figure keeps climbing.