Running Kimi K3 and Other Big Models Locally with Five Open Repos

Running frontier models on a desk instead of in a data center

Hosted models from the big labs are impressive and expensive. Build anything serious on them and the bill climbs fast. Open-weight models that come close to the paid ones have been around for a while, but with a normal machine most people still ended up buying inference from someone else. A handful of open source projects now change that. They don't target toy models, either. They aim at frontier-class models that cost millions to train, running offline on your own hardware at no per-token cost.

The five repos below all attack one problem: how do you shrink something built for a data center until it fits on a desk? They answer it in very different ways.

What these projects have in common

The gap is huge. A model like Kimi K3 needs 1.56 TB just to store its weights, and a laptop typically has 16 or 32 GB of RAM. Every tool here is a workaround for that mismatch, and the workarounds fall into three families:

• Stream the weights from disk instead of loading them all into memory.
• Compress the model hard enough that it ends up a fraction of its original size.
• Skip the neurons that contribute nothing to the current prompt, since most of them don't, so there's no reason to load them.

They differ in other ways too. Some are general inference engines that handle hundreds of models, others are built around exactly one architecture. Some need a GPU and some deliberately avoid one. A few are research experiments pushing at what's possible, while others are ready to deploy today.

1. Kimi K3 in C

Kimi K3 is a mixture-of-experts model from Moonshot AI with 2.78 trillion parameters, and on several benchmarks it beats Claude. Until recently you'd have assumed a warehouse-sized server farm was the entry ticket.

This repo disagrees. It's an inference engine in portable C99, 176 KB of code, with no dependencies at all: no BLAS, no PyTorch, no GPU. Its author got it to run the full model on a laptop with as little as 8 GB of RAM.

Output is byte-identical on an 8 GB laptop and on a 128 GB workstation, and only speed changes. With 8 GB you wait about 26 seconds per token, which is painful but works. Give it enough RAM to keep the entire model resident and the wait shrinks to 5.6 seconds per token.

The trick is disk streaming. Because K3 is a mixture-of-experts model, only around 4% of its parameters are active for any given token. The engine reads just the experts it needs from disk, straight into the computation, and never tries to hold the entire model in memory. The whole engine is one C file with no dependencies.

To try it:

# Clone the repo
git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c

# Build
make

# Run with a preset matching your hardware
./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
  --tok ~/k3model --prompt "The capital of France is" --gen 8

2. WARP

If the C engine above is the minimal proof of concept, WARP is the engineering-heavy version. It comes from the SQLiteAI team, and it's an embeddable C inference engine that keeps the same stream-from-disk idea and piles real optimization on top.

The shared trunk of the model stays in memory, and only the experts selected for the current token come off disk. Whatever RAM is left over becomes a bounded expert cache, so experts that get picked often stay resident. A lookahead router goes a step further: it predicts which experts the next layer will want and starts reading them early.

End result: the K3 container drops from 1.56 TB to 982 GB, and a 64 GB MacBook Pro can run it, though only at roughly 0.6 tokens a second.

The project describes itself as driven by humans and written by LLMs. Its long-term goal is a little wild: get WARP running K3 well enough that the model could work on improving itself.

Setup:

# Clone
git clone https://github.com/sqliteai/waste.git
cd waste

# Build (requires C compiler)
make

# Run with auto-detected memory budget
./waste run ~/models/k3.waste "What is the capital of Italy?"

# Or specify a memory budget manually
./waste run ~/models/k3.waste "Explain quantum computing" --budget 48GB

3. Deltafin

Deltafin starts from a question few people think to ask. What if you ran the complete K3, all 16 experts on every token, never pruned, on consumer hardware, and then made that as fast as you could?

That's as much a philosophy as an engineering choice. Other projects take shortcuts: pruning experts, skipping layers, or using a draft model to speculate. Deltafin won't. Its quality rule reads like this: K3 alone decides every token. Small draft models may guess ahead, and that's where a lot of the speed comes from, but K3 checks each guess and nothing reaches you without its approval.

The numbers are modest and openly stated. An M1 Max laptop currently manages about 0.29 tokens per second. The progress chart in the README is the interesting part, though. In roughly a week the project went from 0.014 to 0.29 tokens per second, a 20x improvement from sustained optimization work. Its mission statement puts the stance plainly: speed must never come from reducing model quality.

The team counts it as progress if a frontier model can run on a home setup that costs around $15,000 instead of $2,000,000 worth of infrastructure.

Getting started:

# Download the latest release from GitHub
# https://github.com/gavamedia/deltafin/releases

# Run with model path (auto-detects hardware)
./deltafin --model /path/to/k3-checkpoint


# The binary handles expert routing, memory management,
# and hardware acceleration automatically.

4. llama.cpp

This is the name you've most likely heard already. llama.cpp is what made local inference mainstream.

Its creator, Georgi Gerganov, began with a simple idea: run LLaMA models in plain C/C++ with as few dependencies as possible. No PyTorch, no Python runtime, just direct performance from the metal to the model. The idea grew into the most widely used local inference engine around. It supports hundreds of models across CPUs, GPUs (CUDA, Metal, Vulkan, SYCL) and some unusual hardware as well.

An ecosystem formed around it. Ollama, LM Studio and GPT4All, along with many other "just run a model locally" tools, are built on llama.cpp underneath. You could call it the Linux kernel of local AI inference.

Usage:

# Install via Homebrew (easiest)
brew install llama.cpp

# Download and run any GGUF model from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Or launch an OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

# Build from source
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp && cmake -B build && cmake --build build

5. FlexGen

FlexGen suits a different situation: you have one commodity GPU and want to push millions of tokens through a model overnight.

It's a high-throughput generation engine from the FMInference team for running large language models on limited GPU memory. It relies on IO-efficient offloading. When weights don't fit in GPU memory they go to CPU RAM, or even disk, and the whole scheme is tuned for throughput instead of low latency.

As an example, it can run OPT-175B, which needs 350 GB of GPU memory at full precision, on a single 16 GB T4 by offloading weights to disk. Per-token latency is poor for chat. For overnight batch jobs such as benchmarking (it integrates with Stanford's HELM framework) or data wrangling, it's what you want.

All five repos go after the same goal of running AI locally, each from a different angle. Taken together they point in one direction: away from sending your data to a server farm and toward the strongest models living on your own device.