Running AI Agents on a MacBook: Why oMLX Beats Raw MLX
Raw MLX needed about 20 seconds for an 8.7k-token agent context on an M1 Max. oMLX cut it to roughly 3 with batching and a tiered KV cache. Numbers and why.
This article is published in English.
Sources: I Tried Running AI Agents on My MacBook. MLX Was Too Slow. Then I Found oMLX
The problem: raw MLX felt too slow for agents
A common goal on Apple Silicon is to run a local LLM as the brain for agents. Two machines under test here: an M1 Max with 64GB unified memory and an M4 Max with the same. Both run Apple's MLX. Both look capable on paper.
A 4-bit MLX build of the Unsloth Qwen3.6 35B MoE checkpoint (about 21GB on Hugging Face) loaded through raw MLX. The aim was long-context agent work: codebases, documents, conversation history, with responses that feel real-time.
Raw MLX numbers disappointed. On the M1 Max, processing an 8,700-token context took about 20 seconds before any output token. The M4 Max did better at roughly 5.75 seconds, still not great. An agent that makes dozens of requests per task, each re-reading the whole context, is unusable at those speeds.
Then oMLX
oMLX is an open-source LLM inference server on top of MLX, aimed at Apple Silicon, with continuous batching and tiered KV caching. Skeptical of another wrapper, it still got a try: clone, venv, point at existing MLX models, start the server.
git clone https://github.com/jundot/omlx.git
cd omlx
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[mcp]"
# Start the server with my existing models
omlx serve --model-dir ~/.lmstudio/models
The server discovered the Qwen model and exposed an OpenAI-compatible API at http://localhost:8000/v1. The same benchmarks as raw MLX ran again. The numbers were hard to believe at first glance.
From twenty seconds to about one
Same model, same hardware, warm cache, both Macs, raw MLX versus oMLX.
Response time
With raw MLX, long-context waits made agent work feel impossible: roughly twenty seconds just to read 8,700 tokens. With oMLX, that long-context bar dropped to about 2.95 seconds on the M1 Max and about 1.01 seconds on the M4 Max. Short and medium responses improved too, but long context was the dealbreaker, and oMLX fixed it.
Output generation
Once prefill finishes, tokens stream. Raw MLX and oMLX generate at roughly the same speed: around 42 tok/s on the M1 Max and around 95 tok/s on the M4 Max. oMLX does not speed generation. It does not need to; 95 tok/s is already interactive. The bottleneck was never decode. It was prefill.
Prefill: where oMLX shines
Prefill is how fast the model reads input before it starts generating. For agents that stuff thousands of tokens of code, system prompts, tools and history into each call, prefill is everything. Slow prefill makes the agent feel dead.
On the M1 Max, raw MLX read context around 579 tok/s; oMLX around 2,975 tok/s, about 5.1x. On the M4 Max, raw MLX around 1,520 tok/s versus oMLX around 8,664 tok/s, about 5.7x.
Just as important: oMLX caches that context. After the model reads an 8,700-token codebase, the KV state stays available. The next request reuses it. Raw MLX has no such cache and starts from zero every time.
M4 Max versus M1 Max
With oMLX, the M4 Max is clearly faster: short responses from about 1.08s to 0.16s, long context from about 2.95s to 1.01s, roughly 2–3x across the board. Output roughly doubles (42 → 95 tok/s) and prefill nearly triples (2,975 → 8,664 tok/s). For all-day agent work that matters.
Still, the older M1 Max with oMLX is fast enough for real work. The twenty-second long context becomes three seconds. Output is interactive. Cache makes repeats feel instant. For a MoE model like this Qwen3.6–35B-A3B, oMLX mattered more than a new MacBook. For denser models such as Qwen3.6–27B, an M4 Max or newer may still be needed.
Why oMLX beats raw MLX
Not magic. Four features raw MLX lacks.
Continuous batching
Agents fire many requests: tool calls, follow-ups, parallel lookups. Raw MLX does them one at a time. oMLX batches concurrent requests into a single GPU pass. Real-world impact when the agent is busy: often 2–5x.
Tiered KV cache (RAM + SSD)
Hot tier in RAM for active context; cold tier on SSD for older context that should not evict hot data. After the first read of a large codebase, later requests reuse cached KV pairs. Context switching and revisiting earlier turns stay cheap.
OpenAI-compatible API
http://localhost:8000/v1 matches the OpenAI shape. Point Claude Code, Cursor, a custom agent framework or any OpenAI-speaking client at localhost. Raw MLX is a Python library; you build the server yourself.
Multi-model serving
oMLX auto-discovers models in a directory and can serve several at once, swapping them in and out of memory by demand. Embeddings for retrieval plus a 35B for reasoning can share one process.
How to reproduce the check
- Install oMLX from the GitHub repo and create a virtualenv as in the snippet above.
- Point
--model-dirat a folder that already holds MLX weights (LM Studio's models directory works). - Hit the OpenAI-compatible
/v1/chat/completionsendpoint with a prompt whose prompt tokens sit near 8–9k. - Time wall clock to first token (prefill-dominated) and tokens per second during decode.
- Repeat the identical prompt immediately. If the tiered cache is working, the second run should feel much cheaper on prefill.
Compare those timings against a raw MLX script that loads the same weights without a server. The gap on long prompts is where the agent story lives.
Caveats worth stating
Numbers above are from one model family, two Macs and warm-cache runs. Cold start, different quantization, dense versus MoE architectures and competing GPU workloads will move the curves. Treat the 5x prefill claims as "order of magnitude on this setup," not a universal guarantee. Also, local serving still means you own updates, disk and thermal limits. oMLX removes the worst prefill pain; it does not remove ops.
Bottom line
Raw MLX made local agents on a MacBook feel impractical: twenty seconds for one 8,700-token request on an M1 Max. With oMLX: about three seconds on the M1 Max and about one second on the M4 Max, with context cached afterward. The hardware was not the main problem. The inference stack was.
Apple Silicon with 32GB+ of RAM is already enough for serious local agents if the server adds batching and KV cache. No cloud API, no monthly bill for inference, no round-trip latency beyond your desk.
- oMLX: github.com/jundot/omlx
- Model: Qwen3.6–35B-A3B-UD-MLX-4bit (21GB, 4-bit)
- Hardware: M1 Max and M4 Max MacBook Pros, 64GB each
- macOS: 26.x (Sequoia)