Running Local Coding Agents on a Mac With oMLX: Disk-Backed KV Cache Explained

Twelve minutes into a coding session the local model stops behaving like an agent and starts behaving like a fax machine. Early tool calls returned in two seconds. Now each turn takes thirty, sometimes ninety. The task has not changed. The context simply grew long, and the local server is quietly recomputing the whole conversation from scratch with every request.

That symptom, slow starts and not slow tokens, is what a small Apple Silicon project called oMLX was written to remove. It is not a model and not an agent. It is an inference server with one narrow promise: whatever the agent has already read stays cached, even as the session grows, so turn twelve feels like turn one. The honest test is not a flashy demo but how it behaves at turn forty of a genuine working session, on hardware you actually own.

Why Macs struggle with coding agents

On paper, Apple Silicon is a strong platform for local inference. Unified memory gives the CPU and GPU a single pool, so a quantized model sits in RAM without crossing a bus, and Apple's MLX framework was designed for this kind of workload.

Trouble begins when chatting turns into agent work. A coding agent rereads most of its earlier context on nearly every turn, calls a tool, receives the result appended to the prompt, and repeats dozens of times. The prefix, meaning the shared beginning of the prompt, almost matches the previous turn apart from one new tool result at the end.

Most local servers read that near match as a complete mismatch, throw away the key-value cache and recompute from position zero. On a short prompt nobody notices. On a 20,000-token session it separates a two-second reply from a ninety-second one. The project's creator named precisely this as the motivation on Apple's MLX discussion board: coding agents send requests whose prefixes shift, and each existing MLX server invalidates the cache and recomputes everything when that happens.

What oMLX is

oMLX is an open-source inference server built on Apple's MLX and mlx-lm. It ships as a native macOS menu bar app and also as a Python command-line tool. A developer named Jun Kim created it, and it lives on GitHub as jundot/omlx under the Apache 2.0 license. At the time of writing it had around 18,600 stars, 1,600 forks and well over two thousand commits.

None of that proves reliability. Stars measure attention and not maturity, and a project that publishes several release candidates a month is still finding its footing. Popularity is a reason to test it, not a replacement for testing.

A single process serves two API styles, OpenAI-compatible chat completions and a native Anthropic-style /v1/messages route, along with completions, embeddings and reranking routes. It can keep language models, vision-language models, OCR models and rerankers loaded together, with LRU eviction and per-model pinning. For agents, what matters is what happens underneath that surface.

Prefixes go to disk, not the bin

Plain MLX inference holds a conversation's cache in GPU memory only while the prefix keeps matching. The central bet in oMLX is a two-tier cache that survives both prefix drift and restarts. Frequently used blocks stay in a hot tier in memory. When that fills, blocks are written out to a cold tier on SSD in safetensors format, managed block by block with prefix sharing and copy-on-write. If a later prompt matches a prefix that was pushed to disk, those blocks are read back and not recomputed.

A librarian who never reshelves a book you recently borrowed is a fair picture. Ask again later, even after the library reopens, and the book is already on the counter.

An SSD is not RAM, so restoring a large cache still takes real time, though that is much cheaper than recomputing every token. The payoff is biggest for the agent pattern described earlier: long, mostly repeated prefixes with a small change at the end. The server also does continuous batching through mlx-lm's BatchGenerator, serving concurrent requests instead of queueing them, with the level configurable from the command line or the admin panel.

Two numbers count most. Time to first token is how long before anything streams back. Generation speed is how fast tokens arrive once they start. Cache reuse goes after the first one and leaves the second untouched. A model that generates slowly keeps generating slowly, but it begins answering earlier.

Installing on Apple Silicon

The requirements are macOS 15 (Sequoia) or newer, a Python between 3.11 and 3.13, and a chip from the M1, M2, M3 or M4 families. Three routes exist, and the first suits most people.

The recommended one is the macOS app. Download the signed and notarized .dmg from GitHub Releases, drag it to Applications and launch it. A welcome screen guides the choice of model directory, starting the server and the first download, and the app installs a small command-line shim.

With Homebrew:

brew tap jundot/omlx https://github.com/jundot/omlx
brew install omlx
omlx start

From source:

git clone https://github.com/jundot/omlx.git
cd omlx
pip install -e .            # core
pip install -e ".[mcp]"     # optional MCP support

Anyone planning to serve GLM-5.2, MiniMax M3 or Qwen3.5 should know that a plain pip install skips their native custom kernels, and the affected models then silently take a much slower path. The project reports its fused prefill kernel as roughly 30 times faster on an M3 Ultra for one of them. Building the kernels needs full Xcode and not only the Command Line Tools, which is the simplest argument for the DMG.

Before connecting anything, confirm the server is alive:

curl -s http://localhost:8000/v1/models

An empty list is fine if no model directory has been set yet.

Loading a first model

oMLX detects language models, vision-language models, embedding models and rerankers automatically in a directory it is pointed at. It also reads an existing Hugging Face cache and a LM Studio folder directly, so nothing needs downloading twice.

omlx serve --model-dir ~/models

The model should be sized to the machine's memory, not to the biggest file available. Leave macOS four or five gigabytes of headroom and budget the rest for weights plus cache growth. A 9B model at 4-bit suits machines with 16 to 32 GB. Coding models in the 30 to 40B class want 64 GB and up. The biggest models in the project's own benchmarks need Mac Studio memory beyond 400 GB.

Models can be fetched from the Hugging Face browser built into the admin dashboard at http://localhost:8000/admin, or with any existing hf download workflow aimed at the model directory. A basic request looks like an ordinary OpenAI chat completion:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "your-model-name", "messages": [{"role": "user", "content": "Say hi in five words."}]}'

If a completion comes back, the endpoint works.

Wiring up Claude Code, Codex, Cursor and OpenCode

There are two ways oMLX connects to agents, and which one applies depends on how the agent stores its configuration, not on how well it is supported.

Tools with a config file

OpenClaw, OpenCode, Codex and Pi read a config file that oMLX can reach directly, JSON for the first two and TOML for Codex. The integration adapters write to that file and the Integrations tab of the admin dashboard hands over one command:

omlx-cli launch codex --model your-model-name --api-key your-key
omlx-cli launch opencode --model your-model-name --api-key your-key
omlx-cli launch pi --model your-model-name --api-key your-key

Running it updates the tool's config and starts the agent in a single step. Any existing config is backed up first as a timestamped .bak file, so the change can be reversed if something looks off.

Hermes Agent and the Copilot entry

Both appear on the README's one-click list, and Hermes Agent's own documentation confirms its side separately: the setup wizard has a custom OpenAI-compatible endpoint option for pointing at any local server, oMLX included. To skip the dashboard, point it at http://localhost:8000/v1 by hand.

The Copilot entry deserves a second look before anyone relies on it. GitHub's real Copilot Chat authenticates through OAuth against api.githubcopilot.com and is not built to take an arbitrary local base URL. The one-click Copilot integration therefore either targets a different tool with the same name or uses a mechanism that public documentation does not confirm. Check the Integrations tab directly before assuming it does what the label implies.

Claude Code

Claude Code reads its server address from the ANTHROPIC_BASE_URL environment variable at launch and not from a file oMLX can edit, so this one stays manual. Because oMLX serves the native /v1/messages route, no translation proxy is needed:

export ANTHROPIC_BASE_URL="http://localhost:8000"
export ANTHROPIC_AUTH_TOKEN="local"
export ANTHROPIC_DEFAULT_SONNET_MODEL="your-model-name"
export ANTHROPIC_DEFAULT_OPUS_MODEL="your-model-name"
claude

oMLX also ships features aimed at Claude Code specifically. Context scaling reports token counts so that auto-compact triggers correctly for a smaller local context window, and SSE keep-alive prevents read timeouts during a long prefill. These are real agent-specific details and not a generic compatibility checkbox.

Cursor

Cursor's local-model setting sits inside its graphical interface, so there is no file for oMLX to write. In Settings, under Models, find the OpenAI API section and turn on the Override Base URL toggle. Cursor has renamed that toggle before, so the label may differ by version. Enter http://localhost:8000/v1 and put any non-empty string in the API key field, since a server on localhost does not check it. Add the loaded model under Custom Models with its exact name from oMLX, and disable Cursor's default models so it does not also try to validate against OpenAI's servers.

Checking that it is really working

During a real session, the admin dashboard's hot and cold cache panels should keep accumulating blocks and not reset on every turn. A reset each turn usually means that system-prompt or tool-schema content is being reordered between calls, which breaks the prefix.

Measuring warm against cold

Instead of repeating a vendor claim as fact, a test can be run locally. Fix the model, quantization, context length and prompt across every tool, then record cold-start time, cold time to first token, warm time to first token on a repeated prefix, steady generation speed, peak memory, behavior after 15 to 20 simulated agent turns, setup time, and whether tool calls still parse correctly through the whole session.

The project's own published numbers, taken on an M3 Ultra with 512 GB and clearly first-party, show generation around 45 to 59 tokens per second for an 8-bit Qwen3-Coder-Next across contexts from 1k to 32k. Continuous batching reaches about a 4.14 times generation speedup at eight concurrent requests against one. That is an upper bound for top-end hardware and not what a 16 GB MacBook Air will see.

One independent write-up reports that an 8,700-token context took roughly twenty seconds on plain MLX, while the warm-cache answer came back so quickly that the author ran the benchmark a second time to make sure. That is a third-party account and not a controlled test, so it is worth reproducing before trusting.

Cold-start time to first token is not meaningfully different from raw MLX, since there is nothing to reuse yet. The whole benefit sits in warm, repeated-prefix turns, which make up most of a coding agent session.

Operating the server

Start, stop and restart, with the foreground variant attached to the terminal:

omlx start          # background service, auto-restarts on crash
omlx stop
omlx restart
omlx serve --model-dir ~/models   # foreground, attached to this terminal

Confirm that it is running and serving a model:

curl -s http://localhost:8000/v1/models

Keep memory use from fighting with the rest of the system:

omlx serve --model-dir ~/models --memory-guard safe
omlx serve --model-dir ~/models --memory-guard-gb 48

Tune the two-tier cache directly:

omlx serve --model-dir ~/models --paged-ssd-cache-dir ~/.omlx/cache
omlx serve --model-dir ~/models --hot-cache-max-size 20%

Raise concurrency when several agents hit the server at once:

omlx serve --model-dir ~/models --max-concurrent-requests 16

Homebrew installs get the same lifecycle through brew services:

brew services start omlx
brew services stop omlx
brew services info omlx