Run a Local Coding Model with llama.cpp and Wire It Into VS Code

AI coding agents can read a repository, draft patches, run the test suite and explain code you've never seen, and teams are measurably faster for it. What's still contested is the invoice. Autonomous agents chew through tokens: a single request such as "refactor this module and fix the failing tests" may consume a hundred thousand input tokens before any code is produced.

There's a middle path between paying per token and giving up on AI: host the model yourself. A laptop with a discrete GPU can run a capable coding model through llama.cpp, expose it as an OpenAI-compatible HTTP API, and connect it to VS Code through the Cline or Kilo Code extensions. Each extra token costs nothing, and no data leaves your network. Here's a setup that worked.

What you need first

Hardware: a dedicated NVIDIA GPU is strongly preferred, and more VRAM is better. Typical corporate laptops ship with around 8 GB, which is enough if you choose the model and quantization carefully. System RAM counts too, since you'll deliberately push part of the model into it.

Software: install the CUDA Toolkit, version 13.3 or newer, from NVIDIA's download page, using the local exe installer for your platform. It supplies the libraries and APIs that allocate VRAM and run the model on the graphics card, and you'd also need it if you ever compile llama.cpp yourself. You'll also want a browser and a Hugging Face account to fetch weights.

Getting llama.cpp

llama.cpp is the inference engine here. Written in C and C++, it runs GGUF models efficiently on ordinary hardware, takes advantage of CUDA, and includes an HTTP server that mimics the OpenAI API, which is the feature this whole setup relies on.

Grab the sources from the llama.cpp repository, and note that prebuilt Windows and Linux binaries are on its releases page. Download the build that matches your CUDA version, for example a win-cuda-13.3-x64 package. No compiling is needed. Unzip it somewhere simple such as D:\ai\.

Two executables matter:

• llama-cli.exe gives you an interactive terminal chat, handy for checking the install.
• llama-server.exe runs an HTTP server with an OpenAI-style /v1 endpoint and a chat UI in the browser.

Picking the model

Everything else is mechanical. Model choice is where you win or lose. Download the .gguf format from Hugging Face, because that's what llama.cpp reads.

Match the model to the job instead of expecting one to do everything:

• Coding and agent work in the IDE is the demanding case. The agent needs strong instruction following, dependable tool calling and a big context window so it can hold several files at once. Qwen3.6 35B A3B performs well here and is the default for in-IDE agent work. Its GGUF builds can be downloaded from the unsloth repository, and small dense models cover the rest.
• Quick Q&A, summaries and commit messages are fine on a much smaller dense model, which will feel snappier. Cline and Kilo Code both let you assign a lightweight small model to these side tasks, separately from the main coding model.
• Reasoning over long documents calls for context length over raw parameter count, and you should leave VRAM for the KV cache.

Favor Mixture-of-Experts models. The "A3B" in that name means 35 billion total parameters with only about 3 billion active per token. You get a large model's knowledge at close to a small model's speed, so on limited hardware an MoE will comfortably outrun a dense model of comparable quality.

Choose quantization honestly. Quantization is the compression level of the weights: the lower the Q number, the smaller the file and the worse the output quality. File size is a good proxy for how much VRAM and RAM the model will take. With 4 to 8 GB of VRAM, start looking in the Q4 down to Q2 range, such as UD-IQ4_XS or UD-IQ2_M. If you're unsure, give any AI assistant your specs, VRAM first and system RAM second, and ask which quantization to use. It takes two minutes and pays off.

First, a smoke test

Prove the stack works before touching VS Code. Open a command prompt in the llama.cpp folder and run:

"D:\ai\llama-b9590-bin-win-cuda-13.3-x64\llama-cli.exe" -m "D:\ai\llms\Qwen3.6-35B-A3B-UD-IQ2_M.gguf"

Check two things. First, the banner shows up, the model loads and you can chat in the terminal, with a token rate reported. On an RTX PRO 2000 laptop GPU the result was roughly 18 tokens per second for the prompt and 30 for generation. Second, open Task Manager, then Performance, then GPU, and confirm dedicated GPU memory jumps. If it doesn't move, the model is running on the CPU and you have a CUDA or driver problem. Fix that first, because everything downstream will be unusably slow otherwise.

Starting the server

Now for the real thing. llama-server provides both the browser UI and the API your IDE will talk to:

"D:\ai\llama-b9590-bin-win-cuda-13.3-x64\llama-server.exe" ^
  -m "D:\ai\llms\Qwen3.6-35B-A3B-UD-IQ2_M.gguf" ^
  --host 0.0.0.0 ^
  --port 8080 ^
  -ngl all ^
  -c 64000 ^
  -fa on ^
  --cache-type-k q4_0 ^
  --cache-type-v q4_0 ^
  --n-cpu-moe 22 ^
  -b 2048 ^
  -ub 2048 ^
  --no-mmap ^
  --reasoning off ^
  --reasoning-format none ^
  --reasoning-budget 0

Wait for the message saying the server is listening on port 8080 and all slots are idle, then open http://localhost:8080 in a browser and say hello.

What the flags do

Don't copy that command blindly. These are the settings you'll adjust when a model doesn't fit.

• Basics: -m is the weights file, --port sets the HTTP port for the UI and API, and --host 0.0.0.0 binds every interface so other LAN machines can connect, while you still use localhost locally.
• GPU placement: -ngl all pushes the standard layers, meaning the attention blocks and non-MoE weights, into VRAM. --n-cpu-moe 22 keeps that many layers' worth of expert weights in system RAM to avoid overflowing VRAM, and it's your main tuning dial on a small GPU.
• Memory and context: -c 64000 caps the context window in tokens, and more context costs more memory. -fa on enables FlashAttention kernels that speed up prompt processing and trim VRAM overhead. --cache-type-k and --cache-type-v at q4_0 store the key and value caches in 4 bits, a large saving at long context.
• Throughput: -b and -ub set the batch size for prompt prefill and the GPU micro-batch. Keep them equal for best throughput. --no-mmap loads the whole model into physical memory up front rather than paging it in.
• Reasoning output: --reasoning off, --reasoning-format none and --reasoning-budget 0 turn off built-in chain-of-thought handling, stop think tags from appearing in the stream, and give explicit reasoning a zero token budget.

That last group matters more than it looks. Agent extensions expect clean, parseable tool-call output, and leaving reasoning traces in the stream is a frequent reason a local model "just doesn't work" with an agent.

When the model won't fit, resist jumping to a smaller quant. Do these in order: reduce -c, then raise --n-cpu-moe, then quantize the KV cache. You'll usually find room.

Run it as a service

Leaving a console window open indefinitely is brittle. Windows lets you register llama-server.exe as a service so the endpoint returns after a reboot. On a shared team machine rather than individual laptops this is practically required, and you should put TLS in front using --ssl-cert-file and --ssl-key-file, so prompts and code never cross the network unencrypted.

Connecting VS Code

The value of an OpenAI-compatible endpoint is that agent extensions already know how to use it. Install Cline from the VS Code marketplace. At first launch it asks how you want to use it. Choose the option to bring your own API key, then open Settings and API Configuration and fill in:

• Provider: the OpenAI Compatible entry.
• Address: http://0.0.0.0:8080/v1, adjusted if your server uses another host or port.
• API Key: none.
• Model ID: the exact name the server reports, such as Qwen3.6-35B-A3B-UD-IQ2_M.
• Context Window Size: 64000.

Match the context window to the -c value you launched with. If Cline believes it has 128K while the server only allocated 64K, you get truncation errors mid-task and a confused agent.

Under Advanced, consider turning on separate models for Plan and Act modes. A stronger reasoning model drafts the plan and a faster one carries it out, which is as useful locally as it is with hosted APIs.

Kilo Code works almost the same way. Add a custom provider, give it an ID such as llamacpp, point the Base URL at your server's /v1, and it discovers the served model. One lesson learned the hard way: pick your model explicitly instead of trusting the default, and repeat that choice for each mode (Code, Ask, Debug, Orchestrator and Plan) under global config.

Security comes first

A single bad setting in an agent extension can ship your employer's source code out to the open internet. Running locally is meant to prevent exactly that, so guard it carefully.

• Check the Base URL every time. It should point at your own host. Anywhere else, and your code goes there.
• Turn off telemetry. In Cline, uncheck error and usage reporting under General settings.
• Disable web tools such as browsing and search unless you have an approved reason for them.
• Turn off MCP servers you don't control, and keep shell execution on ask rather than allow.
• Never switch on YOLO mode for a corporate codebase. Running agent-generated tasks without confirmation is as reckless as it sounds.
• Use TLS whenever the server isn't on localhost. For a shared team desktop, serve HTTPS and install the server certificate on each client machine, on Windows through certutil.
• Don't upload code or designs to outside servers. Local inference exists so you don't have to.

Is it worth it?

Do the math and local wins easily. A paid coding seat means a fixed fee per person and a monthly cap on tokens. Pay-as-you-go APIs grow in step with usage, and with decent tooling usage gets heavy. A llama.cpp server on your own machine just costs power and a GPU you already own.

The trade-offs are honest ones. Local models are slower than frontier hosted ones and weaker on truly hard reasoning, and Cline's own settings note that it works best with frontier models. Don't expect to replace your best hosted model for architecture-level decisions.

But architecture isn't where most tokens go. The bulk goes to boilerplate, refactors, test scaffolding, code explanations, commit messages and repetitive edits, and a sensibly picked MoE coding model copes with every bit of that on your own hardware at zero marginal cost, with your IP staying in the building. Use your own GPU first, and keep the paid seats for work that really needs them.