Running Local Coding Models on a 12GB RTX 3060 With a Model Scheduler

Several local models on one 12GB GPU

Getting one LLM to run locally is easy these days. Keeping several specialized models available for coding, reasoning and agent work on a single 12GB card is the part that gets interesting.

Picture a fairly ordinary development box: an RTX 3060 with 12GB of VRAM, a Ryzen 7 5800, 64GB of RAM and Ubuntu. The aim isn't a local chat window. It's local models that coding agents like OpenCode, and other OpenAI-compatible tools, can actually work with. That puts coding ability, tool calling, context size and latency on the checklist, plus one more thing: switching between models without babysitting GPU memory by hand.

What 12GB can really do

Modern 7B to 12B models, especially well quantized ones, are genuinely useful for coding on a 12GB card. The trap is optimizing for the biggest model that loads. Weights aren't the only thing eating VRAM. Context, the KV cache, runtime buffers and other processes on the GPU all want their share.

Coding agents make that worse, because one request can carry a system prompt, tool definitions, repository context, source files and the conversation so far, all stacked together.

A better rule is to run the strongest model that fits comfortably, not the largest one that technically loads.

On constrained hardware, GGUF with llama.cpp is usually the better pick, since quantization and CPU/RAM offloading give you a lot of room to maneuver. On bigger GPU setups where throughput counts, SafeTensors with vLLM becomes attractive. And that creates the next headache, because you might want both.

The real problem is switching

A realistic workflow needs more than one model: one tuned for coding, one for reasoning, one verified for tool calling, an embedding model, and a few experimental ones under evaluation. A 3060 can't keep them all in VRAM. It doesn't have to, as long as the right one is loaded when a request arrives. The behavior you want is roughly this:

Request → Model B

And behind that, a decision flow along these lines:

Model B running?
      │
      No
      ↓
Drain Model A
      ↓
Unload Model A
      ↓
Free VRAM
      ↓
Load Model B
      ↓
Health check
      ↓
Serve request

In other words, the request decides which model sits on the GPU. At that point the job stops being about inference and turns into scheduling.

Why Ollama or LM Studio doesn't cover it

Both are very good. Ollama is one of the easiest ways to download a model and start using it, and LM Studio is great for finding, configuring and experimenting with local models. Both also have increasingly capable model management.

The requirement here is a bit different: several applications should treat the whole collection of local models as shared infrastructure. Without that you end up with a pile of separate ports:

Coding model       → localhost:8081
Reasoning model    → localhost:8082
Experimental model → localhost:8083
vLLM               → localhost:8000

What you'd rather have is every client talking to a single endpoint:

OpenCode ────┐
AI Agents ───┤
Python SDK ──┼──→ One OpenAI-compatible API
Node SDK ────┤
Apps ────────┘
                    │
                    ▼
              Model Scheduler
                /        \
               /          \
          llama.cpp       vLLM
              │             │
             GGUF      SafeTensors

Clients shouldn't have to know which runtime is serving them. They ask for a model, and the infrastructure works out how to run it.

GGUF Switchboard

That idea turned into GGUF Switchboard, an open-source model scheduler for local AI setups. It exposes one OpenAI-compatible endpoint and handles the model lifecycle underneath. The motto is short: one endpoint, multiple local models, and the infrastructure manages the GPU.

Right now it concentrates on four things.

Hardware-aware model discovery. A search like this shouldn't just list every Qwen model:

ggs models search "qwen"

It should tell you which variants make sense for the RAM and VRAM the machine really has.

Request-driven switching. When a request needs a different model, Switchboard drains the current workload, unloads the loaded model, loads the requested one and routes traffic to it.

Multiple inference backends. Each model format maps to its own runtime:

GGUF → llama.cpp
SafeTensors → vLLM

Clients don't need a separate configuration per runtime.

Tool conformance testing. A model that writes good code isn't automatically reliable inside an agent. Tool selection, structured arguments and runtime behavior should be verified before a model gets trusted with agentic workflows.

The difference in one sentence

Ollama and LM Studio make running a local model easy. GGUF Switchboard aims to make operating a whole set of local models boring. On a 12GB GPU, boring infrastructure is exactly the goal.

Where to find it

The project is open source, and it's being tested on consumer and multi-GPU machines. The repository is at https://github.com/pradeepgudipati/gguf-switchboard.

If you run 8 to 16GB consumer cards, the interesting open questions are how you handle multiple models today (one model for everything, manual switching, Ollama's model management, llama.cpp directly, another scheduler) and which models have proven dependable for tool calling in OpenCode, Cline, Cursor or similar agents. If a hardware and model combination misbehaves, an issue on the repository with the details is the most useful report, since real configurations drive the project's improvements.