A Useful Local LLM in 30 Minutes: Coding, Document Q&A and Voice

A local model can answer a real coding question with the network unplugged, on a laptop that is already sitting on the desk. Setup takes around eight minutes and the monthly bill is nothing. A prompt like the one below is all it takes to see it work.

$ ollama run qwen3:8b
>>> rewrite this function to use async/await

Quality is the surprising part. It does not match the best cloud models on hard multi-step reasoning, but it handles ordinary engineering chores reliably enough that the gap between a weekend toy and a working tool has narrowed a lot.

Three shifts over roughly the last year and a half made this practical. Model files shrank to fit ordinary RAM: Qwen 3 ships an 8-billion-parameter variant of about 5GB. Qwen3-Coder, released in February 2026, reaches 70.6 percent on SWE-bench Verified with a 30B mixture-of-experts design that activates only 3 billion parameters per token. And installation went from a tangle of Python environments to one binary, thanks to Ollama.

Below are three separate builds of about half an hour each: a coding assistant inside the editor, a Python script that answers questions about a folder of documents, and a voice loop that listens and talks back. All share a five-minute base and then diverge, so the sensible move is to pick the one closest to daily work and leave the others for later.

The shared base

Each path needs an inference engine and a set of model weights.

Ollama plays the engine role. It deals with hardware differences, noticing whether the machine has an NVIDIA GPU, an Apple Silicon chip or only a CPU, and prepares the execution graph to match. It runs in the background and offers a local HTTP API that behaves like the big cloud providers' endpoints.

Installation depends on the platform. On macOS, Homebrew does it:

brew install ollama

Linux has an install script:

curl -fsSL https://ollama.com/install.sh | sh

Windows users can take the MSI installer from the Ollama website.

With the engine in place, a model is next. The baseline here is qwen3:8b, which needs about 5GB of disk and sits comfortably in 16GB of RAM. Without a discrete GPU, Ollama falls back to CPU inference at around 8 to 12 tokens per second, which is still perfectly readable.

Download it:

ollama pull qwen3:8b

Then confirm everything works with a quick run:

ollama run qwen3:8b "Write a python script to reverse a string."

The reply streams into the terminal. Behind the scenes Ollama loads a GGUF file, a format that memory-maps the weights from disk straight into RAM, so a look at the system monitor shows memory rising by about the size of the file.

A machine with 8GB of RAM or less will start swapping to disk, and inference slows to a crawl. For those, qwen3:4b or gemma3:4b is the better pick. With a 24GB GPU there is headroom for qwen3:14b as the natural step up.

Path A: a coding assistant in the editor

Anyone who lives in a code editor gets the fastest payback here. The goal is to connect a local model to VS Code for refactors, docstrings and boilerplate.

The general Qwen 3 model is competent, but code work benefits from specialized training. As of May 2026, Qwen3-Coder is the strongest all-round local coder, scoring 70.6 percent on SWE-bench Verified and 44.3 percent on SWE-Bench Pro. Its total size is 30 billion parameters, yet only 3 billion are active for each token, so it fits in 24GB of VRAM or runs smoothly on an M-series Mac with 32GB of unified memory.

A 16GB laptop with no discrete GPU will probably choke on the 30B model. The fallback is qwen2.5-coder:7b, which needs 4.7GB and still reaches roughly 92 percent on HumanEval.

# For 24GB+ VRAM or 32GB+ Mac Unified Memory
ollama pull qwen3-coder:30b

# For 16GB RAM laptops
ollama pull qwen2.5-coder:7b

The editor needs a front end to talk to the model. Cline, an agentic VS Code extension with more than 5 million installs, reads files, proposes diffs and can run terminal commands. It is available by searching the marketplace for its name.

One trap deserves attention. Cline carries a long system prompt that explains how to format diffs, read the file tree and use tools, and that prompt often passes 25,000 tokens before any code is attached. Ollama's default context window for Qwen 3 models is 40,000 tokens. Open a few files and the total easily exceeds that, at which point Ollama truncates the prompt without warning. The model then loses the formatting instructions, ignores the rules and produces plain text instead of valid diffs.

The cure is a custom Modelfile that forces a larger window.

A Modelfile tuned for Cline

Save the following in the working directory as a file named Modelfile.

# Modelfile — cline-tuned qwen3-coder
# Save as: ./Modelfile

FROM qwen3-coder:30b

# Cline's system prompt is ~25-30K tokens before your code is added.
# Ollama's default num_ctx for Qwen 3 is 40K, but Cline ships an
# expanded system prompt in v3+ that comfortably exceeds it on any
# real task. 65536 is the safe floor for serious use; 131072 if
# you have RAM/VRAM to spare.
PARAMETER num_ctx 65536

# Code edits want low-variance output. The default 0.7 is too loose.
PARAMETER temperature 0.2

# Stop after the model's natural turn. Cline parses this.
PARAMETER stop "<|im_end|>"

Build the model from it with the Ollama CLI, then check that the context setting took effect:

# Build the Cline-tuned model from the Modelfile in the current dir
ollama create qwen3-coder-cline -f ./Modelfile

# Verify the context window actually took
ollama show qwen3-coder-cline --modelfile | grep num_ctx
# Expected output:  PARAMETER num_ctx 65536

# Sanity-check it answers
ollama run qwen3-coder-cline "write a python one-liner to read /etc/hostname"

In VS Code, open the Command Palette with Cmd+Shift+P and choose the Cline settings entry. Point it at the local engine:

API Provider:      Ollama
Base URL:          http://localhost:11434
Model:             qwen3-coder-cline
Context Window:    65536

The context window value in the GUI must match num_ctx in the Modelfile exactly. If it is lower, Cline assumes a smaller window and quietly limits the files it sends.

To try it, open a Python file with a synchronous function, bring up the Cline chat panel and ask for a refactor to async/await. Cline reads the file, sends it to the local Qwen3-Coder model, takes the response and shows a clean diff in the editor. Accepting it is a single click, and the whole round trip happens offline.

This configuration copes with single-file refactors, tests for isolated functions and boilerplate generation. Community testing puts it at about 70 to 85 percent of frontier cloud quality on narrowly scoped tasks.

It breaks down on large multi-file architecture changes. Moving a twenty-file Flask project over to FastAPI is where it fails: the thread gets lost, imports are invented and the codebase ends up broken. Work of that size still belongs with a frontier cloud model. For the dozens of small, tedious edits made every day, though, the local loop is enough.

Path B: retrieval over your own documents

A folder of markdown notes, meeting transcripts or project documentation that gets searched by hand is the case for this path. The whole pipeline fits in about 60 lines of ordinary Python using only the Ollama client and numpy.

An embedding model converts text to vectors. Nomic Embed Text v1.5 has 137 million parameters, outputs 768-dimensional vectors, occupies under 300MB, handles an 8192-token context and scores well on MTEB retrieval benchmarks.

Fetch the embedding model and install the Python packages:

llama pull nomic-embed-text
pip install ollama numpy

Then save the script as rag.py.

"""Local RAG over a folder of notes — Ollama embeddings + numpy cosine.

No vector DB. For a few hundred docs, an in-memory list + numpy is
faster to set up and fast enough to run. Swap in sqlite-vec when
the corpus outgrows it (typically past ~5000 chunks).

Setup:
    ollama pull qwen3:8b
    ollama pull nomic-embed-text
    pip install ollama numpy

Run:
    python rag.py ./notes "what did the team decide about pricing?"
"""
from __future__ import annotations
import glob
import os
import sys

import numpy as np
import ollama

EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "qwen3:8b"
CHUNK_WORDS = 400          # ~500 tokens; fits 4 chunks in an 8K context
TOP_K = 3


def load_chunks(folder: str) -> list[str]:
    """Read every .md / .txt under folder, split into ~CHUNK_WORDS chunks."""
    chunks: list[str] = []
    for path in glob.glob(os.path.join(folder, "**/*"), recursive=True):
        if not path.endswith((".md", ".txt")):
            continue
        with open(path, encoding="utf-8") as f:
            words = f.read().split()
        for i in range(0, len(words), CHUNK_WORDS):
            chunk = " ".join(words[i : i + CHUNK_WORDS])
            if chunk.strip():
                chunks.append(chunk)
    return chunks


def embed(texts: list[str]) -> np.ndarray:
    """Embed a batch of texts via Ollama. Returns an (N, D) matrix."""
    resp = ollama.embed(model=EMBED_MODEL, input=texts)
    return np.array(resp["embeddings"], dtype=np.float32)


def top_k_indices(
    query_vec: np.ndarray, doc_mat: np.ndarray, k: int
) -> list[int]:
    """Return indices of the k most cosine-similar rows in doc_mat.

    The trick: if both vectors are unit-length (norm == 1), their
    dot product equals their cosine similarity. So we normalize
    once, then one matrix multiply gives a similarity score for
    every chunk against the query — no Python loop needed.
    """
    # Normalize query and every chunk to unit length.
    # The 1e-8 prevents division by zero on a zero vector.
    q = query_vec / (np.linalg.norm(query_vec) + 1e-8)
    d = doc_mat / (np.linalg.norm(doc_mat, axis=1, keepdims=True) + 1e-8)

    # One matmul across all chunks: scores[i] = cos(query, chunk_i).
    scores = d @ q

    # Sort descending, take the first k indices.
    return np.argsort(scores)[::-1][:k].tolist()


def answer(query: str, chunks: list[str], doc_mat: np.ndarray) -> str:
    """Retrieve top-K chunks, stuff them into a prompt, generate."""
    q_vec = embed([query])[0]
    idx = top_k_indices(q_vec, doc_mat, k=TOP_K)
    context = "\n\n---\n\n".join(chunks[i] for i in idx)

    prompt = (
        "Answer the question using ONLY the context below. "
        "If the context does not contain the answer, say so plainly.\n\n"
        f"CONTEXT:\n{context}\n\nQUESTION: {query}"
    )
    resp = ollama.chat(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp["message"]["content"]


if __name__ == "__main__":
    if len(sys.argv) != 3:
        sys.exit('usage: python rag.py <folder> "<question>"')

    folder, query = sys.argv[1], sys.argv[2]

    chunks = load_chunks(folder)
    if not chunks:
        sys.exit(f"no .md or .txt files found under {folder}")

    print(f"indexing {len(chunks)} chunks...")
    doc_mat = embed(chunks)

    print(answer(query, chunks, doc_mat))

Point it at a directory of notes and ask a question:

python rag.py ~/Documents/meeting_notes "what did the team decide about pricing?"

The script walks the folder, reads the files and cuts them into chunks of 400 words. The full list of chunks goes to Ollama's embedding endpoint in one batch. For each question, the query vector is compared with every document vector using cosine similarity.

The top three chunks are inserted into the generation prompt and qwen3:8b writes the answer. One instruction in that prompt is strict: answer using only the context provided, and say so plainly if the context lacks the answer. Local models readily produce helpful-sounding fabrications when retrieval returns nothing, and this line pushes them to admit ignorance.

This is not production retrieval. Chunking is a blunt word count that sometimes splits a sentence in half. Embeddings live in memory and vanish at exit, so the corpus must be embedded again on every run. Once a collection grows past a few thousand documents and startup becomes irritating, swapping the numpy matrix for sqlite-vec stores vectors in a local SQLite file. For personal notes, the short script is plenty.

Path C: a voice loop

For talking to a model without typing and without sending audio to a cloud service, this path builds a complete voice-to-voice loop on local hardware.

Three parts make up the pipeline: speech-to-text to transcribe the microphone, the LLM to compose a reply and text-to-speech to read it aloud.

Transcription uses Whisper.cpp, a heavily optimized C++ port of OpenAI's Whisper that runs on CPU, Apple Metal and NVIDIA CUDA. It needs the ffmpeg dependency for audio conversion. On macOS:

brew install whisper-cpp ffmpeg

On Linux, install ffmpeg through the package manager, clone the whisper.cpp repository and run make.

Whisper.cpp wants weights in GGML format. The ggml-base.en.bin file, about 150MB, is available from the official Hugging Face repository and is fast and accurate for English. For other languages or higher accuracy, ggml-large-v3.bin is the alternative. Put the file in a known folder such as ~/models/.

Speech output comes from Kokoro ONNX, a lightweight TTS engine whose weights total about 300MB and which synthesizes close to real time on a normal CPU. It currently requires Python 3.10 through 3.13 and fails to install on 3.14.

pip install -U sounddevice ollama kokoro-onnx

The pipeline script

Save the following as voice.py and change the WHISPER_MODEL path so it points at the GGML file just downloaded.

"""Local voice loop — mic → whisper.cpp → Ollama → Kokoro → speaker.

Everything runs offline. Nothing leaves the machine.

Setup:
    brew install whisper-cpp ffmpeg                  # macOS
    # (Linux: apt install ffmpeg; build whisper.cpp from source)

    pip install -U sounddevice ollama kokoro-onnx

    # Whisper GGML model — pick one:
    #   ggml-base.en.bin   (~150MB, fast, English-only)
    #   ggml-large-v3.bin  (~3GB, accurate, multilingual)
    # Download from https://huggingface.co/ggerganov/whisper.cpp

    # Kokoro model files auto-download to ~/.cache/kokoro-onnx/
    # on first run, or grab them manually from
    # https://github.com/thewh1teagle/kokoro-onnx/releases

Run:
    python voice.py
"""
from __future__ import annotations
import subprocess
import tempfile
import wave
from pathlib import Path

import sounddevice as sd
import ollama
from kokoro_onnx import Kokoro

WHISPER_MODEL = Path.home() / "models" / "ggml-base.en.bin"
CHAT_MODEL = "qwen3:8b"
KOKORO_VOICE = "af_sarah"      # see kokoro voices.json for all options
SAMPLE_RATE = 16000            # whisper.cpp requires 16kHz mono
RECORD_SECONDS = 5

# Kokoro auto-downloads the model files on first instantiation.
kokoro = Kokoro("kokoro-v1.0.onnx", "voices-v1.0.bin")


def record() -> Path:
    """Record from the default mic, write a 16kHz mono WAV, return its path."""
    print(f"listening for {RECORD_SECONDS}s...")
    audio = sd.rec(
        int(RECORD_SECONDS * SAMPLE_RATE),
        samplerate=SAMPLE_RATE,
        channels=1,                    # mono
        dtype="int16",                 # 16-bit PCM is what wave.open expects
    )
    sd.wait()                          # block until recording finishes

    # Fixed path in the system tempdir — overwritten each invocation.
    wav_path = Path(tempfile.gettempdir()) / "voice_in.wav"
    with wave.open(str(wav_path), "wb") as w:
        w.setnchannels(1)              # mono
        w.setsampwidth(2)              # 2 bytes per sample == int16
        w.setframerate(SAMPLE_RATE)    # 16 kHz — whisper.cpp's required rate
        w.writeframes(audio.tobytes())
    return wav_path


def transcribe(wav_path: Path) -> str:
    """Run whisper.cpp on a WAV file, return the transcript text."""
    result = subprocess.run(
        [
            "whisper-cli",
            "-m", str(WHISPER_MODEL),
            "-f", str(wav_path),
            "-otxt",                   # write the transcript to <input>.txt
            "-np",                     # suppress progress prints on stdout
        ],
        capture_output=True,
        text=True,
        timeout=60,
    )
    if result.returncode != 0:
        raise RuntimeError(f"whisper-cli failed: {result.stderr}")

    # -otxt writes the transcript next to the input file, with .txt
    # tacked onto the existing name — so /tmp/voice_in.wav becomes
    # /tmp/voice_in.wav.txt (same directory, same stem, extra suffix).
    txt_path = wav_path.with_suffix(wav_path.suffix + ".txt")
    return txt_path.read_text(encoding="utf-8").strip()


def respond(text: str) -> str:
    """Send the transcript to the local LLM, return the reply."""
    resp = ollama.chat(
        model=CHAT_MODEL,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a voice assistant. Keep replies to one or "
                    "two short sentences. No markdown, no lists."
                ),
            },
            {"role": "user", "content": text},
        ],
    )
    return resp["message"]["content"]


def speak(text: str) -> None:
    """Synthesize the reply with Kokoro and play it back."""
    samples, sample_rate = kokoro.create(
        text, voice=KOKORO_VOICE, speed=1.0, lang="en-us"
    )
    sd.play(samples, sample_rate)
    sd.wait()


if __name__ == "__main__":
    wav = record()
    heard = transcribe(wav)

    if not heard:
        print("nothing heard. exiting.")
        raise SystemExit(0)

    print(f"you said: {heard}")
    reply = respond(heard)
    print(f"model:    {reply}")
    speak(reply)

Running it makes the script listen for five seconds, transcribe, generate an answer and speak it. The first run pauses while Kokoro downloads its ONNX files into ~/.cache/kokoro-onnx/.

Three design choices matter more than they look. Recording at 16kHz mono is mandatory: Whisper was trained on 16kHz audio, and the 44.1kHz a default microphone delivers leads to made-up text or empty output. Running whisper-cli as a separate subprocess rather than a Python binding sidesteps lock contention and leaky C extensions, and its text-output flag drops the transcript into a file so nobody has to scrape standard output. The system prompt also explicitly bans markdown, because Kokoro reads asterisks out loud and a formatted reply wrecks the experience.

The loop handles one turn only. It does not stream audio chunks live or keep conversation history. A true conversational agent would add voice activity detection to know when speech ends, instead of a fixed five-second window. As a baseline it shows that local hardware can run the whole speech-to-text, model and speech-synthesis chain in a few seconds.

What the hardware allows

Local models are bound by memory bandwidth far more than by raw compute. How quickly weights move from RAM to the compute cores matters more than how fast the processor does arithmetic, and that one fact decides which model suits a machine.

A 16GB laptop with no discrete GPU tops out at 50 to 70 GB/s of system memory bandwidth. It can run qwen3:8b or qwen2.5-coder:7b on the CPU at 8 to 12 tokens per second, fine for retrieval and voice and slow for coding. Apple Silicon reverses the picture with unified memory at 200 to 400 GB/s, so an M-series Pro or Max with 24 to 36GB handles qwen3-coder:30b comfortably, since the MoE design activates just 3 billion parameters per token.

NVIDIA cards trade capacity for bandwidth. With 12 to 16GB of VRAM, qwen3:8b runs instantly and quantized qwen3-coder:30b fits. At 24GB or more, such as an RTX 3090 or 4090, the 30B model runs at full speed and feels close to a paid subscription for single-file work. Under 8GB of RAM the coding path is mostly out of reach, so qwen3:4b or phi-4-mini with voice and small retrieval jobs is the realistic scope.