Building a Local AI Agent: Models, Memory, Storage and Interface

A local AI agent keeps the model on your own hardware, acts for you and carries context from one session to the next, without shipping data to an outside API. A basic chatbot answers and forgets. An agent can reason across several steps, use tools and accumulate knowledge over time. Running it locally gives you an assistant without the privacy trade-offs or per-call fees of hosted services.

Five layers have to work together to build one: an LLM layer running inference on your machine, an agent framework that routes and carries out actions, a memory layer that makes the agent better over time, a storage layer that keeps what it picks up, and an interface layer that fits the way you already work. Below, each layer gets its options, trade-offs and a recommendation, so you can assemble a stack that suits your situation.

Layer 1: the model

This is the engine. The model detects intent, generates code and extracts structured data that the rest of the stack relies on, and everything else is plumbing built around what it can do reliably. Three tools dominate local inference.

Ollama is the easiest way in. One command installs it, one command pulls a model, and you immediately have an OpenAI-compatible API on localhost. It manages models, picks quantization and sets context windows without configuration. It isn't built for production traffic and suits a personal assistant or developer tool better.

ollama pull qwen3:8b
ollama pull nomic-embed-text

llama.cpp sits a level lower. It's a C++ engine that runs quantized GGUF models directly, with no daemon and no HTTP layer. Its tiny memory footprint makes it a good fit for edge devices such as a Raspberry Pi or embedded Linux, where Ollama would weigh too much. The cost is no built-in model management and more manual setup.

vLLM is the production option. It uses PagedAttention and continuous batching, the techniques behind large-scale cloud inference. If you serve many users or have real throughput needs, this is the right layer, though it expects proper GPU infrastructure and scales better than the other two.

For a single developer's local assistant, Ollama is the sensible default, because simplicity matters more than throughput. The qwen3:8b model deserves a mention: it gives dependable JSON for intent detection, writes code cleanly and fits in 8 GB of VRAM. One quirk to know about is that it wraps its answer in <think>…</think> blocks before the real content, which breaks any downstream JSON parser working on the raw response. A small wrapper that sets think: False everywhere fixes it:

_orig_ollama_chat = ollama.chat
def _no_think_chat(*args, **kwargs):
    opts = kwargs.get("options") or {}
    if isinstance(opts, dict):
        opts.setdefault("think", False)
        kwargs["options"] = opts
    return _orig_ollama_chat(*args, **kwargs)
ollama.chat = _no_think_chat

Because it uses setdefault, individual callers can still switch thinking back on when they want it. For JSON extraction and code generation in a tool pipeline, thinking mode adds latency without improving quality.

Layer 2: the agent framework

The model generates text. The framework decides what to do with that text and what to do next, covering intent classification, routing and execution.

OpenClaw is tool-first and works at the filesystem level. Each action is a command dispatched to a declared skill, such as creating a file, writing code, opening a folder or running a shell command. You write a SKILL.md that declares the skill's name and how it's dispatched, and the framework handles routing:

- -
name: local-ai-assistant
description: Local AI coding assistant with persistent memory
command-dispatch: tool
command-tool: exec
command-arg-mode: raw
-

When most operations are file I/O, as in a coding assistant, this model fits naturally. The command-dispatch: tool and command-tool: exec lines tell OpenClaw to shell out to Python scripts when a skill is called, so file work passes through the framework. Its limit is that it isn't designed for long multi-step reasoning chains or for coordinating several agents.

LangGraph takes the opposite approach. You define an explicit state graph in which nodes are model calls or actions, edges are transitions and the framework carries state between steps. The price is up-front design: you must settle the state schema and graph shape before any business logic. For simple tool use that's overengineering, but for complicated multi-step pipelines it yields an inspectable record of what happened at every step.

CrewAI is one level higher still. You describe agents with roles and tasks with descriptions, and the framework works out execution order and communication between agents. It's the quickest route to a working multi-agent prototype, though the abstraction makes failures harder to debug.

Whichever you choose, intent classification runs first on every incoming message. Pairing model-based classification with a regex fallback copes with failure better than either method alone:

def detect_intent(message: str) -> dict:
    try:
        resp = ollama.chat(
            model=OLLAMA_CHAT_MODEL,
            messages=[{"role": "user", "content": INTENT_PROMPT.format(message=message)}],
            options={"temperature": 0, "num_predict": 1024},
        )
        return extract_json(resp["message"]["content"])
    except Exception:
        return keyword_intent_fallback(message)

The fallback catches common patterns when the model returns malformed output. With thinking disabled and temperature at 0, most requests are handled by the model path, yet it's the fallback that keeps the agent alive.

Layer 3: memory

Memory separates an agent that adapts to you from one that treats each conversation as the first. It learns your conventions, preferences and project context, and applies them unprompted.

The usual workarounds live inside the context window: a PREFERENCES.md read at startup, a lengthy system prompt or a CONTEXT.md file. They share a flaw. Inside the window they're squeezed out by compaction, token limits and session restarts. What an agent actually needs is memory outside the window, stored durably, retrieved by meaning when relevant and updated as facts change. Three common approaches serve different needs.

Basic RAG is the most widespread and friendliest for beginners. Chunk documents or conversation history, embed the chunks and fetch the top-k most similar ones when a query arrives, and little setup is needed. Its weakness is that memory becomes a pile of documents, with no distinction between what's worth keeping for the long term and a throwaway question.

Session-scoped memory, for example LangChain's ConversationBufferMemory and ConversationSummaryMemory, gives coherence within a session. The agent won't forget what you said five messages ago, and summaries stop the window from filling. It doesn't survive restarts, though, which is the core gap for a coding assistant.

Structured persistent memory, as in Mem0, treats memory as first-class data and not raw text. In place of verbatim messages, the model extracts discrete facts, which are embedded and stored in a vector database. At retrieval time the query is embedded and semantically similar facts come back, even if the new message doesn't use the original wording. Stored facts last between sessions, can be merged when duplicated and get rewritten when reality shifts. Mem0 supports local models, so it's easy to connect to an Ollama-based stack.

Take a developer who mentions liking pytest. A plain RAG system keeps the raw sentence. A structured system extracts and stores the preference itself. Next time you ask for a function with tests, memory returns "user prefers pytest" even if the message never names a testing framework.

Wiring Mem0 into a local Ollama setup takes a short configuration pointing at your local model, local embedding model and a vector store:

from mem0 import Memory
config = {
    "llm": {
        "provider": "ollama",
        "config": {"model": "qwen3:8b", "ollama_base_url": "http://localhost:11434"},
    },
    "embedder": {
        "provider": "ollama",
        "config": {"model": "nomic-embed-text", "ollama_base_url": "http://localhost:11434"},
    },
    "vector_store": {
        "provider": "qdrant",
        "config": {"host": "localhost", "port": 6333, "embedding_model_dims": 768},
    },
}
memory = Memory.from_config(config)
memory.add("I always use type hints and pytest", user_id="dev")
results = memory.search("write a utility function", user_id="dev")

The llm block directs Mem0 to qwen3:8b for fact extraction, turning a raw message into discrete stored preferences instead of verbatim text. The embedder block uses nomic-embed-text to vectorise those facts on write and to embed queries on retrieval, and memory.search() then retrieves by meaning and not by keyword.

An underrated improvement for any memory approach is to filter what goes into storage before writing it. A lightweight classifier running ahead of every write keeps the memory clean and retrieval quality high:

def _is_worth_storing(self, user_message: str) -> bool:
    response = ollama.chat(
        model=OLLAMA_CHAT_MODEL,
        messages=[{"role": "user", "content": SMART_MEMORY_PROMPT.format(
            user_message=user_message
        )}],
        options={"temperature": 0, "num_predict": 512},
    )
    data = self._extract_json_robust(response["message"]["content"])
    return bool(data.get("worth_storing", False))

Its prompt gives the model clear examples of the distinction:

HIGH-VALUE (worth storing):
- "I prefer TypeScript over JavaScript"
- "I use pytest, never unittest"
- "Always use Google-style docstrings"

LOW-VALUE (discard):
- "What does enumerate() do?"
- "Write a retry decorator"
- "Thanks"

Skip the filter and the vector database slowly fills with single-use requests that water down retrieval.

Layer 4: storage

The memory layer decides what to remember, and storage is where it lives on disk.

FAISS is an in-process library: fast, well tested and free of infrastructure dependencies. It can be embedded in your Python process and run in memory or be serialised to disk. Its limitations are operational. Persistence needs explicit serialisation calls, and there's no HTTP API or built-in replication. That suits research pipelines and batch workloads but is fragile for a long-running assistant that restarts often.

Chroma is about the simplest vector database a developer can pick. It starts as a local Python process, or runs embedded in your own code, and saves data to a local directory automatically. Install it with:

pip install chromadb

Thousands of remembered facts in a personal assistant are well within Chroma's comfort zone. It removes the infrastructure dependency entirely, at the expense of filtering that is more limited than Qdrant's.

Qdrant runs as a real service with full REST and gRPC APIs, payload filtering and named vectors. It's aimed at deployments with millions of vectors and concurrent queries. It persists to disk natively and survives Docker restarts without special handling:

# Qdrant via Docker
docker run -d - name qdrant-local -p 6333:6333 \
-v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant

One configuration detail breaks things silently: the embedding dimension in the vector-store config must match the output dimension of your embedding model exactly. The nomic-embed-text model produces 768-dimensional vectors, for example. Change the embedding model without updating that value and inserts will fail without a visible error.

"vector_store": {
    "provider": "qdrant",
    "config": {
        "collection_name": "coding_assistant",
        "host": "localhost",
        "port": 6333,
        "embedding_model_dims": 768,  # must match your embedding model exactly
    },
}

Chroma is a straight swap if you want to drop Docker: change the vector_store provider in the config and the rest of the stack stays the same.

Layer 5: the interface

The outermost layer shapes how you talk to the agent, and the right choice depends on where it sits in your workflow.

• Web UI. The most flexible choice. It runs in a browser, renders markdown, highlighted code and collapsible sections and works on any operating system without installation. A coding assistant can show the current file, generated code and the agent's memory state together on one screen.
• CLI. A terminal assistant slots into shell workflows, pipes output to other tools and adds no visual overhead. Rich output takes extra care and multi-turn conversation is clumsier than in a chat window. It's best for quick lookups, file operations and scripted jobs.
• IDE plugin. An assistant inside VS Code or JetBrains sits beside the file you're editing, knows which file is open and suggests edits inline. You need to learn the extension APIs and keep up with IDE updates, which can be worthwhile for a team tool used daily.
• Chat apps such as Slack, Discord or Telegram. These make sense when the assistant should be reachable from many devices or shared across a team. They aren't suited to code review or file editing, and fit Q&A, status checks and light requests.

How the choices fit together

Each layer constrains its neighbours. The model picks the API shape the framework calls, the memory approach decides what storage must support, and the interface determines how much output formatting matters.

For a single developer's local coding assistant, this stack minimises friction while keeping every layer replaceable:

Interface: Web UI on localhost
│
Framework: OpenClaw - tool-dispatch, SKILL.md routing
│
LLM: Ollama for qwen3:8b
(think mode disabled for JSON reliability)
│
Memory: Structured persistent memory with value filtering
Embeddings via nomic-embed-text
│
Storage: Chroma (zero infrastructure) or Qdrant (more robust)

Since each layer can be swapped independently, start with the simplest option at every level and upgrade as needs grow. Moving from Chroma to Qdrant or from Ollama to vLLM doesn't require rebuilding anything above or below it. If one layer deserves early investment, make it memory. A modest local model with a well-designed persistent memory layer reliably beats a larger one that starts every session from nothing.