A Private AI Second Brain: Local RAG With AnythingLLM, Ollama and Obsidian

A brilliant amnesiac

A local language model can hold a conversation, explain concepts and help think through ideas, all without a byte leaving the building. What it can't do out of the box is recall anything you've saved.

Picture a year of accumulated Obsidian notes: links bookmarked at midnight, half-formed thoughts, a long pasted article about search engines, job listings and workshop notes. All of it sits in a folder on a laptop, invisible to an AI that took a whole weekend to get running offline.

Storage was never the issue. Retrieval is. Pulling the right note at the right moment is what turns a folder of links into knowledge instead of a well-organised junk drawer. So the natural next step for a private AI setup is to give it access to those notes.

What RAG is and why it matters

Most "run AI locally" guides skip this part, probably because embeddings sound dull. They are a bit dull, but they're worth understanding.

RAG stands for Retrieval-Augmented Generation. It lets a language model draw on your own documents, with no retraining and no uploads to a third party's server.

Think of hiring a very fast librarian rather than asking the AI to memorise every book. The librarian doesn't know your notes by heart, only which one to fetch when you ask a question.

Here's what happens underneath:

1. Documents are split into chunks and turned into embeddings, numeric representations of meaning, which are kept in a vector database.
2. A question is turned into an embedding too.
3. The database finds the chunks closest in meaning to the question.
4. Those chunks go to the language model as context.
5. The model answers from your real documents and not only from what it absorbed in training.

The tool: AnythingLLM

AnythingLLM suits this build for several reasons:

• Workspaces keep separate knowledge bases for different contexts, such as notes, research or blog drafts.
• It reads Markdown natively, so Obsidian notes need no conversion.
• It connects straight to Ollama.
• It's self-hosted and runs as a Docker container, with no account and no cloud.
• It also has a desktop app, which helps people who don't have a homelab.

It isn't the only option. Open WebUI has RAG built in as well. But AnythingLLM's workspace model fits the idea of a "second brain" more cleanly, and its interface is friendly enough to recommend to a non-technical reader.

Building the environment

Starting point

The setup assumes a base like this:

• a bare-metal Proxmox VE host;
• one Ubuntu 24.04 VM holding Ollama with Mistral 7B;
• Open WebUI as the chat front end.

The build adds a second VM next to it.

A new VM in Proxmox

Create a fresh Ubuntu 24.04 LTS VM for AnythingLLM. Keeping it apart from the Ollama VM allows snapshots, deliberate breakage and restores without touching the actual inference setup. Breaking things on purpose beats breaking them by accident later.

Suggested specs:

• 4 vCPUs
• 8 GB RAM
• 40 GB disk, since vector databases take space

Nothing here is difficult. The documentation covers it, plus a little trial and error, especially for anyone who already set up the Ollama VM. The real constraint is the hardware at hand, and working within that limit is the point of the exercise.

Installing the embedding model

Most guides skip this step, and it causes the most head-scratching.

RAG needs two models. One is the inference model, which Mistral 7B already provides. The other is an embedding model, which converts text to vectors. They're different things, and few guides say so up front.

On the existing Ollama VM, pull the embedding model:

ollama pull nomic-embed-text

nomic-embed-text is small at roughly 270 MB, quick on a CPU and decent on English text, a sensible choice for hardware with no GPU in sight.

Another snag: Ollama listens only on localhost by default. For AnythingLLM on a different VM to reach it, Ollama must bind to all interfaces. Start by editing the service:

sudo systemctl edit ollama

Then add this override:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Next, open port 11434 in the Proxmox firewall between the two VMs. The lesson repeats itself: the service was fine and the model was fine. What was broken was the assumption that everything can talk to everything by default.

The networking could be stricter, and it probably should be eventually. That's a typical hobbyist stance: get it working, understand what was built, then tear it down and redo it properly.

AnythingLLM through Docker

On the new VM, install Docker first:

# Install Docker
curl -fsSL <https://get.docker.com> | sh
sudo usermod -aG docker $USER
newgrp docker

Then start AnythingLLM:

# Run AnythingLLM
docker run -d \
  -p 3001:3001 \
  --name anythingllm \
  -v ~/.anythingllm:/app/server/storage \
  mintplexlabs/anythingllm

Open http://<new-vm-ip>:3001 in a browser. The first-run configuration is:

• LLM provider: Ollama, at http://<ollama-vm-ip>:11434
• Chat model: Mistral 7B
• Embedding provider: Ollama, with nomic-embed-text
• Vector database: LanceDB, which is built in and needs no extra setup

Feeding it the notes

Create a workspace called "Second Brain" and point AnythingLLM at the Obsidian vault. Its document manager accepts folder uploads, so give it the notes directory and let it work through everything.

It ingests .md, .txt, .pdf, .docx and .csv. Obsidian canvas files are JSON underneath, so they won't cause an error but won't yield anything useful either.

After ingestion, the workspace shows chunk counts and embedding status. Each note is cut into overlapping chunks of around 500 tokens, which means a long pasted article about search engines becomes dozens of separately retrievable fragments.

The moment it clicks

The real test is a plain request, such as asking the assistant to check a specific note called "homelab improvements".

It finds the actual file, cites it by name and quotes the exact line back with a timestamp attached. The citation carries a source document name and a publication date. Retrieval works as advertised: a file gets named and found, with no invented content and no vague paraphrase, just the right information plus a citation.

The amusing part comes when the model tries to explain the note. It describes Claude Code as some kind of personal assistant or service. Mistral 7B, running offline on a homelab box with only a CPU, has no idea what Claude Code or an MCP server is. It can fetch the words but can't fully grasp them.

That is the more useful lesson. Retrieval and comprehension are separate problems. RAG puts the right context in front of the model, but understanding that context is still capped by the model. With a small 7B model and no internet, the ceiling arrives quickly.

What went wrong and what it taught

A few assumptions didn't survive the build.

• RAG isn't magic. It's only as useful as what it's given, and messy notes produce messy results. Garbage in, garbage out.
• Networking becomes part of the project early. Once services are spread over several machines, getting them to talk reliably matters as much as configuring each one.
• Don't blame the model too fast. Sometimes the trouble lies in the context supplied. Good retrieval can make a small model surprisingly useful, but it can't supply knowledge the model never had.

What you end up with, and its cost

A year of saved links, ideas and unfinished thoughts becomes retrievable, and not only by keyword but semantically. Ask about a concept and it surfaces related notes you had forgotten were related, each with a citation back to its source file.

The test machine was an EliteDesk with an i5, 32 GB of RAM and no GPU, and everything ran offline. Measurements from that single run:

The roughly 54 second gap between the two timings is unexplained. A plausible cause is Ollama reloading the model from a hard disk.

None of this is fast, and it isn't meant to be yet. It's the first day of a homelab RAG setup still finding its feet and not a benchmark to compare with anything serious. It will be fun to look back at these numbers once local hardware manages far more tokens per second.

Where this leaves you

The project begins with a simple aim, letting a local AI see your notes, and teaches more than that. Along the way you learn how RAG works, what vector databases do, what makes retrieval good, and why the right context can count for more than a bigger model.

More to the point, the result is something you'll actually use. Twelve months of saved knowledge stops being scattered over folders and becomes something you can simply ask about. It all runs on a second-hand laptop in the corner of the room, which isn't bad for a machine that was heading for retirement.

RAG works. Understanding is another story.

Further reading

The documentation for AnythingLLM, Ollama, LanceDB and Obsidian covers the details. The original RAG paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", explains the idea behind it all. Everything described runs on repurposed hardware with no cloud, no subscription and no data leaving the network.