Local RAG Without Vectors: SQLite FTS5 and BM25 for Code Search

For a couple of years, the default recipe for local code search has been the same: split the repository into chunks, turn every chunk into a 1536-dimensional embedding, and keep an HNSW index alive in a background process. It felt modern, as if the laptop had grown a small search engine that understood what the code meant. On a busy workstation, though, that recipe carries a cost that rarely shows up in a README.

Headline numbers around local vector search are impressive: strong MTEB retrieval scores, millions of code tokens embedded in minutes, "instant semantic intelligence" on consumer hardware. Read the memory notes instead of the marketing, and a different picture appears. Dense vector indexing buys its recall by spending memory bandwidth, RAM or VRAM, and predictability.

Vector everything meets the memory wall

Run a vector database next to a local LLM, an IDE and a few Docker containers and it isn't idle. It keeps large graph structures resident in RAM or VRAM, fights the other tools for bandwidth, and brings in semantic drift: ask for the exact string ERR_AUTH_EXPIRED and get back code that is merely about authentication.

For that reason, plenty of teams are quietly returning to native C algorithms, namely SQLite FTS5 with BM25. Instead of vector math for local lookups, they use deterministic term-frequency ranking and surface exact identifiers in under 5 milliseconds.

Concept matching versus predictable speed

This is more than a tweak. It changes how developer tooling gets built. Swap local vector overhead for FTS5 and you give up theoretical concept matching in return for lean, fast execution.

• Efficiency: dropping dense indexing frees gigabytes of memory. Work that spent around 200 ms computing embeddings finishes natively in under 5 ms.
• Precision: lookups of exact keywords and function names, such as getUserById(), are fully deterministic, so the index won't hand back unrelated function bodies.

The two flows compare like this:

[Dense Vector RAG Flow]
Code Query -> Generate Vector Embeddings -> HNSW Index Search -> Semantic Drift Risk -> LLM Context

[Deterministic BM25 Flow]
Code Query -> Native SQLite C-Index -> Exact Keyword Match (5ms) -> Minimal LLM Context

The symptoms you may already know

Anyone running local agents or code search has probably noticed the friction:

• Resource contention. The fan spins up, the battery lasts two hours, and local inference crawls because a background engine is competing for memory.
• The exact-match penalty. Searching for a specific error string or variable returns five files that are conceptually near authentication, but not the single file that contains the actual flag.

A side-by-side view, with rough numbers:

Dimension	Local vector DB (Chroma, Qdrant)	SQLite FTS5 (BM25)
Memory footprint	About 1.5 to 4 GB of RAM or VRAM	About 15 to 30 MB of RAM
Search latency	120 to 400 ms, mostly embedding time	Under 5 ms, native C
Exact identifier matching	Weak, prone to drift	Strong, deterministic keyword match
System overhead	Background daemons, C++ bindings	Nothing running while idle
Best for	Broad, thematic searches in prose	Precise lookups in code, logs and local RAG

Treat these as ballpark figures, since results depend on corpus size, model choice and hardware.

What it means for engineering leads

The reflex of attaching a vector database to every local problem is fading. A more useful habit is a sense of proportion. A senior architect's skill isn't fitting heavier AI infrastructure onto developer laptops. It's recognizing when ten lines of native code or a 30 MB SQLite file beats a neural pipeline measured in gigabytes.

As tools get quicker and workflows get leaner, the stacks that win won't be those running the biggest models. They'll be the ones that ship clean code without bloat. That doesn't make embeddings useless: for fuzzy, concept-level questions over prose, they still earn their place. For finding an exact identifier in a codebase, start with the cheap deterministic tool and add vectors only where it fails.