Benchmarking Five Embedded Vector Stores: FAISS, Chroma, LanceDB, VectLite

Most "best vector database" comparisons read like the vendors' own landing pages rephrased: Pinecone is fast, Qdrant is solid, Chroma is simple, everybody wins. This one starts from a narrower need. No cluster, no Docker, no open ports, no managed plan. Just an embedded vector store that's one local file, a pip install and three lines of Python. So here's a home-made benchmark of five engines for that exact use.

The numbers

The data set held 5,000 vectors of 384 dimensions, queried 50 times with cosine similarity. Everything ran on macOS (x86_64) with Python 3.12.8, using each engine's default settings.

In ten seconds: FAISS is the fastest, though it's hardly a database. Chroma is the easiest start, but its default recall of 84% is a problem many people quietly ignore. VectLite reaches 96.6% recall in embedded mode with no tuning, which was the surprise. LanceDB chases perfect recall and pays in latency. NumPy is the referee and not a contender.

Why "fastest" is the wrong ranking

The vector DB market is a marketing brawl. Each vendor publishes a benchmark it happens to win, comparison pieces repeat the figures without context, and everyone stares at latency. Latency alone misleads. Answering in half a millisecond with the wrong neighbors is worse than taking 5 ms to return the right ones. That's what recall measures, and it's why serious suites such as ANN-Benchmarks (https://ann-benchmarks.com/) plot recall against throughput.

So this run measures both at once: p50, p95 and p99 latency and queries per second on one side, and recall@10 against an exact cosine search in plain NumPy on the other.

What was and wasn't tested

The project is a small Python package, vectlite-benchmark-lab. Every engine implements one shared VectorStore interface and the runner calls them in turn. It covers:

• Batch ingestion
• k-NN search with recall measurement
• Exact-match metadata filtering on a tenant_id
• Cold start time
• Backup and portability
• Exports to CSV, JSON and Matplotlib charts

And what it leaves out, stated up front: no distributed or multi-node setup, nothing beyond 5,000 vectors in this round, no index tuning at all, and no Qdrant or pgvector, since they need a running server and the aim was a pure local run. That last choice is deliberate. The first useful question isn't who wins after two weeks of tuning. It's how an engine behaves when you install it and touch nothing.

Method

Ground truth is a brute-force cosine search in NumPy. An engine's recall is the share of true neighbors it returns, averaged across queries. The configuration for the run:

vectlite      : 0.10.0
vectors       : 5,000 (synthetic, normalized)
dimensions    : 384
queries       : 50
top_k         : 10
metric        : cosine
filter        : tenant_id (exact-match)
warmup        : 5 queries discarded
hardware      : macOS x86_64, Python 3.12.8

To reproduce it on your machine, run the full command:

PYTHONPATH=src python -m vectlite_benchmark_lab run \
  --stores vectlite,faiss,chroma,lancedb,numpy_exact \
  --vectors 5000 --dimensions 384 --queries 50 \
  --top-k 10 --batch-size 500 \
  --output-dir results/vectlite-0.10.0 \
  --data-dir data/vectlite-0.10.0

Then compare your results with these and look at the differences.

FAISS: the speed ceiling

FAISS wins every latency column: 0.65 ms p95 with perfect recall, which is about what you'd expect from a C++ library tuned for exactly this for a decade. But it isn't a database. It has no native metadata filtering, no built-in persistence and no portable backup. You can wrap it in a layer that adds those, and people do, yet out of the box it's closer to a math library than something you'd ship to customers. Comparing it with Chroma is like comparing an F1 engine with a Toyota: one is faster around a track, the other gets the groceries home. It stays in the table as a reference. If all you need is raw vector search, it's the obvious pick.

Chroma: easy start, watch the recall

Chroma is the default for RAG prototypes, and the developer experience earns that. Install chromadb, write five lines and documents are being indexed.

The catch is recall. Out of the box on this data, with no filter applied, Chroma finds only 84% of the true neighbors, which means roughly one expected hit in six never shows up. That's fine for a prototype. In a production RAG, where retrieval quality shapes the model's answer, it's a silent leak of relevance that the API never reveals. Adjusting the HNSW settings recovers most of the gap, provided you know that's needed. With filtering, recall is 100%, but p95 latency climbs to 18.76 ms, noticeably slower than the rest on the same job. Verdict: great on day one and still good on day thirty, provided you tune before production.

VectLite: the surprise

Version 0.10.0 reaches 96.6% recall at default settings, the highest of any embedded engine here. Plain search p95 lands at 5.71 ms, in the middle of the pack (Chroma 3.14, LanceDB 12.20). On the filtered workload it takes 10.28 ms at p95, against 18.76 ms for Chroma, so it's close to twice as quick there.

Progress since the earliest releases is visible. Compared with v0.1.x, ingestion went from 47 to 154 vec/s (about 3.3 times faster), recall rose from 0.948 to 0.966, and filter p95 fell from 18.6 ms to 10.3 ms, a 45% drop.

Ingestion is the weak spot. 154 vec/s is slow next to Chroma's 2,000 and more and very slow next to LanceDB. For a few thousand documents indexed once, that doesn't matter. For a million vectors it means an evening of waiting, so it's probably the next thing to fix. Still, this was the one engine in the group whose numbers needed no excuses.

LanceDB: perfect recall, slower search

LanceDB reaches 100% recall and ingests 20,120 vec/s, impressive on paper. The cost is search latency: 12.20 ms p95 is the slowest of the five. For a batch pipeline processing thousands of queries overnight that's irrelevant. For an interactive tool with a person waiting at the keyboard, it adds up. Good for batch and analytics, a bit slow for interactive use.

NumPy: the referee

Brute-force cosine search over the full matrix gives perfect recall by definition, with acceptable latency up to a few tens of thousands of vectors, after which it falls off a cliff. It's not a product. It's the arbiter every other engine is judged against.

Which one to choose

The benchmark doesn't crown a winner. It exposes trade-offs. A rough mapping by use case:

• Quick RAG prototype: Chroma, thanks to a two-minute setup and a large ecosystem
• Embedded AI memory: VectLite, for the best default recall and a portable file
• Local batch pipeline: LanceDB, for perfect recall and fast ingestion
• Pure ANN without filtering: FAISS, for unbeatable latency
• Distributed production: Qdrant or Milvus, a different category not tested here

Cold start and backup

Two measurements that most benchmarks skip. Startup, the time to reopen the database after a restart: FAISS takes 0.001 ms, LanceDB 2 ms, Chroma 5 ms and VectLite 3,059 ms. VectLite pays dearly on reopen. That doesn't matter for a long-running process, but for a serverless function or a CLI tool spinning up on demand, a three-second cold start is a hard sell and likely the first thing to fix.

Backup: VectLite writes out a 19 MB portable artifact in 54 ms. Copying, versioning and deploying it is just file handling. That's the part of "embedded" that pays the bills, the same property that kept SQLite alive for two decades.

Limits of this run

Every benchmark needs an asterisk. Five here:

1. 5,000 vectors is small, and behavior changes at 100k and above.
2. Only default settings were used, and Chroma and VectLite can almost certainly be tuned higher.
3. No server engines took part: Qdrant, Milvus, Weaviate and pgvector are out.
4. It's a single run, with no confidence intervals and no repeats.
5. It ran on one machine, so other hardware may give other numbers.

The code is open source and the results are reproducible, so a spotted bias or bug is worth an issue.

Next steps and takeaways

This run is a starting point, not a verdict. The short list for the next round: a 100k vector, 768-dimension run across all engines; a RAG-style test using a Markdown and PDF corpus with queries written by people; tuned recall, so engines face off at comparable precision; a shorter VectLite cold start; and repeated runs with confidence intervals.

The search for "the best vector database" ends without a universal answer, since it depends on how you weigh recall against latency, ingestion and startup. What stood out wasn't a runaway leader but that each engine has a personality. FAISS is a speed reference posing as a database. Chroma is an easy on-ramp that needs tuning before production. LanceDB is the perfectionist that doesn't mind being slow. NumPy is the impartial judge that can't scale. VectLite is the one that quietly hit the recall target without negotiation.

For a local agent, a CLI tool or an offline-first RAG on a laptop, VectLite makes a credible case: SQLite made local relational data boring and portable, and VectLite aims for the same role with AI memory. Here that means a 19 MB file that answers in about 6 ms.

Reference links: ANN-Benchmarks at https://ann-benchmarks.com/, Qdrant's benchmark notes at https://qdrant.tech/benchmarks/, FAISS at https://github.com/facebookresearch/faiss, Chroma docs at https://docs.trychroma.com/ and LanceDB at https://lancedb.github.io/lancedb/.