Tuning HNSW Indexes and Scaling Vector Search for Production RAG

A retrieval-augmented generation (RAG) prototype with a few thousand vectors answers instantly on almost any setup. As the collection grows into the millions, latency and resource usage start to dominate, and index configuration becomes a production concern. This guide covers the two HNSW parameters that matter most, how to set them in Chroma, and how to recognize when tuning is no longer enough and the system has to scale.

The two HNSW parameters that control almost everything

You do not need the full theory of HNSW (Hierarchical Navigable Small World) graphs to tune one. The index links each vector to nearby vectors across several layers, and a query walks this graph toward its nearest neighbors. Two parameters shape that graph and that walk. For more background, see our explainer on how vector databases work, from embeddings to hybrid search.

M: maximum connections per node

M sets the maximum number of neighbor links each node keeps in the graph.

• Low M: fewer links, a smaller index, less memory and faster traversal, but lower recall because the search has fewer paths to good candidates.
• High M: more links per node, more memory, a larger index and a slower build. Queries get somewhat slower, but recall improves because the graph offers more routes to the true nearest neighbors.

M is fixed when the index is built, so changing it later means rebuilding the index.

ef: how hard each search tries

ef controls how many candidates the index keeps in play while it searches for a query's nearest neighbors.

• Low ef: fewer candidates are examined, so retrieval is fast, but relevant documents can be missed and accuracy drops.
• High ef: more candidates are examined, so retrieval is slower, but the result set comes much closer to the exact nearest neighbors.

Many engines split this into a construction-time value, which controls how carefully the graph is built, and a search-time value you can usually change without a rebuild, which makes it the cheapest knob to experiment with.

Balancing accuracy, speed and memory

Tuning HNSW is a three-way trade-off between accuracy, speed and memory. These combinations are reasonable starting points for common project stages, with ef meaning the search-time value:

+----------------+----+-----+----------+
| Use Case       | M  | ef  | Priority |
+----------------+----+-----+----------+
| Prototype      | 16 | 40  | Speed    |
| Production     | 16 | 100 | Balanced |
| High accuracy  | 32 | 200 | Accuracy |
+----------------+----+-----+----------+

Treat them as baselines to benchmark against: measure recall and latency on representative queries before and after each change.

Setting HNSW parameters in Chroma

Each vector database exposes these settings differently. In Chroma, they can be passed as collection metadata when the collection is created. The example below sets M to 16, the construction-time ef to 100 and the search-time ef to 50:

#Chroma-HNSW Settings
collection=client.create_collection(
name='new_collection',
metadata={
'hnsw:M':16,
'hnsw:construction_ef': 100,
'hnsw:search_ef': 50
}
)

This metadata-key style may differ in newer Chroma releases, which also offer a dedicated configuration object, so check the current documentation for your version. Set M and construction ef before ingesting data, since they are baked into the graph.

When tuning stops being enough

Even a well-tuned index eventually hits a ceiling. Once retrieval regularly takes 100 ms or more, a common cause is that the index no longer fits comfortably in memory, and further parameter tweaks will not fix that. A rough estimate helps: raw vector storage is the number of vectors times their dimensions times 4 bytes for 32-bit floats, and the HNSW graph adds overhead that grows with M. At that point the answer is scaling, which comes in two forms.

Vertical scaling

For collections in the range of roughly 5 to 10 million vectors, adding RAM or CPU to the existing machine is usually the first step. With enough memory, the whole index fits in RAM, which restores fast retrieval and improves overall efficiency.

Vertical scaling needs no architectural or code changes, but a single machine has a memory ceiling, and larger instances get expensive quickly.

Horizontal scaling

Beyond about 10 million vectors, horizontal scaling becomes the practical option. The collection is sharded, meaning the data is split across several instances, and each shard holds and searches its own portion.

This removes the single-machine ceiling. The price is complexity: every query fans out to the shards, and their partial results must be merged into one ranked list, which adds latency and operational work.

Key takeaways

• M shapes the graph: higher values raise recall and memory use, and changing it requires a rebuild.
• ef shapes the search: raising the search-time value is the cheapest way to trade latency for recall.
• Start from a balanced configuration such as M 16 with a search ef of 100, then benchmark on your own queries.
• Most production RAG systems get by with index tuning plus vertical scaling; sharding is powerful but rarely needed, so reserve it for collections that genuinely outgrow one machine.