turbovec: Compress RAG Embeddings 16x With TurboQuant in Rust

Take a retrieval pipeline of the kind that actually ships: ten million documents, each embedded with OpenAI's text-embedding-3-small at 1536 dimensions and stored as float32. Before a single byte of metadata gets added, the vectors alone weigh 31 GB. At that size, "where does this run?" stops being a casual question. It means a memory-heavy cloud machine, likely hosted by someone else, along with all the privacy and operations baggage that comes with it.

Push the same data through turbovec at 2 bits per dimension and it shrinks to about 4 GB, which fits in a laptop's RAM with plenty to spare. The rest of this piece covers why that works, where it's the right pick and where another tool serves better.

What turbovec is

turbovec is a Rust vector index with Python bindings, MIT-licensed, with roughly 950 GitHub stars when this was written. It implements TurboQuant, a vector quantization algorithm from researchers at Google Research and NYU that was accepted at ICLR 2026.

One detail is worth knowing: Google never released reference code for its own algorithm, so the community stepped in. The RyanCodrai/turbovec project targets vector search, which is the retrieval use case, while a separate project, hackimov/turboquant-kv, applies the same idea to compressing the KV cache of language models. Both are independent open-source efforts.

In other words, using turbovec means using a community implementation of a Google Research algorithm, and on the benchmarks the FAISS team itself uses, it beats Meta's FAISS, long the production standard for in-memory vector search.

How the algorithm works

Once you see it, the method is surprisingly tidy. It runs in four steps.

1. Separate length from direction

Every embedding has a magnitude (its L2 norm) and a direction. Pull the magnitude out and keep it as one float per vector, and what's left is a unit-length direction on a high-dimensional sphere. That sounds minor, but everything after it depends on it, because vectors living on a sphere surface can be handled with geometry.

2. Rotate randomly

Next, every vector is multiplied by the same random orthogonal matrix, generated once from a QR decomposition of a Gaussian matrix. This is the step that feels like a magic trick on first read.

The intuition: in high dimensions, a random rotation makes the distribution of each coordinate predictable. After rotating, every coordinate of every vector follows a known, tightly concentrated Beta distribution, which approaches a Gaussian as dimensionality grows. The distribution is the same whatever the input. Legal documents, product reviews and medical literature all end up looking statistically alike, so the scheme doesn't depend on the data. Mathematicians call this concentration of measure on the sphere, and it explains why no training pass is needed: the rotation does the work a trained codebook would otherwise do.

3. Pick the best buckets

Knowing the distribution each coordinate follows means the best way to bucket values can be worked out ahead of time. Two-bit quantization gives four buckets per coordinate, four-bit gives sixteen. The calculation is called Lloyd-Max scalar quantization, which answers a simple question: given a known distribution, where should the dividing lines sit to keep average rounding error as low as possible? With the distribution known, the answer comes out in closed form. And because the post-rotation distribution is identical for all vectors, the buckets come from the math rather than from your data.

4. Pack the bits

Each coordinate is now a small integer: 0 to 3 at 2-bit, or 0 to 15 at 4-bit. Packed tightly, a 1536-dimensional vector that took 6,144 bytes as float32 needs 384 bytes at 2-bit or 768 at 4-bit. That's 16x and 8x compression.

Searching without unpacking

Queries never decompress the stored data. The query is rotated once into the same space, then compared straight against the quantized codes with lookup tables built in advance. The inner loop relies on SIMD instructions, with NEON for ARM, AVX-512BW for recent x86 and AVX2 as the fallback.

This is where the library, as distinct from the algorithm, earns its keep. Its hand-written SIMD kernels outrun FAISS's FastScan by 12 to 20 percent on ARM and match or beat it on x86.

There's also a theoretical result behind the method. The TurboQuant paper proves its distortion stays within a factor of about 2.7 of the information-theoretic lower bound, Shannon's distortion-rate limit. Put plainly: information theory sets how far data can be squeezed before something is lost, and TurboQuant gets within 2.7x of that wall without any training. Nobody can do dramatically better for a given bit budget, and the paper backs that with a formal proof.

Three cases where it pays off

Memory that fits

Back to the opening numbers: 31 GB of float32 versus 4 GB through turbovec at 2-bit. Having an index that small changes what's possible.

• It can live next to the application on one server, with no separate vector database instance.
• Developers can run it on laptops for local testing.
• Cheaper memory tiers on cloud instances become viable, which cuts cost noticeably.
• The whole index stays in RAM, with no disk spill and no sharding across nodes.

For collections in the 100K to 100M vector range, which covers most knowledge bases, documentation sites, support archives and product catalogs, this is the gap between having to design around the vector index and treating it as just another Python object.

Privacy and air-gapped setups

Retrieval pipelines carry a quietly serious privacy issue. Two attacks illustrate it:

• BadRAG reaches a 98.2% success rate by poisoning only 0.04% of a corpus with carefully built adversarial documents.
• Vec2Text recovers short original texts from their embeddings, matching them exactly 92% of the time.

Embeddings carry no cryptographic protection, so anyone with read access to a vector store can recover what was embedded. Each cloud vector DB call, each embed() request to OpenAI and each upsert() to Pinecone is a point where data leaves your perimeter.

Plenty of real deployments are fully air-gapped already. At UC San Diego, Onyx handles 37,000 users without needing the internet, and Ramp, Thales, L3Harris and Astranis run comparable setups. turbovec's own tagline, no managed service and no data leaving your machine or VPC, fits that group well. Pair it with a locally hosted embedding model such as BGE-M3 or BGE-large-en-v1.5 and retrieval makes zero external network calls.

No codebook to train

This advantage is structural and easy to underrate. Classic Product Quantization, the PQ in IndexPQ and IndexIVFPQ, needs a codebook trained on representative data. When the data drifts, say new product categories, new languages or new document types, that codebook goes stale and recall suffers.

TurboQuant has no codebook at all. The random rotation makes the math hold whatever the input looks like. In practice that means:

• Streaming inserts work cleanly, with no batched retraining.
• Switching embedding models, from OpenAI to BGE to Voyage, doesn't force an index rebuild from scratch.
• Multi-tenant indexes need no per-tenant calibration.
• The cold-start problem, where too few vectors exist to train a decent codebook, disappears.

For systems that grow gradually or span mixed document types, that simplicity is a real operational win.

When to use something else

Some retrieval patterns are better served elsewhere, and it's best to be straight about them.

• High-recall retrieval. HNSW with full-precision vectors remains the gold standard. turbovec gives up a few recall points in exchange for 16x compression. That's acceptable for most RAG work and painful when a missed result is expensive.
• Metadata filtering and hybrid search. turbovec is only an index, with no database around it. Something like "find vectors close to X, restricted to documents tagged compliance and edited after December 2025" needs a real vector DB such as Qdrant, Weaviate, Milvus or pgvector.
• Low-dimensional embeddings (256 dimensions or fewer). By its own README's admission, turbovec lags FAISS by 3 to 6 recall points on GloVe at d=200, since the Beta-to-Gaussian approximation is looser when there are few dimensions and the theory's advantage shrinks. If the stack uses older or smaller embedding models, FAISS is the safer choice.
• Billion-vector scale. turbovec scans every compressed vector exhaustively. That's fine up to around 100M vectors. Beyond it, IVF or graph-based partitioning is needed on top, and turbovec doesn't include that.
• Hosted-service preference. A team that wants a managed vector database with an API key will find turbovec the wrong tool, however good the engineering.

Getting started

For in-memory retrieval over 100K to 100M vectors with dimensions of 384 or more, and a team ready to run its own infrastructure, here's the practical path. Install from PyPI:

pip install turbovec

Rust projects add the crate instead:

cargo add turbovec

Basic usage takes a few lines to build and query an index:

from turbovec import TurboQuantIndex

# Build the index
index = TurboQuantIndex(dim=1536, bit_width=4)
index.add(document_embeddings)
# Search
scores, indices = index.search(query_embedding, k=10)
# Persist
index.write("kb.tq")
loaded = TurboQuantIndex.load("kb.tq")

That is the whole surface for the simple case.

Choosing a bit width

This is the most consequential setting, and the trade is straightforward.

• 2-bit: 16x compression versus float32, a larger recall gap, and more sensitivity to dimension.
• 4-bit: 8x compression, with recall that essentially matches FAISS on OpenAI's 1536 and 3072-dimensional embeddings. Both converge to 1.0 by k of 4 to 8.

The sensible default is 4-bit. Move to 2-bit only after measuring real memory pressure and the effect on recall for your own queries.

Choosing an index type

There are two. TurboQuantIndex addresses vectors by their position, which keeps it simple and a touch faster when your IDs simply follow insertion order. IdMapIndex lets you attach your own uint64 IDs and removes entries in constant time, so it's the choice when keys come from a database or documents get deleted.

For most RAG systems IdMapIndex is the better default, since application-database IDs survive deletes and rebuilds without breaking external references.

from turbovec import IdMapIndex
import numpy as np

index = IdMapIndex(dim=1536, bit_width=4)
index.add_with_ids(vectors, np.array([1001, 1002, 1003], dtype=np.uint64))
scores, ids = index.search(query, k=10)  # returns your IDs
index.remove(1002)  # O(1) by ID

Embedding models

For air-gapped retrieval the embedding model has to run locally, because calling OpenAI's API cancels out the privacy benefit. Two combinations are well tested:

• BGE-M3 with turbovec. A strong multilingual model that runs locally and makes a good default for international knowledge bases.
• BGE-large-en-v1.5 with turbovec. English-only, smaller and faster, and a good default for English documentation.

Where privacy isn't a hard requirement, OpenAI's text-embedding-3-small (1536 dimensions) or text-embedding-3-large (3072) work fine with turbovec. Just remember that the cloud embedding call, not the index, is where data leaks.

Framework integrations

turbovec offers first-class integrations for LangChain, LlamaIndex and Haystack:

pip install turbovec[langchain]
pip install turbovec[llama-index]
pip install turbovec[haystack]

A stack already built on one of these should take that route, since it involves the least friction.

Limits to keep in mind

• Single machine, in memory. There's no distributed mode, so horizontal scaling means sharding in the application layer.
• Exhaustive scan. Every compressed vector gets scanned on every query. SIMD keeps that fast up to about 100M vectors, and past that an IVF or graph layer is needed.
• Hardware dependence. The 12 to 20 percent ARM gain was measured on Apple M-series chips. Older ARM or x86 without AVX-512 sees smaller gains, though it still beats FAISS in most configurations.
• Recall at low dimensions. Below roughly 256 dimensions FAISS wins by a visible margin. The method is at its best with the 768-plus dimensions that current embedding models emit.

The bigger pattern

turbovec fills a specific niche: in-memory vector search, mid-sized corpora, infrastructure you own, and the dimensions modern embedding models output. It doesn't try to replace Qdrant or Pinecone. It's a lower-level building block for the place where FAISS would otherwise go, with better compression and no training pass.

The backstory is arguably the more interesting part. A four-author Google Research paper was accepted at ICLR 2026, the authors released no code, and two community developers independently built implementations, one for KV-cache compression and one for vector search. The vector-search one beats the production incumbent on that incumbent's own benchmarks.

The path from academic algorithm to community implementation to a production-grade library that outperforms the standard tool is going to repeat, because research output is accelerating faster than vendors can productize it. turbovec is one worked example. Understanding how it works, and where it fits, is useful for anyone designing retrieval systems today.