Vector Quantization Compared: Int8 Versus Binary Recall and Storage

Shrinking embeddings sounds like free money, and plenty of guides stop there. A quick test shows that "safe" and "reckless" compression sit much further apart than those guides admit.

Here's the headline. Int8 quantization usually keeps 95% or more of retrieval accuracy while cutting storage by 4x. Binary quantization offers a spectacular 32x reduction, but unless the embedding model was trained for it, you may see a third or more of your retrieval accuracy disappear. These two aren't points on one sliding scale. They behave differently enough that choosing the wrong one for your model can quietly ruin search quality.

Quantization works by storing each dimension with less precision. You go from 32-bit floats to 8-bit integers (scalar quantization), or all the way to one bit per dimension (binary quantization). The savings are real: RAM use drops by up to 24x with binary and 3.75x with scalar, and the effect grows once a corpus passes millions of vectors, which is increasingly normal for agent memory and enterprise RAG in 2026.

What the numbers look like

The test corpus was synthetic: 2,000 vectors with 768 dimensions, grouped into 20 loose clusters to imitate real semantic structure. Retrieval accuracy was measured as recall@10 against the uncompressed float32 baseline, after applying each of two quantizers. First, int8, which scales by the largest absolute value and rounds into the int8 range:

def quantize_int8(vecs):
    scale = np.max(np.abs(vecs))
    q = np.round(vecs / scale * 127).astype(np.int8)
    return q.astype(np.float32) / 127 * scale

And binary, which keeps nothing but the sign of each value:

def quantize_binary(vecs):
    return (vecs > 0).astype(np.float32) * 2 - 1

The results came out like this:

Format       | Recall@10 vs float32   | Storage (MB)   | Reduction
----------------------------------------------------------------------
float32      | 1.000 (baseline)       |          6.144 | 1x
int8         |                  0.961 |          1.536 | 4.0x
binary       |                  0.326 |          0.192 | 32.0x

Int8 kept 96% of retrieval accuracy for a 4x cut in storage, which is a good deal for most production systems. Binary, on this generic embedding simulation that wasn't built with quantization in mind, fell to 32.6% recall despite its flashy 32x compression. That's no rounding error. Roughly two of every three relevant results that the float index would have returned are gone.

Why binary breaks so much harder

It comes down to how much information survives. Int8 keeps a fine-grained number for every dimension, which is enough resolution to preserve most of the relative geometry between vectors. Binary keeps only the sign and throws magnitude away entirely, a far harsher simplification of the embedding space.

This isn't a law of nature, though. It depends on the model. Independent benchmarking has found that some embedding architectures barely notice binary quantization and hold around 98% of float-level quality, while other families take a bigger hit and land near 87-92%, depending on model size. What separates them is whether the model was trained so that it holds up under quantization.

Which one to pick

• Make int8 your default. You give up a little accuracy in a way that's easy to predict, and the space you save is still big enough to matter on a large index.
• Use binary only with a model made for it. Providers now ship quantization-aware embedding variants because generic embeddings degrade badly under binary compression. Check that your exact model has been validated for it before deploying, and don't assume the technique is safe everywhere.
• Look at Matryoshka-style dimension truncation as a middle route. Some newer models can be cut to fewer dimensions (2,048 down to 256, for example) and keep most of their retrieval quality, which gives you a tunable trade-off between int8's modest savings and binary's aggressive but risky ones.
• Measure recall on your own corpus first. How tightly your real embeddings cluster, and how many near-duplicate results your queries can tolerate, will push these numbers in either direction from whatever any generic benchmark says, this one included.

Common questions

How much accuracy does int8 cost? Usually very little, commonly under 5% recall loss versus full float32, in return for about 4x less storage.

Is binary quantization safe? Only with embedding models trained to tolerate it. Generic models can lose a third or more of retrieval accuracy, whereas quantization-aware ones can keep 98% or more of float-level quality.

How much storage do you save? Around 4x with int8. Binary can reach 24-32x, with a much bigger and less predictable accuracy bill.

Do you need to re-embed everything? Not necessarily. You can quantize existing float32 embeddings after the fact, as the code above does. Models trained with compression in mind will generally beat an off-the-shelf embedding quantized afterward, though.

One last caveat: the corpus here is synthetic and cluster-structured, only for illustration. Real recall depends on your embedding model, the structure of your data and your query patterns, so benchmark on your own data before settling on a compression level.