Home / Articles

Evaluating RAG Retrieval Beyond ANN Recall, with Turbopuffer as the Example

A fast vector index can still miss the answer. Learn Turbopuffer's storage tradeoffs, hybrid BM25 plus vector search, and how a golden set measures real recall.

Publisher: REACTAPP.TOP7 min read1671 words

This article is published in English.

Sources: Turbopuffer Made Me Rethink RAG: A Fast Index Does Not Mean Correct Retrieval

Teams starting a RAG project usually open with one question: which vector database to pick. It's a fair thing to ask, yet it isn't the main one. A better database can cut costs, speed up search and make a large system simpler to operate. What it can't say is whether the user actually got the information they needed.

Turbopuffer is a good case to examine for two reasons. Its storage and serving model is unusual, and it highlights the step most RAG teams skip: testing retrieval properly.

How Turbopuffer stores and serves data

The durable copy of the data lives in object storage, with memory and NVMe acting as a cache. Query machines are stateless, so one machine can serve many namespaces and doesn't have to hold a single customer's data forever.

That suits a large corpus shared among many tenants. Picture a B2B product where each customer owns a set of documents but only a few customers are active at any moment. Holding every customer's full index in RAM would be costly. Inactive data can sit cheaply in object storage while active namespaces warm up into NVMe and memory.

A tradeoff comes with this design. For a namespace of one million documents, Turbopuffer publishes a warm-query p50 of 14 ms, while the first query against a cold namespace of that size has a p50 of 874 ms. So "always fast" is the wrong description. Better to say it's cheap while data rests and fast once data is active. Many large RAG systems should take that deal, though it's poor when each request must be quick even for data nobody touched recently.

The product also concentrates on first-stage retrieval, narrowing a huge corpus down to a modest candidate set. The rest stays with the application: hybrid search, reranking and the logic deciding what reaches the model. That split keeps one truth in view. The database can make retrieval possible, and the application still decides whether retrieval is any good.

Why the basic flow falls short

The simplest RAG pipeline looks like this:

User question → vector search → top 5 chunks → LLM answer

Fine for a demo, weak in production. Real users don't ask only tidy semantic questions. They paste error codes, ask about policy exceptions, use acronyms and product names, need an older document version or want an answer that spans two documents.

Using both kinds of search makes sense:

  • Vector search handles meaning. It can tie "time off after having a baby" to a page titled "parental leave".
  • BM25 handles literal terms, so a query containing ERR_PAYMENT_042, SKU-78291 or AuthMiddleware lands on the right page.

Both are available in Turbopuffer via multi_query, and results can be merged with Reciprocal Rank Fusion, known as RRF. The application can then rerank the best 20 to 50 hits and give the LLM only a few strong chunks. Hybrid search isn't automatically superior. Vector and keyword retrieval break in different ways, so an application's own test set should settle whether both earn their keep.

Three things to measure

A high vector-recall number leads many teams to assume RAG is working. That isn't what the number means.

Three separate qualities need measuring:

  1. Index fidelity: does approximate vector search return what exact vector search would?
  2. Retrieval relevance: does the search return the evidence that answers the question?
  3. Answer quality: does the model use that evidence correctly and safely?

Turbopuffer covers the first one well. It samples 1% of live query traffic, compares approximate nearest-neighbor output with exhaustive vector search and tracks how much they overlap. Its guidance puts 90 to 95% recall@10 as a sensible balance for many workloads. That shows whether the index finds what exact vector search would find, and nothing about whether exact vector search suited the question.

Consider a user asking:

What does ERR_PAYMENT_042 mean?

The answer is present in the knowledge base, on an error-code page, but the code shows up only once inside a giant table. Vector search may return three general payment guides because they sit close to the question in meaning, and exact vector search would return those same three. The database then reports 100% ANN recall while the real answer never reaches the model.

A single BM25 search for ERR_PAYMENT_042 could fix this. No amount of vector-index tuning would, since the vector index was never broken. It did its job perfectly and was simply given the wrong job. Hence the core idea: ANN recall isn't RAG recall. One gauges how closely the index matches a mathematical search, and the other gauges whether the user received the right evidence.

Build a small golden set

Before changing chunk size, embeddings, top-K or rerankers, gather a compact set of real questions and mark which chunks should answer each. A hundred questions is plenty to begin with, and it needn't be flawless. Draw on search logs, support tickets and failures users reported, and add a few from domain experts. Each entry needs very little:

{
  "query": "How much parental leave do employees get?",
  "relevant_chunk_ids": ["benefits/parental-leave#3"],
  "query_type": "semantic_paraphrase",
  "must_contain": ["16 weeks"],
  "must_abstain": false
}

The set should mix different kinds of questions:

  • exact identifiers and acronyms,
  • ordinary semantic questions,
  • long, rambling user questions,
  • questions needing two documents,
  • questions with no answer, and
  • permission-sensitive questions where the answer exists but the user mustn't see it.

The no-answer group matters. A RAG system should be able to say it couldn't find something in the available documents, which beats a confident invention.

A rule worth adopting: turn each real RAG failure into a test case before fixing it. Skip that and the bug gets fixed, the example gets forgotten and the same defect comes back six weeks later.

Start with plain retrieval metrics

An LLM judge shouldn't come first. Simple retrieval metrics do the early work:

  • Recall@K asks whether the right chunk appeared in the first K results.
  • Mean Reciprocal Rank (MRR) asks how early the first correct chunk showed up. A correct chunk at rank 1 isn't equal to one at rank 5 behind four noisy chunks.
  • Normalized Discounted Cumulative Gain (nDCG) becomes useful once some chunks can be graded "very relevant" and others "somewhat relevant". Turbopuffer's hybrid-search guidance suggests building a test suite of queries with ideal results and evaluating nDCG.

A single big average hides too much, so results should be split by question type. A useful finding might read: hybrid search lifted overall recall, but nearly all the gain came from error codes and acronyms, with little added for ordinary semantic questions. An average can't say that.

Compare setups one change at a time

Keep the experiment small and repeatable, and run the same golden dataset through a few configurations:

  • Run A: vector-only search, the baseline.
  • Run B: BM25-only search, showing where exact words win.
  • Run C: vector plus BM25 merged with RRF, testing whether hybrid helps.
  • Run D: Run C plus a reranker, showing whether the quality gain justifies the extra latency and cost.

Alter only one thing per run, and log these numbers: Recall at 5 and at 10, MRR (plus nDCG once labels are graded), cold-start and warm latency kept apart, and what 1,000 queries cost. Support for test namespaces and branches of a production namespace in Turbopuffer simplifies the exercise, since real data can be exercised while the live namespace stays untouched.

Turbopuffer's own ANN recall should be tracked too, as an index health check. If it falls, the index isn't matching exact search well. If it's healthy while the golden-set score is poor, the trouble lies elsewhere: embeddings, chunking, hybrid search, filters or reranking. That's a very useful diagnosis.

Then check the answer

Once the right chunks reach the LLM, examine the final answer with five questions. Is it correct? Is every important claim backed by retrieved text? Does it include all the key information? Do the citations support the claims next to them? Did it decline when no answer existed?

Plain facts deserve plain checks: an answer that must state "16 weeks" can simply be searched for that phrase. Reserve an LLM judge for tougher calls, like whether a long reply is grounded and complete. Tools such as Ragas can assist, though they can't stand in for a good test set.

The risky case is when retrieval fails and the LLM still gives a plausible answer from its own training. An end-to-end check may say the answer looks fine while the RAG pipeline has failed. That gets dangerous when the answer is company-specific, recently changed or sensitive.

The bigger lesson

Turbopuffer fits when retrieval must span a large amount of data and many namespaces, with a cost model that doesn't keep idle data expensive. Yet the main lesson isn't about the product. It's about measurement. Turbopuffer checks whether its approximate search matches exhaustive search on live traffic, which is good engineering. A RAG system should copy the habit one layer up by asking whether the right evidence was retrieved, whether the model used it well and whether a new change made a real user workflow worse.

Speed isn't accuracy, since a quick retriever can hand back the wrong document in a few milliseconds. The remedy is to assemble the small golden set, change one variable at a time and convert every failure into a regression test. That's how RAG turns from something that feels like magic into a system that can be improved step by step.