Beyond Top-K: Relevance Thresholds, Hybrid Search and Reranking in RAG

Retrieval-augmented generation is usually introduced as a three-step recipe: find documents related to the question, hand them to a language model, and let it write the answer. The diagram is simple; making it reliable is not. A pipeline that connects an embedding search straight to an LLM will answer confidently from weak context, miss exact identifiers and bluff when the knowledge base has nothing to say. Here is where that naive design breaks and which stages, from structure-aware chunking to measurable evaluation, turn it into a retrieval architecture you can trust.

The naive pipeline and its hidden assumptions

The baseline most tutorials build looks like this: split documents into chunks, embed them, store the vectors, and at query time embed the question, fetch the top_k nearest chunks and paste them into the prompt. It works in a demo, but it silently assumes that each chunk is a meaningful unit and that "nearest" means "relevant".

Chunk along the document's structure

Consider an employee handbook. The weak approach cuts it into arbitrary 1,000-character slices, which happily splits a policy in the middle and glues the end of one topic to the start of another. The stronger approach follows the document's own outline, producing chunks such as Remote Work, Security, PTO and Expenses.

A section that is self-contained gives the embedding model enough context to capture what the text is actually about and how its statements relate. The resulting vectors are cleaner, and semantic search returns more relevant hits.

Top-K returns the closest results, not the relevant ones

Better chunks lead straight to the next misconception. Setting top_k = 5 is often read as "give me the five relevant results". What it really asks for is the five closest results, whether or not any of them are good. A typical score distribution for a remote-work question might be:

Remote Work       0.62
Paid Time Off     0.36
Security          0.30
Expenses          0.28
Other Policy      0.24

The first hit is probably what the user needs. The other four are weak matches that only made the list because something had to fill the slots. Passing all five to the model mixes useful information with possibly useful, loosely related and irrelevant text. The model must now separate signal from noise on its own, which raises token cost and makes the answer less predictable.

Unanswerable questions need their own behavior

A particularly important case is a question the knowledge base simply cannot answer, such as "What is the company's health insurance deductible?" when the handbook never mentions it. Top-K will still return five chunks, and a naive pipeline will treat the closest one as evidence and guess.

The right behavior in an enterprise setting is to state that the available documents do not contain enough information. A well-designed system should be able to return something like:

No sufficiently relevant context was found.

Add a relevance filter between retrieval and the model

Handling both weak matches and unanswerable questions requires one more stage. The naive flow is:

Vector Search
   ↓
Top-K
   ↓
LLM

The improved flow treats Top-K as a candidate list and inserts a filter before the model:

Vector Search
   ↓
Top-K candidates
   ↓
Relevance Filter
   ↓
LLM

The simplest filter is a similarity threshold. Candidates at or above it survive:

score >= threshold
    → keep

and anything below it is dropped:

score < threshold
    → discard

If nothing survives, the system returns the "no relevant context" response instead of calling the model with noise.

Choosing the threshold from measurements

The threshold should come from data. One small experiment on an enterprise handbook dataset compared recall on answerable ("known") queries with the rejection rate on unanswerable ("unknown") ones across several values:

| Threshold | Known-query recall | Unknown-query rejection |
| --------: | -----------------: | ----------------------: |
|      0.20 |               100% |                      0% |
|      0.25 |               100% |                      0% |
|      0.30 |               100% |                     50% |
|      0.35 |               100% |                     50% |
|      0.40 |               100% |                     50% |
|  **0.45** |           **100%** |                **100%** |
|      0.50 |               100% |                    100% |
|      0.55 |               100% |                    100% |

On this dataset, 0.45 was the first threshold that kept every known query while rejecting every unknown one, making it the best separation among the values tested. That number is not portable. Similarity scores depend on the embedding model, the corpus, the phrasing of queries and the retrieval setup; different models spread their scores over very different ranges. The rejection rate moving in 50% steps also suggests a very small set of unknown queries, so a real deployment needs a larger labeled set before trusting the cutoff. What transfers is the method: measure retrieval behavior on known and unknown questions and pick the threshold from evidence, not intuition.

Retrieval and ranking are different jobs

Suppose retrieval returns 20 candidates. It has done its job: it found 20 pieces of text that are plausibly related. A second question remains: which five of them are the best context for this particular question? That is the job of reranking.

Retrieval is optimized for recall over a large collection, narrowing something like 10,000 chunks to a manageable candidate set:

10,000 chunks
      ↓
retrieval
      ↓
50 candidates

Reranking then re-scores that small set more carefully, typically with a model that reads the question and each candidate together, and keeps the strongest few:

50 candidates
      ↓
reranker
      ↓
5 strongest candidates

The pipeline now looks like this:

Question
   ↓
Embedding
   ↓
Vector / Hybrid Search
   ↓
Candidate Set
   ↓
Reranking
   ↓
Best Context
   ↓
LLM

Semantic search misses exact terms

Embeddings excel at meaning, but enterprise data is full of tokens whose value lies in their exact spelling:

INC-48271
ERR_CONNECTION_RESET
POL-104
AWS us-east-1
customer_12345

An embedding model may understand that a query is about a connection error while ranking the chunk containing the precise code ERR_CONNECTION_RESET below a generic networking paragraph.

Hybrid retrieval combines both signals

The standard answer is hybrid retrieval, which runs semantic and lexical search together:

Semantic Search
+
Keyword / Lexical Search

Both searches run for the same question and their results merge into one candidate set, often with a fusion method that combines the two rankings:

Question
                    │
          ┌─────────┴─────────┐
          ↓                   ↓
   Semantic Search      Keyword Search
          │                   │
          └─────────┬─────────┘
                    ↓
              Candidate Set

The system gains semantic similarity for paraphrased questions and exact term matching for identifiers.

The production-oriented pipeline

With every stage in place, the full flow is:

Documents
    ↓
Semantic Chunking
    ↓
Embeddings
    ↓
Vector / Hybrid Retrieval
    ↓
Relevance Filtering
    ↓
Reranking
    ↓
LLM
    ↓
Answer + Sources
    ↓
Evaluation + Observability

The model no longer sits directly behind a vector database. A real retrieval architecture now stands between the user and the LLM. For a deeper look at the ranking stage and when its latency pays off, see our piece on why reranking must earn its latency.

A baseline worth building first

A good way to learn these trade-offs is to build incrementally. A small baseline might combine Python, FastAPI, OpenAI embeddings and LLMs, and Pinecone as the vector store, together with:

• heading-aware chunking and chunk metadata
• semantic retrieval with a configurable top_k
• similarity filtering
• source attribution
• retrieval evaluation

Its flow is:

Question
   ↓
Embedding
   ↓
Pinecone Retrieval
   ↓
Top-K Candidates
   ↓
Similarity Threshold
   ↓
Relevant Context
   ↓
LLM
   ↓
Answer + Sources

The natural next step is to add hybrid retrieval and reranking and compare them against this baseline with the same evaluation set, so every improvement is measured rather than assumed.

Measuring whether the system is reliable

"Does the chatbot work?" is not a useful question. Break it into dimensions you can measure:

• Retrieval quality: did the correct information come back at all?
• Ranking quality: did the most useful context land near the top?
• Groundedness: is the answer supported by the retrieved context?
• Citation accuracy: do the cited sources back up the claims?
• Unknown-query handling: does the system recognize when the answer is not in the knowledge base?
• Latency: how long do retrieval and generation take together?
• Cost: what does each query cost?

The goal shifts from "can the app answer questions?" to "can we measure whether the retrieval architecture is reliable?"

Key takeaways

• RAG is more than giving an LLM access to documents; the hard decisions are what to retrieve, what to trust, what to pass on and when to refuse.
• top_k guarantees quantity, not relevance, so filter candidates with a threshold calibrated on your own data.
• Hybrid search catches the exact identifiers that embeddings blur, and reranking turns a broad candidate set into precise context.
• Answer quality is determined largely before the model sees any context: better context leads to better answers and more reliable systems.
• Treat production RAG as an architecture problem with measurable stages, not as a single LLM integration.