Home / Articles

Hybrid Retrieval for RAG: BM25, Rank Fusion and Reranking Beat Vectors

Pure vector search misses exact terms. BM25 plus embeddings, Reciprocal Rank Fusion and a cross-encoder reranker took this RAG setup to 91% retrieval accuracy.

Publisher: REACTAPP.TOP6 min read1365 words

This article is published in English.

Sources: `I Stopped Using Vector Databases. My RAG Got Better.`

Early RAG advice is almost unanimous: pick Pinecone, Weaviate or Chroma, embed the documents, store the vectors, run semantic search and call it done. A lot of teams follow that recipe exactly. They chunk documents, create embeddings with a hosted model, load a vector database and write a retrieval function that grabs the five most similar chunks for each query. It works, up to a point, and the point is where precision starts to matter.

Where pure vector search breaks

Simple, open-ended questions get decent retrieval. Anything that needs exactness goes badly: specific dates, exact figures, references to a particular section, keyword-heavy technical queries. The LLM then confidently builds an answer from the wrong chunks. The model isn't at fault, because retrieval handed it the wrong context.

The usual reaction is to blame generation. People tweak prompts, add "only answer from the provided context" instructions and build self-reflection loops. None of that helps when the retrieved material is wrong. The weak link is the search, not the generation.

Embeddings handle meaning well but stumble on specifics, and a concrete case shows how. Say a report contains a line saying fourth-quarter 2024 revenue reached 12.8 million dollars, up 23% on the third quarter. A user asks what the Q4 2024 revenue was. It looks like an easy job for embeddings.

Here's what really happens. The embedding model turns the query and every chunk into high-dimensional vectors and then looks for conceptual closeness to "revenue", "financial performance" and "quarterly results". It can return chunks about revenue in general, forward-looking projections or Q3 figures, since all of them sit near the idea of quarterly revenue. The chunk holding "$12.8 million" may land fourth or fifth, or vanish entirely if chunk boundaries cut the sentence in an awkward place. A plain keyword search for "Q4 2024 revenue" would have found it immediately.

Scale makes it worse

With 50 documents, semantic search is fine, partly because few chunks compete. With 5,000 documents it falls apart. A larger corpus holds more chunks that are semantically close but factually irrelevant, so the signal-to-noise ratio collapses and retrieval pulls passages from "the right neighborhood" that carry the wrong information.

One test on a technical corpus of roughly 3,000 pages (API docs, changelogs and integration guides) put pure vector retrieval accuracy near 62%. That means close to four queries in ten reached the LLM with incomplete or wrong context. No prompt engineering repairs that, since the foundation itself is broken.

Dropping vector search wasn't the answer. Relying on it alone was the problem. A three-part stack lifted accuracy in that test from 62% to 91%.

Part one: BM25 for exact terms

BM25 comes from the 1990s and still ranks among the strongest retrieval algorithms around. Where embeddings blur things, it matches literal terms: for "Q4 2024 revenue" it pulls each chunk containing those words and orders them by term frequency and relevance within the corpus. It's quick, gives the same output every time and runs without a GPU. A small library covers it, and no vector database is involved, only a text index built on the chunks.

Part two: keep embeddings, but as one voice

Embeddings keep their value because they pick up on intent and surface related content. A user asking "How do I handle authentication?" against docs titled "Managing user sign-in" gets a match from semantic search that BM25 would never make. So embeddings become one source of candidates, never the only source.

The top results from both retrievers get merged with Reciprocal Rank Fusion (RRF), a straightforward method that combines ranked lists by rewarding chunks that show up in both. A chunk ranked first by BM25 and third by semantic search scores far higher than one appearing in a single list. That naturally pushes up chunks that are relevant by keyword and by meaning. The implementation is around 20 lines of Python, with no framework required.

Part three: rerank

The reranking stage is the one most teams leave out, and it moved the numbers more than anything else. Once the hybrid merge yields 15 to 20 candidates, a cross-encoder reranker scores each. An embedding model encodes the query and a document in isolation from each other. A cross-encoder instead feeds the query and one candidate through the model jointly, so the score reflects how well that pair fits.

A small reranking model adds roughly 200 ms of latency and sharply improves which chunks reach the final context. The top five after reranking are almost always the right ones. A fishing metaphor fits: BM25 plus semantic search casts a wide net, and reranking sorts the catch and keeps only the best.

Before and after

The figures reported for the hybrid pipeline against pure vector search:

Metric Vector search only Hybrid with reranking
Top-5 retrieval accuracy 62% 91%
Answer correctness 58% 87%
Hallucination rate 31% 8%
Wrong-answer reports from users about 3 per day about 2 per week

An unexpected side effect was cost. With the hosted vector database cancelled, infrastructure spending fell about 60%. BM25 runs on one CPU, and the reranker needs only a small GPU or a CPU with tolerable latency.

Why the advice skews toward vector databases

Vector database companies have raised huge sums, and their marketing shows up in every tutorial, course and video that opens with "Step 1: set up your vector database." Nobody selling one profits from admitting that a decades-old algorithm plus a small reranker beats the product in most scenarios.

Vector databases aren't useless, though. They're valuable for:

  • Exploratory search, where users browse rather than look up facts.
  • Multi-modal retrieval across images, audio or code.
  • Massive collections, in the billions of documents, where BM25 indexes become unwieldy.

For most RAG systems, such as internal knowledge bases, documentation search, support bots and compliance tools, hybrid retrieval without a dedicated vector store performs better and costs less.

A practical migration path

A full teardown isn't required. A gradual route works:

  1. Add BM25 next to the existing vector search. A Python library such as rank-bm25 can index the very same chunks, which takes about an hour.
  2. Merge results with Reciprocal Rank Fusion. The scoring formula is short and well documented.
  3. Add an open-source cross-encoder reranker, run locally or through an API. Pass the merged candidates through it and keep the top five.
  4. Measure. Build a small evaluation set, say 50 questions with known answers, and compare retrieval accuracy before and after. The gain shows up quickly.
  5. Then ask whether the vector database is still needed. For many systems, it isn't.

The bigger pattern

Hype rarely points to the best option. The better choice tends to be dull, simple and well understood. BM25 is dull, hybrid search is straightforward, and information retrieval researchers have studied reranking for decades. A story about reusing a 30-year-old algorithm won't attract venture funding or social media attention. Dependable AI still comes from picking the right tool, even when it's unfashionable.

When a RAG pipeline underperforms, the temptation is to blame the LLM, rewrite prompts or fine-tune the embedding model. Retrieval is the better place to look, because that's where the problem usually sits. The fix might be an older, simpler search method paired with a modern reranker to close the gap. Sometimes progress means looking backward first.