Hybrid Search for RAG: BM25, Vectors and Reciprocal Rank Fusion in Practice

Adding keyword search to a vector-only retriever is one of the cheapest accuracy gains in RAG, and Anthropic measured it on its own corpora: contextual BM25 on top of contextual embeddings cut failed top-20 retrievals by up to 49 percent, with no change to the embedding model.

Consider the failure behind that number. A user types a vehicle identifier such as GX-V5-5N. A pure vector system returns generic registration instructions and service histories, and the correct document is missing from even the first ten results. The embedding grasps the automotive context perfectly, and that is exactly why it drops the literal string. This isn't about weak embeddings. It is structural, and it affects any RAG system built on vectors alone, which describes what most default tutorials produce.

Below: why combining lexical and vector search needs different math from averaging scores, what Reciprocal Rank Fusion does, where the hybrid approach stops helping, and what to measure before turning it on.

The idea in two minutes

Pure vector search systematically loses exact matches such as error codes, product identifiers and proper names. The embedding is not bad. Semantic closeness and literal matching are simply different properties.

BM25, a lexical ranking function from the 1990s, supplies what vectors lack. It catches rare, specific tokens almost for free and misses paraphrases and synonyms in exchange. The two methods fail in opposite directions, which is the structural reason to combine them, and not a passing engineering fashion.

The complication is that their outputs can't be added. BM25 scores are unbounded and depend on the corpus, while cosine similarity lives between negative one and one. A sum of 15.3 and 0.82 means nothing. So modern engines don't merge scores. They merge rank positions with Reciprocal Rank Fusion.

Hybrid does not always win, though. On queries built around exact identifiers, BM25 alone can beat the whole fusion. The useful question is not whether to add hybrid but what share of your queries actually need it, and that is something you measure.

Two complementary blind spots

Imagine two junior colleagues with opposite weaknesses.

The first is as literal as Ctrl+F. Asked for the page on error code ERR_CONNECTION_RESET, he finds it instantly. Asked about connection problems, he overlooks the document on session loss, because none of the query's words appear in it.

The second reads between the lines. He knows connection problems and session loss describe the same thing. Asked for a precise error code, though, he hands back a generic note on network failures, since the rare string was diluted in the averaged meaning of the whole sentence.

Neither is enough alone. That is the whole argument in two paragraphs.

What BM25 counts

BM25 is neither magic nor obsolete. It runs on an inverted index, a table from each word to the documents containing it, so the engine never scans the whole collection. Ranking rests on three intuitions:

• Rare terms weigh more. The identifier GX-V5-5N says far more than the word vehicle.
• Repetition saturates. A tenth occurrence of a word adds much less than the second, because the curve flattens after a few hits.
• Length is penalized. Long documents shouldn't win merely because they have more room for chance matches.

A vector model works on another principle. Text becomes a list of numbers that encodes meaning, and ranking is closeness in that space, whether or not query and document share a word. One engine measures character coverage and the other measures meaning coverage.

A category error worth avoiding

It's tempting to see vector search as an upgraded full-text search, newer and smarter and the natural destination. That is a category error. Vectors do not replace lexical search. They solve a different problem and, in doing so, lose a skill lexical search had for free.

If a corpus contains codes, identifiers or proper names, an embedding alone will lose them however good the model is. The limit comes from what a vector represents. The right framing is which queries need which strength, which leaves the problem of merging two result lists measured in unrelated units.

Why scores can't simply be added

The natural engineering instinct is to normalize, weight and add. That runs into a fact tuning cannot remove.

BM25 has no theoretical ceiling. Its output grows with the number of matched query terms, their rarity across the corpus and the length of the document, so one and the same document receives a different value after ten thousand new files arrive. Cosine similarity, which compares the angle between two vectors, stays inside the range from negative one to one.

Adding 15.3 to 0.82 is like adding kilograms to pounds without converting: arithmetically possible, informationally empty. You can use min-max or z-score normalization, but then both scoring systems need recalibrating whenever the corpus, embedding model or query mix changes. Calibration that has to be redone after each reindex is deferred debt.

Voting on position

Reciprocal Rank Fusion avoids the problem by ignoring scores. It looks only at the rank each engine gave a document. The formula is short: a document's fused score is the sum, over every engine, of one divided by (k plus its rank in that engine's list). A document missing from a list contributes nothing.

So a document in the top three of both lists beats one that is first in one list and absent from the other. RRF rewards agreement in ordering and not certainty, and rank position is a unit every ranker produces in the same way.

The method comes from Cormack, Clarke and Büttcher at SIGIR 2009, long before RAG. Their reasoning: top ranks should count most, lower ranks shouldn't vanish entirely, and the constant k softens the influence of any single outlying system.

A footnote that comes up in design reviews: the standard k of 60 came out of a single pilot run and was not revised in later validation. The authors themselves described it as an uncritical choice. At k=60 rank one scores about 1.6 percent above rank two, while rank sixty gets about half of rank one's score.

In practice k is a gentle dial and not a lever. What matters more is how many candidates each leg contributes. If the vector side supplies fifty candidates, a document at position fifty-one contributes exactly zero.

How hybrid became the default

The clearest figures come from Anthropic's 2024 contextual retrieval experiments on its own datasets, tracking how often the right chunk was missing from the top twenty. Contextual embeddings by themselves reduced misses by 35 percent. Layering contextual BM25 on top took the reduction to 49 percent, meaning the lexical leg delivered a large fraction of what the embedding upgrade did.

Resist one misreading. The same write-up reports a 67 percent reduction once reranking is added, but reranking is a separate stage using a different class of model. Fusion and reranking aren't synonyms, and mixing them on one slide is a common presentation mistake.

Vendor convergence says more than any single benchmark. Elasticsearch, OpenSearch, Weaviate, Qdrant and Azure AI Search all ship RRF natively. Hybrid search has moved from an advanced option to the factory setting.

The cost of a second index

There is a price, and it should be counted before the decision and not after the first incident:

• more disk, in an amount that depends on configuration,
• a few extra milliseconds for the fusion step,
• and, most importantly, a second thing that can drift out of sync. If a document is removed from the vector index but survives in the lexical one, it keeps resurfacing like a ghost.

Teams already on Postgres can often get a lexical leg by pairing pgvector with built-in full-text search, entirely in SQL and without new infrastructure. If a current RAG system is vector-only, the better question is why it lacks BM25, which is a matter of technical debt and not innovation.

Where hybrid loses

Vendor unanimity might suggest hybrid always wins. Direct tests are more nuanced, and the awkward results appear once you break scores down by query type instead of averaging.

In Microsoft's 2023 Azure AI Search comparison, hybrid retrieval beat vector-only retrieval in aggregate. Looked at by category, the pattern is less comfortable: for purely keyword-style queries, the single lexical engine can score higher than the fused result. The average hides what a per-segment view exposes.

A second engine does not always add signal

Wherever the lexical engine is right, a second vote dilutes it. Suppose BM25 ranks the correct document first and the vector ranks it tenth. Fusion pushes it below a document placed third by both. Agreement beats accuracy, which is by design but costs something on some query types.

How good the second leg is also matters. Tests of first-stage retrieval on a scientific corpus showed a hybrid built on a weak embedding model landing under BM25 alone, whereas the same setup with a stronger model landed clearly above it. Fusion creates nothing new; it only blends existing evidence, and noise gets a vote equal to signal.

This doesn't contradict the opening point. Embedding quality decides how much the vector leg adds, but even the best embedding can't recover an exact identifier it doesn't represent. Those are two separate axes.

Neither result argues against hybrid search. Both argue against enabling it without measuring your own query distribution. Many teams pick hybrid as an obvious default and never count that distribution.

The measurement is cheap. Split query logs into queries with an exact identifier and purely conceptual ones, and compute a metric for each group separately. That takes hours and not a sprint, whereas a single aggregate number hides the regression that later hurts in production.

The consensus problem

Rank fusion has a subtler flaw that shows up when the engines agree and not when they disagree.

Take a search over technical documentation, with a query about configuring OAuth through environment variables. The vector leg returns a general security guide: relevant in spirit, but without OAuth specifics. BM25 returns a precise OAuth match, but in a changelog three years old. Neither is the best answer, and fusion still ranks the weaker of the two first.

This is a direct result of how RRF works. The outdated changelog was rated mid-level by both engines, so its two reciprocal ranks added up to more than the fresh page that only a single engine liked. Two lukewarm opinions beat one confident and correct one.

Filter before fusion

The fix lives upstream of the math. In most cases hard metadata filters, such as freshness, permissions and document status, should run before fusion and not after. Fusion shouldn't see candidates that were never allowed in play. Some production systems filter afterward and report good results, so this is an architectural decision to test on your own corpus and not a universal law.

Sequence is not cosmetic here. Filtering after fusion drops the stale page from the output, yet the fresh page that already lost the vote and fell under the cutoff stays gone. Filtering before fusion alters the candidate pool both engines vote on in the first place.

In practice, apply three filters at the entry of both legs at once: user permissions, document status and a freshness window suited to the domain. Both legs must receive exactly the same filter. Otherwise one fills its candidate list with documents the other never saw, and fusion converts that asymmetry into a ranking advantage.

One more point for morphologically rich languages. Heavy inflection breaks lexical matching. Without good lemmatization BM25 loses much of its edge, because case variants of one word, like Polish umowy and umowie or German Vertrags and Verträgen, look like entirely different tokens. This is a rare case where the vector leg rescues the lexical one and not the other way around.

Hybrid therefore repairs one family of errors while creating a new one, and nothing in a retrieval stack can be trusted as a default without checking it.

What to take from this

The 49 percent drop in failed retrievals did not come from a bigger model or a larger GPU budget. It came from recognizing that a single text representation can't serve two fundamentally different query types.

• BM25 and vectors fail in opposite directions, which is the structural reason to combine them.
• RRF fuses rank positions and not scores, because the BM25 and cosine scales are incompatible and normalization never removes the calibration cost for good.
• Hybrid search is the default among infrastructure vendors, so skipping the second leg is technical debt and not conservatism.
• Default does not mean universally better. On exact-identifier queries BM25 alone can beat fusion, and an averaged metric will hide that.
• Apply metadata filters before fusion, or agreement between the engines will promote an outdated document above a fresh one.

For next week, take the last thousand real queries, label each as containing an exact identifier or being purely conceptual, and work out the ratio. Then evaluate accuracy separately for both groups and keep two result rows and not a blended average. That table, far more than any vendor benchmark, shows whether hybrid will improve your system or just give you a second index to look after. The result also tells you how much of your traffic depends on exact strings, which is rarely something anyone has counted.