BM25 Plus pgvector With Reciprocal Rank Fusion: Hybrid Search for RAG in Python
Hybrid search for RAG in Python: why dense retrieval misses exact terms, how BM25 and pgvector cosine search complement each other, fused with RRF on Postgres.
This article is published in English.
Sources: Why BM25 + pgvector Hybrid Search Beats Naive RAG — A Deep Technical Breakdown
Dense vectors alone miss exact keywords, rare terms and domain jargon. Combining a lexical ranker with a vector index patches that hole, and the whole thing fits into a Postgres-centered Python stack. The math, the architecture and the code follow.
What goes wrong with plain RAG
A basic retrieval-augmented pipeline is short:
- Embed the query with a model such as
text-embedding-3-small. - Run a cosine search against a vector store like pgvector, Pinecone or Weaviate.
- Take the top-K chunks.
- Place them in the LLM prompt.
This works until a query hinges on a literal string. Suppose a user asks how a project uses BM25 for ranking. The query embedding lands near vectors about ranking, search algorithms and relevance scoring, so a document that contains the literal token BM25 can lose to a vaguer page about retrieval models. That is the vocabulary mismatch problem, which information retrieval researchers have known since the 1990s.
Embeddings squeeze meaning into a continuous space, and in the process they blur exact matches. Closeness between "myocardial infarction" and "heart attack" is welcome. Closeness between BM25 and Okapi ranking is a handicap when the query is specific.
BM25 in a nutshell
BM25, short for Best Match 25, is a probabilistic term-weighting function from the Okapi family of retrieval models. For a document d and query q, it sums over the query terms:
score(q, d) = Σ IDF(tᵢ) · [f(tᵢ, d) · (k₁ + 1)] / [f(tᵢ, d) + k₁ · (1 - b + b · |d| / avgdl)]
The symbols mean the following:
tᵢis one term of the query.f(tᵢ, d)counts how often that term occurs in the document.|d|is the document length in tokens andavgdlis the average length across the corpus.k₁, typically between 1.2 and 2.0, sets how quickly term frequency saturates, so a higher value gives frequency more weight.b, between 0 and 1, controls length normalization, with 0 meaning none and 1 meaning full.IDF(tᵢ)is the log of the ratio between documents lacking the term and documents holding it (each shifted by half a document), plus one. Here N counts all documents and n(tᵢ) counts those containing the term.
Three intuitions explain why it behaves well. Term frequency saturates, so mentioning BM25 ten times doesn't earn ten times the score of one mention, and k₁ shapes that curve. Length normalization stops short documents from being punished for a low raw count, with b scaling by average length. And inverse document frequency favors rare terms: "BM25" in a corpus of general machine learning text has a high weight, while "the" has almost none.
All of that makes BM25 excellent for exact-match, rare-term and keyword-heavy queries, which is the territory where dense embeddings stumble.
pgvector and dense retrieval
pgvector is a PostgreSQL extension that stores embeddings and offers approximate nearest neighbor search through HNSW graphs or IVFFlat indexes. Similarity between a query vector and a document vector is the cosine:
cosine_sim(q, dᵢ) = (q · dᵢ) / (‖q‖ · ‖dᵢ‖)
HNSW queries take roughly logarithmic time, with memory proportional to N times M, where M is the number of bidirectional links kept per node. That delivers retrieval in well under a millisecond across millions of vectors. Dense search shines at semantic matching, such as paraphrases, language variation and concept-level similarity. It under-indexes exact strings for the reason described above.
Merging the two with Reciprocal Rank Fusion
Two ranked lists exist, one from BM25 and one from pgvector. Reciprocal Rank Fusion combines them by giving each document a score based on its position in every list:
RRF(d) = Σᵢ 1 / (k + rankᵢ(d))
The constant k is 60. It softens the effect of very high ranks, and it has held up across many tasks. The sum covers both retrievers.
Why prefer RRF to a learned combiner? No labeled relevance data is needed. It works on ranks and not on scores, so it dodges the mismatch between BM25 values, which are unbounded, and cosine values, which sit between -1 and 1. And Cormack and colleagues showed in 2009 that it matches or beats a weighted linear combination with no tuning. A naive blend such as α · bm25_score + β · cosine_score is shakier, because the raw numbers live on incompatible scales.
Evidence from benchmarks
On the BEIR benchmark, a mixed suite covering biomedical, news and technical corpora, combining BM25 with dense retrieval tends to beat either approach alone. TREC-COVID is a telling case: BM25 outperforms the dense model there, since biomedical queries such as "SARS-CoV-2 spike protein ACE2 binding" are packed with highly specific terms that embeddings blur. The hybrid keeps the strengths of both.
Python implementation
Setup
Everything lives in one Postgres schema. Install the dependencies first:
# requirements: psycopg2, pgvector, rank-bm25, openai, numpy
Then set up the connection, the vector type and the libraries:
import psycopg2
from pgvector.psycopg2 import register_vector
from rank_bm25 import BM25Okapi
import openai
import numpy as np
from typing import List, Tuple# ── DB setup ──────────────────────────────────────────────────────────────
conn = psycopg2.connect("postgresql://user:pass@localhost/ragdb")
register_vector(conn)with conn.cursor() as cur:
cur.execute("CREATE EXTENSION IF NOT EXISTS vector;")
cur.execute("""
CREATE TABLE IF NOT EXISTS documents (
id SERIAL PRIMARY KEY,
content TEXT NOT NULL,
embedding vector(1536) -- text-embedding-3-small dim
);
""")
# HNSW index: m=16 (edges/node), ef_construction=64 (build-time beam width)
cur.execute("""
CREATE INDEX IF NOT EXISTS doc_hnsw_idx
ON documents USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
""")
conn.commit()
Indexing
A helper wraps the embedding API call:
def embed(texts: List[str]) -> List[List[float]]:
resp = openai.embeddings.create(
model="text-embedding-3-small",
input=texts
)
return [r.embedding for r in resp.data]
And the indexing function stores each document with its embedding:
def index_documents(docs: List[str]) -> None:
embeddings = embed(docs)
with conn.cursor() as cur:
for doc, emb in zip(docs, embeddings):
cur.execute(
"INSERT INTO documents (content, embedding) VALUES (%s, %s)",
(doc, emb)
)
conn.commit()
Hybrid retrieval
The lexical search ranks the corpus with BM25Okapi:
def bm25_search(
corpus: List[str],
query: str,
top_k: int = 20
) -> List[Tuple[int, float]]:
"""Returns [(doc_idx, bm25_score), ...] sorted descending."""
tokenized = [doc.lower().split() for doc in corpus]
bm25 = BM25Okapi(tokenized, k1=1.5, b=0.75)
scores = bm25.get_scores(query.lower().split())
ranked = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)
return ranked[:top_k]
The dense search queries pgvector and the fusion step then applies RRF:
def dense_search(
query: str,
top_k: int = 20
) -> List[Tuple[int, float]]:
"""Returns [(doc_id, cosine_sim), ...] sorted descending."""
q_emb = embed([query])[0]
with conn.cursor() as cur:
cur.execute("""
SELECT id, 1 - (embedding <=> %s::vector) AS cosine_sim
FROM documents
ORDER BY embedding <=> %s::vector
LIMIT %s;
""", (q_emb, q_emb, top_k))
return [(row[0], row[1]) for row in cur.fetchall()]
def reciprocal_rank_fusion(
*ranked_lists: List[Tuple[int, float]],
k: int = 60
) -> List[Tuple[int, float]]:
"""
RRF(d) = Σᵢ 1 / (k + rankᵢ(d))
k=60 is the standard constant from Cormack et al. 2009.
"""
scores: dict[int, float] = {}
for ranked in ranked_lists:
for rank, (doc_id, _) in enumerate(ranked, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)
def hybrid_retrieve(
corpus: List[str],
doc_ids: List[int], # DB IDs aligned with corpus
query: str,
top_k: int = 5
) -> List[dict]:
"""
Full hybrid pipeline:
1. BM25 sparse search
2. pgvector dense ANN search
3. RRF fusion
4. Return top-K merged results
"""
bm25_results = bm25_search(corpus, query, top_k=20)
# Map corpus index → DB id for BM25
bm25_db = [(doc_ids[idx], score) for idx, score in bm25_results] dense_results = dense_search(query, top_k=20) fused = reciprocal_rank_fusion(bm25_db, dense_results, k=60) # Fetch content for top-K
top_ids = [doc_id for doc_id, _ in fused[:top_k]]
with conn.cursor() as cur:
cur.execute(
"SELECT id, content FROM documents WHERE id = ANY(%s)",
(top_ids,)
)
rows = {row[0]: row[1] for row in cur.fetchall()} return [
{"id": doc_id, "rrf_score": score, "content": rows.get(doc_id, "")}
for doc_id, score in fused[:top_k]
]
Answer generation
The final stage fetches the top results and builds the context:
def rag_answer(query: str, corpus: List[str], doc_ids: List[int]) -> str:
results = hybrid_retrieve(corpus, doc_ids, query, top_k=5)
context = "\n\n---\n\n".join(r["content"] for r in results)
A restrictive prompt keeps the model grounded in what was retrieved:
prompt = f"""You are a precise technical assistant.
Use only the provided context to answer the question.
If the context doesn't contain the answer, say so.Context:
{context}Question: {query}
Answer:""" resp = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0
)
return resp.choices[0].message.content
Parameters worth tuning
BM25 k₁ defaults to 1.5, where a bigger value gives repeated terms more weight, which suits verbose queries. BM25 b defaults to 0.75, and raising it strengthens the length penalty. For HNSW, m defaults to 16, since more edges mean better recall at the cost of memory, and ef_construction defaults to 64, where a larger value improves index quality but slows the build. The RRF k of 60 can go lower if you want the top-ranked result to dominate more. Fetching 20 candidates per retriever is a sound default, and more candidates generally improves fusion.
With domain-heavy corpora such as medical, legal or code, lower b to 0.5 to ease the penalty on long technical documents, and raise k₁ to 2.0 so repeated technical terms are rewarded.
Picking the right retriever
Exact keywords, product codes and API names favor BM25. Conceptual, paraphrased or multilingual queries favor dense search. Most real queries are a mix of both, which makes hybrid with RRF the best choice, and domain jargon in medicine, law or code works best with BM25 or the hybrid.
The takeaway
Naive RAG makes a strong baseline but leaves recall on the table for keyword-heavy and specialist queries. The repair is simple in architecture: run BM25 over the same corpus next to the pgvector search, merge the two ranked lists with RRF at k=60, and send the merged top-K to the LLM. The resulting retriever copes with a semantic question like "what is machine learning" and a lexical one like "show me the BM25Okapi k1 parameter handling" equally well, with no extra model to train and only a microsecond-scale extra pass for BM25.
pgvector ships inside Postgres, BM25 arrives through a rank-bm25 install, and RRF takes a handful of lines. Little stands in the way of adding hybrid search to a production RAG system.