Beyond Top-K Similarity: BM25, Hybrid Search, Reranking and MMR in RAG

The first RAG system most developers build follows one recipe: embed the documents, put them in a vector database, and when a question arrives, grab the closest chunks and hand them to the LLM. It works, until it doesn't. Users begin to complain that obvious answers get missed. The document plainly contains the information, yet retrieval doesn't surface it, or the top hit resembles the question on the surface without being what the user needed. Plain similarity search is a fine starting point with real limits, and once those are clear, better retrieval strategies fix them. Here are the reasons it fails, what hybrid search and reranking do about it, and how to assemble the pieces into one pipeline.

Similarity search, briefly

In a basic setup, the user's question becomes an embedding, a list of numbers standing for its meaning. Every chunk in the vector store has one too. The system finds the chunks whose embeddings sit nearest to the question's, measured by cosine similarity or dot product, and returns the top K.

# Basic similarity search — what most RAG systems do
results = vector_store.similarity_search(query, k=3)

For general semantic matching that does well. A question about how photosynthesis works will pull up chunks about plants turning sunlight into energy even if the wording differs. But it has blind spots.

Where it falls short

Exact keywords get missed. Semantic search captures meaning rather than literal words, which is usually a benefit. When someone searches for a specific term, though, such as a product name, an acronym or a person, the embedding may not match closely enough. Say the user asks about BERT model architecture. A chunk stating that BERT uses bidirectional transformers exists, yet a broader chunk on language model architectures may outrank it because it's semantically wider.

A high score isn't the same as the best answer. Similarity measures how related two texts are, not how useful one is for answering the other. A chunk saying RAG systems are handy for question answering is very close to the query "what is RAG?", while the chunk holding the actual definition is more useful.

Top results can also be redundant. Real documents repeat themselves across chunks, so the top three can all say much the same thing, wasting context window on duplicate material.

BM25 for keyword retrieval

BM25 is a classic information-retrieval algorithm of the kind behind traditional search engines. It matches documents on exact keywords, weighting each by how rare it is across the whole corpus, and it uses no embeddings, only raw text. A search for "FAISS index" returns chunks that literally contain both words, and the rare one counts for more than the common one.

from langchain_community.retrievers import BM25Retriever
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Load and split documents
loader = TextLoader("knowledge_base.txt")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)
# Create BM25 retriever
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 3
# Search
results = bm25_retriever.invoke("FAISS vector index")
for doc in results:
    print(doc.page_content[:200])

It shines with specific product names, people or technical terms, with short precise queries, and wherever exact wording matters more than meaning. It struggles when the user phrases things differently from the document (synonyms, paraphrases), with long conversational queries, and when meaning outweighs literal words.

Hybrid search

Hybrid search runs semantic retrieval and BM25 side by side. Semantic search handles meaning and synonyms, and BM25 handles exact keywords, so together they cover cases either would miss. The two result lists are merged with Reciprocal Rank Fusion (RRF), a technique that combines ranked lists into one without needing their scores to share a scale.

from langchain_community.retrievers import BM25Retriever
from langchain_google_genai import GoogleGenerativeAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.retrievers import EnsembleRetriever
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader
#Load and split
loader = TextLoader("knowledge_base.txt")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)
#Retriever 1 — Semantic (FAISS)
embeddings = GoogleGenerativeAIEmbeddings(
    model="models/embedding-001",
    google_api_key="YOUR_KEY"
)
vector_store = FAISS.from_documents(chunks, embeddings)
semantic_retriever = vector_store.as_retriever(search_kwargs={"k": 5})
#Retriever 2 — Keyword (BM25)
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 5
#Combine both — Hybrid Search
hybrid_retriever = EnsembleRetriever(
    retrievers=[semantic_retriever, bm25_retriever],
    weights=[0.6, 0.4]  # 60% semantic, 40% keyword
)
#Search
results = hybrid_retriever.invoke("What is FAISS and how does it store vectors?")
print(f"Retrieved {len(results)} chunks")
for i, doc in enumerate(results):
    print(f"\nChunk {i+1}: {doc.page_content[:150]}")

A weights parameter sets how much each retriever contributes. Leaning slightly toward semantic search suits most cases, though the right balance depends on your data. Choose hybrid when documents contain technical terms, product names or proper nouns, when users mix conversational and keyword-style questions, or when you want keyword matching as a safety net behind semantic understanding.

Reranking

Reranking works in two stages. First retrieve broadly, fetching the top 20 or 30 chunks with the retriever, which is cheap and fast. Then pass those candidates through a dedicated reranker model that scores each one on how well it answers the question, and keep only the best 3 to 5.

The reasoning is that vector similarity is a quick approximation of relevance, and a reranker is slower but much more accurate. Narrowing with similarity first and then reranking gives you both speed and precision.

from langchain_community.cross_encoders import HuggingFaceCrossEncoder
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain.retrievers import ContextualCompressionRetriever
#Base retriever — fetch top 20 candidates
base_retriever = vector_store.as_retriever(search_kwargs={"k": 20})
#Reranker model
reranker_model = HuggingFaceCrossEncoder(
    model_name="BAAI/bge-reranker-base"
)
#Reranker compressor — keeps only top 3 after reranking
reranker = CrossEncoderReranker(model=reranker_model, top_n=3)
#Combine base retriever + reranker
reranking_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=base_retriever
)
#Search — fetches 20, reranks, returns top 3
results = reranking_retriever.invoke("How does FAISS perform similarity search?")
print(f"Final chunks after reranking: {len(results)}")
for i, doc in enumerate(results):
    print(f"\nTop {i+1}: {doc.page_content[:200]}")

A cross-encoder reranker reads the question and a chunk together as a pair and asks how relevant that specific chunk is to that specific question. Vector similarity looks at each text independently and never weighs their relationship directly. Reach for reranking when precision matters and wrong answers are costly, when you're doing question answering over specific documents, or whenever answer quality directly affects users.

Maximum Marginal Relevance

MMR tackles redundancy. Rather than simply taking the closest chunks, it picks chunks that are both relevant and varied, so the same information isn't returned several times.

# Standard similarity search — might return redundant chunks
standard_results = vector_store.similarity_search(query, k=3)
# MMR search — returns relevant AND diverse chunks
mmr_results = vector_store.max_marginal_relevance_search(
    query,
    k=3,           # final number of chunks to return
    fetch_k=20,    # candidates to consider before selecting diverse ones
    lambda_mult=0.5  # 0 = max diversity, 1 = max relevance
)

The lambda_mult parameter sets the trade-off. A value of 1.0 means pure relevance, identical to ordinary similarity search. A value of 0.0 means pure diversity, with chunks as different from each other as possible. A value of 0.5 balances the two and is usually a good first setting. MMR pays off when documents are repetitive, when you want several aspects of a topic covered, or when the context window is too tight for duplicate chunks.

A full pipeline

Putting it together, hybrid retrieval casts a wide net, reranking picks the most relevant chunks, and a complete RAG chain generates the answer.

from langchain_google_genai import ChatGoogleGenerativeAI, GoogleGenerativeAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_community.retrievers import BM25Retriever
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
from langchain.retrievers import EnsembleRetriever, ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader
# Setup
llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", google_api_key="YOUR_KEY")
embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001", google_api_key="YOUR_KEY")
# Load and split documents
loader = TextLoader("your_knowledge_base.txt")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)
# Build retrievers
vector_store = FAISS.from_documents(chunks, embeddings)
semantic_retriever = vector_store.as_retriever(search_kwargs={"k": 10})
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 10
# Hybrid retriever
hybrid_retriever = EnsembleRetriever(
    retrievers=[semantic_retriever, bm25_retriever],
    weights=[0.6, 0.4]
)
# Add reranking on top
reranker = CrossEncoderReranker(
    model=HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base"),
    top_n=3
)
final_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=hybrid_retriever
)
# RAG prompt
prompt = ChatPromptTemplate.from_messages([
    ("system", """Answer the question using only the context below.
    If the answer is not in the context, say "I don't have that information."

    Context: {context}"""),
    ("human", "{question}")
])
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)
# Complete chain
rag_chain = (
    {"context": final_retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)
# Use it
answer = rag_chain.invoke("Your question here")
print(answer)

Retrieval quality ends up clearly better than basic similarity search alone.

Choosing a strategy

A short decision guide maps situations to strategies:

Just starting out / simple use case?
    → Basic similarity search is fine
Documents have specific technical terms or product names?
    → Add BM25 → use Hybrid Search
Answer quality is critical, wrong answers are costly?
    → Add Reranking on top of any retriever
Documents have lots of repeated content?
    → Use MMR instead of standard similarity search
Production system with high quality requirements?
    → Hybrid Search + Reranking together

Don't over-engineer from day one. Begin with basic similarity search, add hybrid search once you see keyword misses, and add reranking when answer quality needs to rise. Every addition costs something in complexity, latency and compute. For production RAG, hybrid search plus reranking is the standard to aim for, and complexity should come only when a specific problem calls for it.

One practical note on reranking: it needs pip install sentence-transformers, and the BAAI/bge-reranker-base model downloads automatically on first use at roughly 1GB. For production, hosted reranking APIs from Cohere or Jina AI avoid the overhead of running the model locally.