Hybrid RAG Retrieval with pgvector, BM25 and a Cross-Encoder Reranker

A retrieval-augmented generation (RAG) prototype built on plain vector search usually impresses in demos and then disappoints real users. Ask it about the maintenance schedule for a specific pump model and it returns general pump advice; search for an exact error code and the relevant troubleshooting step never shows up. This guide explains why dense retrieval fails on exact identifiers and shows how to fix it with a hybrid pipeline: pgvector for semantic search, BM25 for keyword matching, and a reranker to decide which chunks actually reach the LLM.

Why vector search misses exact identifiers

Embeddings are excellent at meaning. A query for "automobile" lands near documents about "cars" and "vehicles" because the embedding model places related concepts close together in a high-dimensional space. That same property is the weakness. Similarity is about semantic closeness, not exact character sequences.

When a user searches for a part number like TX-99402 or an error code like E-404, the embedding of that string may sit very close to TX-99401 or to generic troubleshooting text. The retriever returns documents that are "about the same thing" rather than the one that contains the exact token the user typed. For technical manuals, product catalogues and support knowledge bases, where identifiers carry most of the meaning, this is the dominant failure mode.

Hybrid retrieval: dense and sparse in parallel

The fix is to run two complementary retrievers and combine what they find:

1. Dense retrieval (vector search) captures context and meaning. Rather than operating a separate vector database, you can store embeddings in PostgreSQL with the pgvector extension. Many applications already run on Postgres, so adding a vector column keeps the stack small and gives you familiar backups, access control and transactions.
2. Sparse retrieval (keyword search) captures exact matches, acronyms and domain jargon. The standard algorithm is BM25, a long-established ranking function that scores documents by how often query terms appear, weighted by how rare those terms are across the corpus and normalised for document length.

A useful mental model: vector search finds the right neighbourhood, and BM25 finds the exact house number. You take the top results from each and merge them. For more on where each approach wins, see our article on hybrid search for technical knowledge.

Why the merged results need a reranker

Hybrid search raises an immediate problem. You now have two ranked lists whose scores are not comparable. A BM25 score depends on term frequencies and is unbounded, while vector similarity comes from cosine distance on a completely different scale. A semantic score of 0.82 is not better or worse than a BM25 score of 14.5; sorting the union by raw score is meaningless.

A reranker resolves this by ignoring the original scores. It is a separate model, commonly a cross-encoder, that reads the query and a candidate document together and outputs a single relevance score for that pair. Because it sees both texts at once, it can judge relevance far more precisely than comparing two independently computed embeddings.

The flow is:

1. Retrieve 10 candidates from each retriever, pgvector and BM25.
2. Pool them, giving up to 20 chunks.
3. Score every chunk against the query with the reranker.
4. Keep the best 3 and pass only those to the LLM.

Fewer, better chunks also means a shorter prompt, less noise for the model to ignore, and lower token cost.

The latency cost

Cross-encoders are expensive. Scoring 20 documents adds noticeable time to every request, and in a streaming chat API (for example one built with FastAPI) it delays the first token. Measure this step separately in your latency tracking. The accuracy gain is usually worth it, but tune the candidate count to your budget; our piece on why reranking must earn its latency goes deeper into that trade-off.

Implementing the pipeline with LangChain

LangChain provides building blocks for each piece, so the whole pipeline fits in two short Python functions. The examples below are conceptual; adapt the connection string, models and file paths to your environment.

Ingestion: chunk, embed and index twice

The ingestion function loads a text file, splits it into chunks of 1,000 characters with 100 characters of overlap, and then indexes the same chunks two ways. First, it embeds them with OpenAI's text-embedding-3-small model and stores them in a pgvector collection via PGVector.from_documents. Second, it fits a BM25Retriever on the chunks and serialises it to disk with pickle, because BM25 builds its index in memory.

import pickle
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_postgres.vectorstores import PGVector
from langchain_community.retrievers import BM25Retriever

CONNECTION_STRING = "postgresql+psycopg://user:password@localhost:5432/mydb"
COLLECTION_NAME = "hybrid_docs"

def ingest_documents(file_path: str):
    # 1. Load and chunk the document
    loader = TextLoader(file_path)
    docs = loader.load()

    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
    chunks = text_splitter.split_documents(docs)

    # 2. Store dense embeddings in pgvector
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    PGVector.from_documents(
        embedding=embeddings,
        documents=chunks,
        collection_name=COLLECTION_NAME,
        connection=CONNECTION_STRING,
    )

    # 3. Fit and save the BM25 sparse retriever
    bm25_retriever = BM25Retriever.from_documents(chunks)
    with open("bm25_retriever.pkl", "wb") as f:
        pickle.dump(bm25_retriever, f)

    print(f"Successfully ingested {len(chunks)} chunks.")

# Example usage:
# ingest_documents("technical_manual.txt")

Things to keep in mind:

• The BM25 index is a snapshot. When documents change, you must refit and re-save it, or it drifts out of sync with the vector store.
• Only unpickle files you created yourself. Loading a pickle executes code, so a tampered file is a security risk.
• The connection string contains credentials; load it from configuration rather than hard-coding it.
• If you would rather keep keyword search inside the database too, PostgreSQL's built-in full-text search is an alternative to an in-process BM25 index, with different ranking behaviour.

Retrieval: ensemble, then rerank

The retrieval function rebuilds both retrievers and chains them. The pgvector retriever returns the top 10 semantic matches (k=10), and the unpickled BM25 retriever is also set to return 10. An EnsembleRetriever merges them with equal weights of 0.5 each. A CohereRerank compressor with top_n=3 wraps the ensemble inside a ContextualCompressionRetriever, so every query goes through retrieval, merging and reranking in one invoke call.

import pickle
from langchain_openai import OpenAIEmbeddings
from langchain_postgres.vectorstores import PGVector
from langchain.retrievers import EnsembleRetriever, ContextualCompressionRetriever
from langchain_cohere import CohereRerank

CONNECTION_STRING = "postgresql+psycopg://user:password@localhost:5432/mydb"
COLLECTION_NAME = "hybrid_docs"

def setup_hybrid_retriever():
    # 1. Initialize Vector Retriever
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = PGVector(
        connection=CONNECTION_STRING,
        embeddings=embeddings,
        collection_name=COLLECTION_NAME,
    )
    # Fetch top 10 semantic matches
    pgvector_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

    # 2. Load Keyword Retriever (BM25)
    with open("bm25_retriever.pkl", "rb") as f:
        bm25_retriever = pickle.load(f)
    # Fetch top 10 exact keyword matches
    bm25_retriever.k = 10

    # 3. Merge pools with EnsembleRetriever (50/50 weighting)
    hybrid_retriever = EnsembleRetriever(
        retrievers=[bm25_retriever, pgvector_retriever],
        weights=[0.5, 0.5]
    )

    # 4. Rerank the combined 20 chunks to output the absolute top 3
    reranker = CohereRerank(cohere_api_key="YOUR_COHERE_API_KEY", top_n=3)
    advanced_retriever = ContextualCompressionRetriever(
        base_compressor=reranker,
        base_retriever=hybrid_retriever
    )

    return advanced_retriever

def query_system(query: str):
    retriever = setup_hybrid_retriever()
    best_docs = retriever.invoke(query)

    for i, doc in enumerate(best_docs):
        print(f"\n--- Result {i+1} ---")
        print(doc.page_content)

# Example usage:
# query_system("What is the warranty period for the TX-99402 sensor?")

Some details that are easy to miss:

• EnsembleRetriever does not add raw scores. It fuses the lists by rank using weighted Reciprocal Rank Fusion, which sidesteps the scale mismatch described above. It also removes duplicates, so the reranker may receive fewer than 20 chunks when both retrievers find the same one.
• Never ship an API key in source code. Read the Cohere key from an environment variable or a secrets manager.
• setup_hybrid_retriever() runs on every query here, reconnecting to Postgres and unpickling BM25 each time. In a real service, build the retriever once at startup and reuse it.
• LangChain has reorganised its packages across releases, and classes such as EnsembleRetriever and ContextualCompressionRetriever may live in a different package in your version. Check the current LangChain docs if an import fails.

Key takeaways

• Pure vector search is weak on exact tokens such as part numbers, SKUs and error codes; BM25 covers that gap.
• pgvector lets you add dense retrieval to an existing PostgreSQL stack without a separate vector database.
• Scores from different retrievers are not comparable, so merge by rank and let a cross-encoder reranker make the final ordering.
• Reranking improves precision but adds latency; size the candidate pool deliberately and monitor it.
• Treat the BM25 index as a build artifact that must be refreshed with the data, and keep credentials out of code.

Hybrid retrieval does not guarantee perfect answers, but it removes the most common reason production RAG systems return plausible yet wrong context.