Hybrid Agent Memory: Fusing BM25 and Vector Search With RRF in Python

An agent that runs long workflows over days or weeks needs a memory it can query reliably, and the default choice, a vector store searched by embedding similarity, breaks down quickly in technical domains. Error codes and function names get lost, cross-document questions go unanswered, and prompts fill up with loosely related chunks. This guide shows how to combine keyword and dense retrieval with Reciprocal Rank Fusion (RRF), walks through a compact Python implementation and its gaps, and explains where a GraphRAG layer helps.

Why Vector Search Alone Is Not Enough

Pure dense retrieval, often called naive RAG, has three recurring weaknesses as an agent's long-term memory:

• Exact identifiers get blurred. Embeddings capture meaning but handle precise tokens such as entity IDs, model numbers, user IDs and function names poorly. A query for one error code can return chunks about a similar-looking one.
• No global view. Retrieval returns isolated chunks close to the query and cannot assemble relationships spread across a whole corpus.
• Context bloat. Pasting raw hits into the prompt burns tokens with no guarantee of relevance.

The usual production answer is layered: hybrid search runs BM25 alongside dense vectors, merges them with RRF, and adds a GraphRAG knowledge graph for broad questions. For a refresher on the vector side, see how vector databases really work, from embeddings to hybrid search.

Three Retrieval Methods and What Each Is Good At

• Dense vector search (HNSW or flat indexes) maps text into an embedding space and matches intent and conceptual similarity, even when wording differs.
• Sparse keyword search (BM25 or TF-IDF) scores exact token matches, which is indispensable for error codes, unique IDs, function signatures and proper nouns.
• GraphRAG extracts entities and relationships into a knowledge graph and suits synthesizing questions such as "How do all services in Cluster B handle database timeouts?" Such graphs are general directed graphs, not necessarily acyclic ones.

The Shape of a Hybrid Memory Query

An agent consulting its memory sends one request that parameterizes both engines:

{
  "query_id": "mem_req_99214",
  "search_text": "HTTP 504 gateway timeout postgres connection pool",
  "top_k": 5,
  "hybrid_params": {
    "dense_weight": 0.6,
    "sparse_weight": 0.4,
    "rrf_k_constant": 60
  },
  "metadata_filter": {
    "environment": "production",
    "service": "billing-service"
  }
}

Besides the query text and top_k, the payload carries weights for each engine, the RRF constant k, and a metadata filter restricting results to one environment and service. Filtering matters: a production incident should not be answered with staging notes.

The fused response returns scored chunks with their source:

{
  "query_id": "mem_req_99214",
  "total_retrieved": 2,
  "results": [
    {
      "chunk_id": "doc_8832_p4",
      "rrf_score": 0.0322,
      "text": "PostgreSQL connection pool exhaustion causes HTTP 504 timeouts. Increase max_connections to 200.",
      "source": "wiki/infrastructure_troubleshooting.md"
    }
  ]
}

The rrf_score is small by design. With k = 60, a chunk ranked first in one list and third in the other scores 1/61 + 1/63, roughly 0.032. These scores are only meaningful relative to each other within one query, so avoid absolute thresholds. (The sample reports total_retrieved as 2 but lists one result; treat it as illustrative.)

Implementing Hybrid Search and RRF in Python

The implementation below uses Pydantic and asyncio with simulated backends, so it demonstrates fusion logic rather than a drop-in component.

Data models for chunks and ranked hits

SearchResult is what each engine returns, with its position (rank) and native score; MemoryChunk is the fused output handed to the agent. Keeping raw_score helps debugging even though fusion ignores it.

import math
from typing import List, Dict, Any, Optional
from pydantic import BaseModel, Field


class MemoryChunk(BaseModel):
    chunk_id: str
    content: str
    metadata: Dict[str, Any] = Field(default_factory=dict)
    score: float = 0.0


class SearchResult(BaseModel):
    chunk_id: str
    content: str
    rank: int
    raw_score: float

The fusion engine

RRF scores each document as the sum of 1 / (k + rank) over every list it appears in. The class applies this to the vector and BM25 lists, remembers each chunk's content, sorts by combined score and returns the top results:

class ReciprocalRankFusionEngine:
    """
    Combines ranked result lists from Dense Vector and BM25 Sparse Search
    using the RRF algorithm: RRF_Score = sum(1 / (k + rank))
    """
    def __init__(self, k_constant: int = 60):
        self.k_constant = k_constant

    def fuse_rankings(
        self,
        vector_results: List[SearchResult],
        bm25_results: List[SearchResult],
        top_k: int = 5
    ) -> List[MemoryChunk]:

        rrf_scores: Dict[str, float] = {}
        content_map: Dict[str, str] = {}

        # Process Vector Search Ranks
        for item in vector_results:
            rrf_scores[item.chunk_id] = rrf_scores.get(item.chunk_id, 0.0) + (
                1.0 / (self.k_constant + item.rank)
            )
            content_map[item.chunk_id] = item.content

        # Process BM25 Search Ranks
        for item in bm25_results:
            rrf_scores[item.chunk_id] = rrf_scores.get(item.chunk_id, 0.0) + (
                1.0 / (self.k_constant + item.rank)
            )
            content_map[item.chunk_id] = item.content

        # Sort by combined RRF Score descending
        sorted_chunks = sorted(
            rrf_scores.items(), key=lambda x: x[1], reverse=True
        )

        output: List[MemoryChunk] = []
        for chunk_id, score in sorted_chunks[:top_k]:
            output.append(
                MemoryChunk(
                    chunk_id=chunk_id,
                    content=content_map[chunk_id],
                    score=round(score, 5)
                )
            )

        return output

A chunk found by both engines gets two contributions, so agreement is rewarded automatically. Note that the payload's weights are unused: this is plain RRF. To make dense_weight and sparse_weight matter, multiply each list's contribution by its weight, a variant usually called weighted RRF.

Orchestrating both searches

The retriever asks each engine for twice as many candidates as it needs, giving fusion more to work with, then passes both lists to the RRF engine:

class HybridMemoryRetriever:
    """
    Orchestrates Vector and BM25 search queries in parallel and applies RRF.
    """
    def __init__(self, vector_db_client: Any, bm25_index: Any):
        self.vector_db = vector_db_client
        self.bm25_index = bm25_index
        self.rrf_engine = ReciprocalRankFusionEngine(k_constant=60)

    async def retrieve_memory(self, query: str, top_k: int = 5) -> List[MemoryChunk]:
        # Fetch rankings from vector space and BM25 index
        vector_hits = await self._fetch_vector_results(query, fetch_k=top_k * 2)
        bm25_hits = await self._fetch_bm25_results(query, fetch_k=top_k * 2)

        # Merge and re-rank using Reciprocal Rank Fusion
        fused_memory = self.rrf_engine.fuse_rankings(
            vector_results=vector_hits,
            bm25_results=bm25_hits,
            top_k=top_k
        )

        return fused_memory

    async def _fetch_vector_results(self, query: str, fetch_k: int) -> List[SearchResult]:
        # Simulated vector DB query
        return [
            SearchResult(chunk_id="doc_102", content="Postgres timeout handling guide", rank=1, raw_score=0.92),
            SearchResult(chunk_id="doc_883", content="Database connection pooling configuration", rank=2, raw_score=0.85)
        ]

    async def _fetch_bm25_results(self, query: str, fetch_k: int) -> List[SearchResult]:
        # Simulated BM25 keyword query
        return [
            SearchResult(chunk_id="doc_883", content="Database connection pooling configuration", rank=1, raw_score=14.2),
            SearchResult(chunk_id="doc_401", content="HTTP 504 status codes and gateways", rank=2, raw_score=11.1)
        ]

The docstring promises parallel execution, but the two await calls run sequentially; use asyncio.gather to overlap the round trips. A real system's _fetch_ methods would call your vector database and BM25 index, apply the metadata filter and honor fetch_k. In the simulated data, doc_883 appears in both lists, so it tops the fused ranking despite being first in only one.

Adding GraphRAG for Corpus-Wide Questions

Hybrid search still retrieves local chunks. For reasoning across thousands of documents, GraphRAG builds a structured layer in three stages:

• Entity extraction. A background pipeline identifies entities such as services, error types and SQL schemas, extracts explicit relationships, and stores them in a graph database such as Neo4j or an in-memory library such as NetworkX.
• Community detection. Clustering algorithms such as Leiden group closely connected nodes into a hierarchy of communities.
• Hierarchical summarization. An LLM writes a report for each community, and broad questions are answered from those summaries instead of millions of raw chunks.

The trade-off is upfront cost: indexing needs many LLM calls, and summaries go stale as documents change. GraphRAG pays off for large, fairly stable corpora queried with global questions.

Common Questions

Why use RRF instead of averaging scores?

Dense and sparse scores live on unrelated scales. Cosine similarity usually falls between 0 and 1 in practice, while BM25 scores are unbounded and often exceed 50. Normalizing them introduces fragile parameters that break as data changes. RRF uses only positions, which keeps it robust across datasets without tuning.

How does GraphRAG reduce prompt size?

Instead of inserting dozens of raw chunks, say 50, the agent reads one or two precomputed community summaries. Savings of up to around 70 percent fewer tokens are cited, but results depend on your corpus and queries, so measure them yourself.

Key Takeaways

• In technical domains, pair embeddings with BM25 and merge rankings with RRF.
• k = 60 is the customary constant; it dampens any single top rank, so documents ranking well in both lists beat those ranking first in only one.
• RRF ignores raw scores; add explicit weights if one engine should count more, and filter metadata before fusion.
• Run both searches concurrently, and verify that "parallel" code really is.
• Use GraphRAG community summaries for system-wide reasoning over large collections, and budget for the indexing cost.