Why Chat Memory Fails at 25% Recall and How to Fix the Retrieval

Chat memory that holds up for one session and then goes blank on anything older than a few months is a pattern you'll see again and again in AI companion apps. A freelance brief posted on Upwork turns the vague complaint into measurements, so it makes a better case study than any hypothetical.

The client's numbers

The client's own tally: roughly 5,000 stored messages, 300 episodic events and about 3,500 semantic insights. They picked eight concrete facts the user had stated back in June and asked about them the way a real user would today. Only two came back, a 25% hit rate, even though every raw message was still stored word for word. Nothing gets lost when memories are written.

Take that last claim seriously. Whatever is going wrong, storage isn't the culprit.

And this wasn't a weekend prototype. The stack ran Postgres with pgvector, retrieval in two stages with a reranker, and custom orchestration plus extraction code. It held several kinds of memory (episodic events, semantic insights, a register of the people in the user's life, and procedural style), each with its own retrieval route. Every turn was instrumented to log what was fetched, what was dropped and what finally reached the model. The team had also benchmarked mem0 against 120 of their own messages and was running a Hindsight instance in shadow mode. Sensible experiments, and the recall rate stayed at 25%.

Where it actually breaks

A single probe makes the problem obvious. Type "what words does my dad use" and the system comes back with earlier messages in which the user complains about their dad. The single message that quotes him doesn't show up.

That isn't a quirk of this codebase. Embedding similarity over a chat log behaves exactly that way, because the text closest to a question is nearly always another question. A query and its answer occupy different neighborhoods of the same vector space, and cosine similarity can't tell they belong together. So the memory store is fine. The weakness sits in orchestration, meaning what gets picked, ranked and packed into the context at generation time.

Four fixes, in order of effort

Rewrite the query. Skip embedding the raw question. Have an LLM produce several phrasings shaped like the answer would be: exact phrases the user's father uses, words he calls the user, direct quotes attributed to him. The search then targets what an answer looks like instead of what a question looks like.

HyDE. Gao and colleagues proposed in 2022 going a step further. Don't turn the question into search terms; ask the LLM to write a hypothetical answer and embed that. An invented answer lands much nearer to a genuine one than the question does. For this particular failure it's arguably the most direct remedy.

Rerank on more than similarity. With a pool of candidates in hand, sorting by semantic closeness alone recreates the original mistake on a smaller scale. Score on entity match, how likely a snippet is to be an answer, whether it sits next to a turn that plausibly contains one, and the time context. Remember the ceiling, though: a reranker only reorders what retrieval handed it and can't conjure something that never surfaced.

Retrieve in several hops. One pass handles single-fact questions. It breaks on aggregate ones. A question like "what nicknames has my dad given me" has an answer spread over five messages across three months. Handling that takes a broad first recall, grouping the candidates by type, a sufficiency check on what's been collected, and another search when it falls short. It's the same retrieve, reflect, decide-whether-to-search-again shape as Self-RAG and Corrective RAG (CRAG). Give it a firm stop, either a cap on hops or a confidence test, or it will loop forever or make every reply expensive.

A bare-bones loop looks like this:

def retrieve_memory(user_query, vector_db, llm, max_hops=2):
    rewrites = llm.rewrite(
        query=user_query,
        instruction=(
            "Turn this question into 3-5 retrieval-oriented phrasings "
            "that describe what the ANSWER would say, not what the "
            "question says."
        ),
    )

    candidates = []

    for q in rewrites:
        candidates += vector_db.search(q, top_k=20)
    ranked = []
    for hop in range(max_hops):
        ranked = rerank(
            user_query,
            candidates,
            factors=[
                "semantic_relevance",
                "entity_match",
                "answer_likelihood",
                "conversational_relationship",
                "temporal_context",
            ],
        )
        if llm.is_sufficient(user_query, ranked[:10]):
            break
        follow_up = llm.next_query(user_query, ranked[:10])
        candidates += vector_db.search(follow_up, top_k=20)
    return ranked[:10]

None of it is exotic. What matters is the sequence: repair recall and query construction first, repair reranking second, and only then think about adding infrastructure.

Advice for the client

The audit request had three parts: where memory leaks between storage and generation, whether the design survives ten times the data, and a straight recommendation between fixing what they have and adopting something new.

Where it leaks. Between retrieval and generation, and precisely at query construction, before reranking can help at all.

At 10x. The multi-type memory design and the logging are sound and will scale. Single-pass embedding search won't, since more data puts more look-alike questions in competition for the same candidate slots. The rewrite and multi-hop layers turn from optional to necessary well before that mark.

Fix or replace. Fix what they own. mem0 and Hindsight are both storage layers for memory. Neither touches orchestration, and orchestration is where the missing 75% disappears. Changing the database won't help a query that's still searching for the wrong thing.

Sometimes a memory system isn't forgetting at all. It's answering a question nobody asked, because the query and the evidence were never after the same thing.