Home / Articles

Six Agent Memory Tools Ranked by One Test: Can They Handle a Changed Fact?

Six agent memory tools face one test: can they handle a changed fact? Zep, Cognee, Mem0, Letta, LangChain Memory and plain vector stores, and where each fits.

Publisher: REACTAPP.TOP7 min read1609 words

This article is published in English.

Sources: I Compared 6 AI Agent Memory Tools. Three Fail One Test.

Picture an agent that was told, half a year ago, that Python was the user's whole identity. Last week the user mentioned a move to Rust. Now someone asks which language the new service should be scaffolded in. A plain vector store returns both facts with the same confidence, because embedding similarity has no notion of which statement is current.

That one failure makes the cleanest yardstick for agent-memory tools. Six of them went through it, and three did not pass.

A three-line test that most memory cannot pass

The eval is short. Store fact A. Store fact B, which supersedes A. Query the topic and see which fact comes back.

Plain storage hands back both facts. Real memory answers Rust and understands that Python is now history. The distance between those two behaviors is where teams burn through a quarter.

The pain is concrete. A support agent kept pushing the annual plan to a customer who had dropped to monthly three weeks before. The downgrade was sitting in the vector store, and so was the original upsell. The cheerful old fact scored a hair higher on cosine similarity, the agent served it up, and a human had to apologize. No one had shipped a bug. Retrieval was doing exactly what it was designed to do, and that was the issue.

This is far from a fringe worry. Three benchmarks define the field now: LoCoMo with 1,540 questions, LongMemEval with 500, and the newer BEAM, which runs at one and ten million tokens. Each has a dedicated temporal or knowledge-update track. Standardized exams exist precisely because the contradiction case is where systems quietly come apart.

Mem0's State of AI Agent Memory 2026 report labels the problem staleness: a stored fact about someone's employer holds up until they switch jobs, after which it is confidently wrong. The report describes a deeper form as well. Most systems overwrite a change and never register it as an evolution, so the knowledge that something used to be true vanishes. LongMemEval tests knowledge updates and temporal reasoning head on, and BEAM includes a category named contradiction resolution.

One caveat before the ranking. LoCoMo and LongMemEval were not re-run on local hardware. The verdicts rest on each tool's published evaluations, its documentation and the one repeatable hand test above. Vendor benchmark figures are self-reported, so they appear here as claims and not as established fact.

What follows runs from best to worst at the contradiction test.

1. Zep: a changed fact becomes an event, not an overwrite

At its heart, Zep runs Graphiti, a temporally aware knowledge graph and not a heap of embeddings. The important property is that it is bi-temporal. Each edge records both when something happened and when the system found out, and every relationship carries an explicit validity window.

Once "switched to Rust" comes in, Graphiti does not simply append a near-copy next to "prefers Python." It notices the clash and retires the older edge instead of removing it: the Python fact gets an end date and stays on record. A query returns today's answer, and the earlier history can still be examined. The design has three layers of subgraph, one for raw episodes, one for semantic entities and one for community clusters, loosely echoing how people keep what happened apart from what they know. In Zep's paper, it edges out MemGPT on the DMR benchmark (94.8% versus 93.4%). On the tougher LongMemEval it claims accuracy gains in the double digits and about 90% less response latency than a full-context baseline.

Use it when the recency of a fact changes the right action: CRM, health, support, or anything personalized over months. The drawback is plain. Running a graph database is real operational weight for a weekend project.

2. Cognee: graph memory without the managed service

Cognee is the open-source version of the same idea. It builds a knowledge graph from unstructured input, then retrieves by walking relationships as well as by vector similarity. Since it models entities and how they connect, it has the machinery to conclude that a new fact about an entity replaces an old one.

Done right, it passes. Second place reflects how much of the temporal wiring Cognee leaves with the user. It delivers the graph and the ingestion pipelines, while the rules for when a new fact retires an old one are for the team to write. Anyone who wants graph memory plus total control, with no managed service, no per-seat charges and data kept in their own store, will see that as the whole appeal. Anyone hoping to be done by Friday will find it closer to a second job.

3. Mem0: a pass through sheer retrieval volume, openly admitted

Mem0 is the widely used dedicated memory layer. It mostly passes, but how it passes is instructive. No graph is involved. Instead, three retrieval signals run side by side (semantic similarity, BM25 keyword matching and entity matching) and the combined results get ranked. Entity matching is usually the step that lifts the fresh fact above the outdated one.

Credit is due on several fronts. The vendor's own report admits that staleness and cross-session identity remain unsolved. It posts 92.5 on LoCoMo using about 6,956 tokens per query, compared with roughly 26,000 when the whole history is stuffed into context, which shows up plainly in inference costs. Its scoping is also sound: each write is tagged with a user, agent, session or organization ID, so one store can follow a single user across sessions and still keep tenants apart. It is the sensible default, offered as managed cloud, open-source self-hosting or a local MCP server. Passing the test, however, is not the same as solving the problem, a point its authors readily make.

4. Letta: solves a different problem and takes the blame for this one

Letta descends from MemGPT and borrows an operating-system idea. The context window plays the role of RAM: frequently needed facts remain in main context, the remainder moves out to external storage, and the agent changes its own memory by calling functions. It suits situations where a conversation outgrows any context window.

Resolving contradictions is not what it was built for. Rust or Python comes back depending on whether the agent happened to rewrite the relevant memory block, so replacing old facts is a behavior to be prompted and not a guarantee from the store. Context management is where it shines, while temporal truth gets only average treatment by design.

5. LangChain Memory: a toolbox, not an answer

LangChain ships memory classes for buffers, summaries, entities and knowledge graphs, all running on whichever store gets plugged in. Think of it as a bin of parts. The stock buffer and summary types flunk the test immediately, because they pile up or squeeze down the history and neither resolves a conflict.

A passing system can be assembled from the entity and graph pieces, but then the temporal logic is the builder's job and the framework has merely supplied the bricks. LlamaIndex Memory belongs in the same bucket, leaning toward remembering documents rather than reconciling facts that evolve. Both are useful glue. Neither is a memory system to trust with the test on day one.

6. A bare vector store: please don't call this memory

Here is the option to avoid, though it is the usual production choice: a raw Pinecone, pgvector or Chroma index filled with conversation turns. That is a storage layer, not memory. Recency, supersession and provenance are all missing. Embedding search hands back the k closest neighbors, and "prefers Python" and "switched to Rust" both land close to "what language?"

Mem0's report begins by calling time on this reflex, in which conversation history gets piled into the context window in the hope that the model keeps track. A team that connected a vector database directly to its agent and named it memory will see the contradiction test fail every time. In production that shows up as an agent firmly quoting a fact the user corrected a week ago.

Where to start

If recency can ever change the correct answer, begin with a temporal approach: Zep when the work should be handled for you, Cognee when owning the graph is preferable. If the need is cheap, good-enough recall across sessions and the occasional stale hit is acceptable, Mem0 is the fast path. A bare vector store makes sense only when a search index is wanted and memory is not.

Not every agent needs a knowledge graph, and nothing here says otherwise. A stateless support bot that forgets a customer between tickets is frequently right to do so. A temporal graph would be over-engineering there, and the much-ridiculed plain buffer is the correct pick. The test matters only when the agent is meant to know a person over time. In that situation, choose a tool able to separate what was said from what is still true.