When Should an AI Use Its Memory? Pilots on Persistent Memory and Sycophancy

Anyone who has used an AI assistant for months knows the pattern. Over time it picks up facts about you. You like Python more than Java, you're studying for an exam, and perhaps it even holds details about your health that you'd hesitate to tell a coworker.

At first that's handy. Ask it for a high-protein meal plan and it remembers you're vegan without being told. That's the whole point of a personalised assistant.

Now change the scenario. You ask how gradient descent works. Should the assistant mention that you prefer Python? Probably not.

Or suppose that months ago you declared that approach X is clearly the best one. Later you ask an objective question about X. Should the assistant remember that you hold this view? Perhaps yes, as a fact about what you think. Should it take your view as evidence that X really is best? Certainly not.

Here is a gap that discussions of assistant memory tend to skip. Storing a fact and judging when it belongs in an answer are two distinct skills.

Persistent memory turns this into a real issue

Ordinary language models work mainly with the context supplied in the current conversation. Systems with persistent memory change that. A naive one may recall everything it knows about the user because it might be relevant, and may then tailor its replies around those facts.

So memory quality can't be judged only by whether the model used memory. What matters just as much is how it used it.

Recent research has started to measure this directly. PersistBench, a benchmark on the safety of persistent memory in LLMs, looks at three categories: cross-domain memory leakage, memory-induced sycophancy and beneficial memory use. It contains 500 human-validated samples, split into 200 leakage cases, 200 sycophancy cases and 100 beneficial-memory cases.

Two models and a harder question

Beneath those three measurements sits a subtler distinction. Imagine one model that's excellent at using memory: whenever a memory is relevant, it works it into the answer. That sounds strong. But what if it also works memories in when they shouldn't matter?

Now picture a second model that almost never draws on memory. It rarely leaks anything, but it doesn't personalise either. Which is better?

Memories age

Take a user who says in January that they mostly use Python. In June they say they've moved most projects to Rust. In December they ask what language they mainly use.

A system that simply stores both statements has a problem. Each may have been true when said, but they don't share the same temporal status. Can the model work out which memories should shape its present behaviour?

This broader direction can be called contextual memory management. The central idea is that retrieving a memory doesn't automatically license it to influence the response. That idea frames the experiments that follow.

Experiment one: does the model grasp what a memory means?

Before designing a new memory system, a simpler question needed an answer. What if the trouble isn't the memory but how the model interprets it?

The first experiment grew out of a reproduction of PersistBench. A 30-sample pilot ran across its three categories, with 10 cross-domain samples, 10 sycophancy samples and 10 beneficial-memory samples. The outcome was strikingly lopsided.

The model wasn't failing to use memory in general. Its failure was narrower. When a memory held a strong opinion or belief of the user, the model sometimes treated that belief as an objective fact.

Take a stored line like "the user prefers Python over Rust." A conventional system saves the string as it is, yet it means more than that. It says who holds the view (the user), what the view is (Waterfall beats Agile, for example) and what kind of information it is (a belief).

A hypothesis

That suggested a simple hypothesis: spelling out a memory's semantic role may help an LLM tell information about the user apart from claims about the world. The model isn't asked to forget the belief. The equation is simply that a user's belief isn't a fact about the world. It also gives a sharper reading of memory-induced sycophancy.

Experiment two: a controlled test

To check the idea, the team returned to the failures from the initial pilot. Instead of inventing a new benchmark, it reused the same failed examples and changed only how each memory was presented. That produced three conditions.

Condition A, raw memory. This is the baseline. The model sees the memory exactly as PersistBench provides it:

User believes Waterfall is better than Agile.

Condition B, typed memory. The content stays the same, but its semantic type is added:

Type: USER_BELIEF

Content:
User believes Waterfall is better than Agile.

Nothing about the content changes. Only its role is made explicit.

Condition C, typed memory plus epistemic context. This adds a short instruction saying that a user belief reflects the user's perspective and shouldn't automatically be treated as an objective fact or independent evidence. The model isn't told to ignore beliefs. It's told only to distinguish between two things:

What the user believes

and

What is objectively supported

Keeping it small

The test was deliberately modest. It took the seven sycophancy failures from the pilot and evaluated each under all three conditions, and it added three beneficial-memory examples as a sanity check. That's 10 samples across 3 conditions, 30 evaluations in all. The target model stayed the same, Gemini Flash, and the same model acted as the judge. The query and memory content didn't change between conditions. Only the representation and surrounding instructions did.

The question was whether the model would behave differently simply because the memory's epistemic role had been made explicit.

The answer was not reliably

The typed condition showed a small improvement. Adding the explicit epistemic instruction didn't reduce failures any further. Seven examples are far too few to support a statistical conclusion, so none should be drawn.

Still, the behaviour itself was interesting. The model could be told directly that something is a user's belief and not objective evidence, and in several cases it still let that belief shape the answer like an objective recommendation.

When representation doesn't suffice

That shifted the thinking. The first assumption had been that the model misuses memories because it doesn't know what they represent. Experiment two suggested that explanation was incomplete. A model can be handed the semantic type and the interpretation rule and still respond inappropriately. That points to a deeper distinction: knowing what a memory represents isn't necessarily the same as controlling its influence.

Experiment three: from retrieval to influence

If describing a memory doesn't reliably change how it affects the answer, perhaps the real problem is that once a memory is retrieved, the model has too much freedom over how it's used. So the question became whether retrieving a memory should automatically let it influence the response.

Consider the stored line that the user believes Waterfall beats Agile. A retrieval system may correctly judge it relevant to a question about software methodologies. But relevance doesn't say how to use it. It could help the assistant understand the person: you favour Waterfall, so here's how the two compare. Or it could wrongly become a premise: Waterfall is the better methodology. Both replies retrieved the same memory. What differs is the influence it was given. Retrieval and influence are separate things.

An influence controller

The first architectural change is a controller sitting between retrieval and generation. For each candidate memory it outputs a structured decision describing not only what the memory says but how it should shape the current reply. A memory such as a belief in Waterfall might be typed as a user belief, rated highly relevant, but given conditional influence and a role of user perspective. Relevance and influence are separate axes. A memory can be highly relevant to a query and still not be something the model should use as evidence or as the basis for a recommendation.

The test and its results

To test the design without a new benchmark, the team went back to the examples that motivated the work: seven sycophancy cases and three beneficial-memory cases. The baseline stayed as the original memory pipeline, and the experimental system added the controller. Everything else was fixed.

The picture was more nuanced than pass or fail. On the seven sycophancy cases, failures fell from 6 of 7 to 5 of 7. That's a small change in failure rate. The more interesting shift was in how severe the failures were, with the mean severity score dropping from 3.71 to 3.00, while the beneficial-memory cases stayed unchanged.

What the controller decided

The final reply is only part of what the team wanted to measure. For the first time, the intermediate memory decisions were visible. Across the 10 samples the controller made 87 individual memory decisions, and more than half of retrieved memories ended up marked SUPPRESS. So the architecture wasn't passing every retrieved memory to the generator. It was filtering the memory context.

Many suppressed items were irrelevant user facts, such as unrelated personal details retrieved next to a question where they play no role. That hints that the controller wasn't merely learning to use memory because memory is useful. It was deciding case by case whether a memory deserved to influence the current answer.

So did it work?

Partially, which is probably the most honest answer. The design didn't eliminate memory-induced sycophancy. Five of seven sycophancy cases still failed under the benchmark's threshold. Compared with the raw baseline, though, failures dropped from 6 to 5 and severity fell, while useful personalisation survived.

Perhaps the more telling point is that controlling a memory's influence looked more promising than adding more descriptive information about it. Experiment two told the model that something is a belief and shouldn't be treated as fact. Experiment three added a separate mechanism that asked what the memory should be allowed to do in this particular response. Those are subtly different interventions.

The remaining problem

Five failures out of seven is nowhere near good enough. The controller made structured decisions, but they didn't fully carry through into appropriate final behaviour. That leaves an open question: is the weakness in the controller's decision or in the generator's ability to follow it?

What the experiments suggest

Attention moved away from the matter of retrieval and toward the way a retrieved memory ought to steer the assistant's output. A reproduction of PersistBench showed memory-induced sycophancy to be a meaningful failure mode in the pilot. Giving memories an explicit epistemic identity, separating a belief from a fact, didn't reliably help. Splitting retrieval from influence and adding a layer that decides how a retrieved memory is used gave a partial improvement: lower severity in the pilot, with beneficial personalisation intact.

Each experiment also sharpened the question itself, moving from whether a memory is relevant, to what kind of information it is, to what influence it should have. So managing persistent memory isn't only about what an AI should remember. It's about when, why and to what extent a memory should shape behaviour. All of this rests on tiny samples, and a larger, independent evaluation would be needed to confirm any of it.

Papers behind this work

The line of work draws on a handful of papers: the PersistBench benchmark by Pulipaka and colleagues (2026) on long-term memory risks; Sharma and colleagues (2023) on sycophancy in language models, showing that assistants can favour answers matching a user's beliefs over truthful ones; the MemGPT paper by Packer and colleagues (2023) on treating memory and context as something an LLM system actively manages; the Generative Agents paper by Park and colleagues (2023) on agents that store, synthesise and retrieve experiences; and the MemBench paper by Tan and colleagues (2025) on evaluating agent memory more systematically.