Building Agent Memory That Survives the Conversation Without Hoarding

A scene that repeats with nearly every agent that reaches production goes like this.

On Monday a user says they manage the Northwest region, only care about active projects and can't stand long answers. The agent says "Got it", and for the rest of that session it means it: replies are short, limited to the Northwest and free of drafts. The user is pleased.

On Tuesday the same person opens a fresh chat and asks how their projects are doing. The agent cheerfully asks which region they'd like to hear about.

The look on that user's face is the reason to think hard about memory.

Why saving everything fails

Nearly every developer reaches for the same fix: store the whole conversation and load it again tomorrow. It feels obvious, and it's a trap. The way it fails teaches more about agent design than most tutorials.

Suppose the full history is persisted and replayed whenever the user returns. Things happen in a predictable order.

Week one, it works. The agent really does remember everything, and you feel clever. This is the dangerous stage, since it seems to confirm the design.

Week three, it turns slow and costly. Each turn now carries thousands of tokens of old chat. You're paying for Monday's small talk while answering Friday's question. The context window is the agent's desk, and you've buried it under last week's paperwork before today's work begins.

Week six, it turns wrong. Nobody warns you about this stage. Old conversations contain statements that stopped being true. The user was focused on the Northwest until a promotion put them in charge of the whole country. The project was late until it shipped. Replaying everything makes the agent weigh stale and current facts equally. The problem is no longer that it remembers too little. It remembers too much, with no sense of what has expired.

Replaying full histories is, in effect, a machine for manufacturing stale context. So "how do I save everything?" was never the right question. These are:

Which things are worth keeping, which should be fetched when needed, and which should be discarded and regenerated from scratch?

Answer them and the memory architecture follows.

A three-tier design

Compare it with how you handle information at work. Some things you keep in your head during a meeting and happily lose afterwards. Some you write in a notebook because you'll need them next month. Some you never memorise, since they change and you check them each time. Agents need the same three tiers.

Tier 1: session memory, which is meant to die

This tier holds everything tied to the current conversation: the message history, intermediate tool results, a half-finished multi-step task, and the LangGraph state moving between nodes.

When the session ends, this tier is allowed to die. That isn't a forgotten save. It is designed to disappear, and that's a feature.

Beginners feel guilty about it, and they needn't. A conversation's play-by-play resembles a meeting: the transcript isn't what counts, the conclusions are. So the interesting question is what gets promoted to Tier 2 when a session closes.

Not the transcript. A small, explicit extraction step pulls out three categories:

1. Stable facts about the user, such as "manages Northwest region", valid until revoked.
2. Preferences, such as "wants short answers" and "exclude draft projects".
3. Decisions, such as "agreed to a weekly summary format".

That Monday chat might run to four thousand tokens. What deserves to outlive it is roughly forty tokens of structured facts, a compression of about a hundredfold. Unlike a summary, nothing important gets blurred, because you extracted named categories instead of a general impression. Summaries quietly drop constraints, and extraction is the cure: a preference either landed in the list word for word or it didn't, and you can write a test for that.

Tier 2: long-term memory, small, structured and editable

This is the compact store that persists across sessions, the user's notebook. Resist the urge to be clever. A boring format is exactly why it works:

{
  "user_id": "u-2481",
  "facts": [
    {"fact": "manages Northwest region", "updated": "2026-08-03"},
    {"fact": "prefers concise answers", "updated": "2026-07-19"}
  ],
  "preferences": [
    {"pref": "exclude draft projects from reports", "updated": "2026-08-03"}
  ],
  "decisions": [
    {"decision": "weekly summary every Monday", "updated": "2026-07-28"}
  ]
}

A few dozen lines of JSON per user, loaded into context at the start of every session. It's cheap enough to include always and small enough never to pollute.

Three rules keep it healthy.

Structured beats prose. Vague prose along the lines of "this person probably handles the Northwest and likes brevity" is useless later. Separate facts, by contrast, can be shown to the user, corrected, and removed individually.

Timestamp everything, and replace instead of append. When the user moves from Northwest to National, you don't add a second fact beside the first. You overwrite it. Once the store holds both "manages Northwest" and "manages National" you have rebuilt the stale-state bug with a database attached, which is worse, because now it stays stale forever.

Let the user see it and fix it. Even a basic "here's what I remember about you" command changes things. Memory a user can't inspect eventually becomes memory they don't trust, typically the moment the agent states, with total confidence, something that went out of date months ago.

What stays out of Tier 2 is anything you can look up. That belongs to Tier 3, and teams most often spend their effort there on the wrong thing.

Tier 3: regenerate on demand, deliberately not remembered

Everything that changes by itself lives here: project statuses, figures, dates, org charts, live numbers, the contents of databases and APIs.

The rule is that this tier is never memorised. Don't summarise it, don't park it in a memory store, and don't carry it over from an earlier chat. At the moment of need, the agent calls a tool and gets the current value.

Written down, that sounds obvious. It's also the most broken rule in agent memory, because caching live data as memory looks like an optimisation. It's actually a time bomb with a slow fuse:

• Store "Project Falcon is delayed" and then Falcon ships. For weeks the agent talks down a project that was delivered.
• Store "the team has 8 members" and then two people join. Every capacity answer is quietly off.

The sharpest summary of the whole architecture is probably this:

Use memory for facts about the person. Use tools for facts about everything else.

User preferences change rarely, and users tell you when they do. The world changes constantly and tells you nothing. So remember the user and query the world.

A failure case: "share everything" meets multi-agent

That's the tidy version. The route to it ran through an incident, in a fictionalised two-agent pipeline at a grid-services company.

Agent A talked to users and collected reporting requirements. Agent B produced the reports. When A passed work to B, the team repeated the beginner mistake at architectural scale: they handed over A's entire context, with the full conversation, every tool output and all state. The reasoning seemed airtight. Surely more context means a better-informed B?

The way it broke is the useful part. In one session a user mused aloud to Agent A: what if the archived projects were included? Then, on reflection, no, that skews the totals, leave them out. A perfectly normal bit of thinking out loud, and A handled it correctly.

Then the handoff fired and B received the whole transcript, tangent included. But B had none of A's conversational context for weighing it. Nothing in the payload said the archived-projects idea had been rejected, and out of context it looked like an instruction. B faithfully included the archived projects, and the totals in an official report were wrong for a reason nobody could find by reading B's code, because B's code was fine.

That's the multi-agent version of everything above: a handoff is a session boundary. When Agent B starts its job, it's Tuesday's new chat. The team had made the same "save everything" mistake sideways, between agents, instead of lengthwise across days.

The fix was the same fix. Transcript passing gave way to a structured handoff payload, in effect a disposable Tier 2 that A builds for B:

{
  "task": "generate weekly regional report",
  "requirements": [
    "region: Northwest",
    "active projects only",
    "exclude drafts AND archived"
  ],
  "format": "summary table first, details after",
  "constraints_confirmed_by_user": true
}

That's about fifty tokens with no ambiguity. B receives conclusions, not the winding path to them. The archived-projects class of bug became impossible, and not merely caught or handled, because abandoned ideas can't physically survive being extracted into a requirements list.

Take one sentence into any multi-agent design: agents should hand each other conclusions, never transcripts.

A checklist to apply this week

Five decisions you can make on an existing agent:

1. Let session memory die. Add an extraction step at session end, or after N turns, that carries over just facts, preferences and decisions and lets the rest lapse along with the chat.
2. Keep Tier 2 as dull JSON. Discrete items, timestamps, updates that replace. Never a growing prose blob and never append-only.
3. Give users a window in. A "what do you remember about me?" path, even a crude one, builds trust and catches stale facts for free.
4. Audit Tier 2 for facts about the world. Anything a tool could fetch fresh, such as statuses, counts, dates and numbers, should be evicted. Memory for the user, tools for the world.
5. Pass conclusions in handoffs. Use a structured payload with task, requirements and constraints. If your handoff carries a raw transcript, you've already shipped the failure above and haven't noticed yet.

The takeaway

Believing that memory means storing more is exactly backwards. Every failure above, from the tempting fix for Tuesday amnesia to stale facts to the archived-projects report, all trace back to hoarding too much, too carelessly.

Agent memory done well is mostly the practice of forgetting. Session state may die, long-term notes stay short enough to read in a single glance, and nothing the world can tell you fresh gets memorised. The agents that feel genuinely personal aren't the ones that remember everything. They remember the right forty tokens.