Agent Memory Explained: Context Window, RAG and Persistent State

Few words cause as much muddle in agent development as "memory". When someone says their agent has it, they might mean the conversation is long, or that a vector database is plugged in, or that facts get saved to a file that survives a restart. Those are three separate systems with different costs, different ways of failing and different reasons to exist.

A single sentence resolves most of the confusion. The model has no memory, only an input, and what people call memory is just a policy for choosing what lands in that input.

The model is stateless. Each API call begins from zero, with no recollection of your last message, your last session or the name you gave it. What it knows during a call is whatever the request contains, and nothing else.

So the useful question isn't whether the agent has memory. It's what earns a place in this request, and where it comes from. There are three answers, and the job is to tell them apart.

Stateless model, stateful application

Newcomers assume the model is holding a conversation. It isn't. The application holds the conversation and replays it. When you send a fourth chat message, your code transmits messages 1, 2, 3 and 4, all of them, every time. The feeling of continuity comes from resending history, and the model simply receives a longer input than before.

With that in mind, the three layers stop being abstract:

• The context window is what sits in the request right now.
• Retrieval (RAG) is how you find material to put in the request.
• Persistent state is how facts outlive a request so they can be added back later.

A container, a lookup and a store: three different things. Blur them and you end up wiring a vector database into a problem that a preferences.json file would have solved.

Layer 1: the window, or the model's desk

Think of the window as everything the model can currently see. The system prompt, the history, tool definitions, tool outputs, retrieved documents and the reply being written all draw on one shared token budget.

Picture a desk. Whatever the model must reason about has to be physically on it, and a filing cabinet in another room doesn't count.

Where it shines: immediacy. Whatever sits in the window is completely visible with no lookup step, and the model can reason over all of it at once. For one task with a limited, known body of material, stuffing the window is the simplest route and frequently the right one.

Costs:

1. Money and latency follow what you resend. Turn 51 of a long chat resends 50 turns of history. Long context isn't free just because it fits.
2. Attention is uneven. Models tend to use the start and end of a long input most reliably and the middle least, the widely observed "lost in the middle" effect. A fact buried 60% of the way into a very long prompt is present but may never be effectively used. Fitting doesn't mean being read.
3. It ends. Once the session is over the window is gone. A context window never persists anything. This is the most frequent beginner mistake: a long conversation feels like memory, but it's a buffer, not a store.

The whole mechanic fits in a few lines:

messages = []

def turn(user_text):
    messages.append(
        {"role": "user",
         "content": user_text}
    )
    # every call resends
    # the ENTIRE history
    reply = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        messages=messages,
    )
    messages.append(
        {"role": "assistant",
         "content": reply.content[0].text}
    )
    return reply

# messages grows forever.
# so does your bill.

That list is the "memory" here. It exists inside your process only, so once the process ends, so does the list.

The window by itself is enough when the job is brief, the material needed is modest and known up front, and nothing has to outlast the session. A summariser, a reviewer for one file or a one-shot extraction needs nothing more. Don't add machinery there.

Layer 2: retrieval, the reference library

Sooner or later the material stops fitting, or stops being worth paying for. Maybe it's 10,000 support tickets, an internal wiki or a codebase. You can't put all of it in every request, and you shouldn't.

Retrieval-augmented generation divides the work: prepare an index in advance, then look things up when a question comes.

Beforehand you cut the documents into chunks, turn each chunk into an embedding (a numeric vector that captures meaning) and store it. When a question arrives, you embed it, find the chunks whose vectors lie closest and paste those few into the context window beside the question.

# 1. AHEAD OF TIME (once, offline)
for chunk in split(documents):
    store.add(
        text=chunk,
        vector=embed(chunk),
    )

# 2. AT QUESTION TIME (every request)
def answer(question):
    hits = store.search(
        embed(question), top_k=5
    )
    context = "\n\n".join(h.text for h in hits)
    return client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        # retrieved chunks land in the same request
        messages=[{
            "role": "user",
            "content": f"{context}\n\nQ: {question}",
        }],
    )

Notice what actually happened. RAG didn't give the model memory. It gave you a selection strategy for the window. The retrieved chunks land in the same request as everything else, so RAG is a funnel into Layer 1 and no alternative to it.

Strength: scale and freshness. Millions of documents can change independently of your code, while the model only ever sees the handful that matter for this question.

Costs:

1. The outcome is decided early. Chunk size, embedding model and index design are locked in when you build the index. A fact cut across two chunks, or worded unlike the way users ask, simply isn't found, and the model then answers with confidence from whatever it received. To an observer, poor retrieval and hallucination look identical.
2. It's mostly read-only. Classic RAG is a library card and not a notebook. The agent looks things up but writes nothing down, and nothing it learns in a session reaches the index unless you build that path.
3. There is an extra step. Embedding the query and searching the index add latency to every request.

Retrieval fits when the knowledge base is big, common to every user and rarely changes, for example documentation, policies, product catalogues and code. The giveaway is that one corpus serves every user. If what you want to remember belongs to a single user and changes as you talk, retrieval is a poor fit and Layer 3 is what you need.

Layer 3: persistent state, the notebook

Persistent state is what survives when the process dies. It isn't what the agent knows about the world but what it has learned that should outlast this session: user preferences, task checkpoints, decisions already taken, things tried and abandoned, distilled summaries of earlier runs. These are small, specific, valuable facts that the agent itself wrote down.

The pattern is a read, modify, write cycle wrapped around the session:

# session start: load
with open("state.json") as f:
    state = json.load(f)

system = (
    "Known about this user:\n"
    f"{json.dumps(state)}"
)

messages = []

def turn(user_text):
    messages.append(
        {"role": "user",
         "content": user_text}
    )
    reply = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=system,   # state in
        messages=messages,
    )
    return reply

# ... along the way ...
state["prefers"] = "metric units"
state["last_checkpoint"] = "step_4_done"

# session end: save
with open("state.json", "w") as f:
    json.dump(state, f)

Nothing more is needed conceptually. A JSON file counts as a real implementation; so do a database row, a key-value store or a folder of markdown notes. The medium is the dull part, and the interesting part is who decides what to save and what to reload.

Making that choice well is the difficult bit, and it's where agent frameworks diverge. The most direct current option hands the job to the model: with Anthropic's memory tool, Claude gets a small set of file commands (view, create, str_replace, insert, delete and rename) that work on a sandboxed /memories directory, so the agent picks what deserves saving and reads those notes at the beginning of the next session. Your own application executes the operations on whatever storage you choose. The tool runs client-side, which keeps the files in your hands.

Strength: continuity, personalisation and long-running work. An agent that picks a multi-day task back up where it stopped needs this, and so does one that shouldn't ask your timezone twice.

Costs:

1. Somebody must curate. Write down everything and you've rebuilt an unsearchable transcript. Write too little and continuity breaks. Good persistent state is distilled hard.
2. Stale facts are worse than none. A preference saved eight months ago and never revisited gets applied with full confidence. Retrieval failures give vagueness, while stale state gives confidently wrong behaviour.
3. Loading isn't free. Whatever you load still takes space in the window, so Layer 3 feeds Layer 1 too.

Reach for persistent state if the same user or task triggers the agent repeatedly, and what run n learns ought to alter run n+1.

The three side by side

The row that matters most is the one about restarts. It's the dividing line between a long chat and an agent with memory.

They compose

A real agent uses all three, and it helps to watch them work in one request. A support agent is asked whether an order qualifies for the extended return window.

1. Persistent state is loaded first: this customer has the business plan and likes brief answers. Two facts, roughly forty tokens.
2. Retrieval runs on the question and pulls the returns-policy section for business plans along with the two latest exceptions. Three chunks, about twelve hundred tokens.
3. The context window takes both, plus the live chat and the tool definitions, and the model reasons across the entire desk together.
4. After replying, the agent writes one line back to persistent state: asked about the returns policy in mid-August, so the next session doesn't treat them as a first-time caller.

One request, three layers. The window is where everything ends up, while retrieval and persistent state are two separate routes to it.

This is also why long-running agents are usually advised to pair memory with context editing: old tool results get cleared from the window to keep it light, and the facts worth keeping were already saved, so the window stays small without losing knowledge.

Which one do you actually need?

Beginners nearly always grab a vector database first. It's the most visible part of the stack and the least often the answer. Climb the ladder instead.

Start with the window. Does everything relevant fit in a single request at a price you like? Then stop. No index, no store.

Bring in persistent state once the agent runs a second time. When session two should know what session one learned, Layer 3 is needed, and a JSON file is a genuine answer. Do it ahead of retrieval; it's cheaper, simpler and what most people actually want when they say they'd like the agent to remember them.

Bring in retrieval when the corpus is bigger than the window. That means lots of documents, shared between users, edited separately from your code. It's the heaviest layer to build and keep running, so only take it on when it's earned.

One trap to dodge: building a RAG pipeline for what was really a preferences file, then puzzling over why the agent still forgets someone's name. Retrieval searches a corpus, and it was never going to hold the fact that a person prefers metric units.

The short version

• Context window: the desk. Instantly visible, costly per token, and cleared the moment the session closes.
• Retrieval: the library. Huge, searchable, refreshed on its own schedule, and something the agent only reads.
• Persistent state: the notebook. Small, written by the agent, and the only layer that carries a lesson from today into tomorrow.

The model itself keeps nothing. Apparent memory is an engineering choice made again with every request: what earns a spot in this window, and where should it come from?

For further reading, Anthropic's engineering posts on effective context engineering and on building effective agents, plus its documentation for the memory tool and context editing, cover this ground in more depth.