Customer Memory for AI Agents Under a Strict Token Budget

A support agent talks to a customer, and everything it needs has to squeeze into one context window. GPT-4o offers 128K tokens, and smaller models may offer 32K or less. That window must hold the system prompt, tool definitions, retrieved knowledge, the running conversation and whatever the agent remembers about this particular person. This piece describes a memory design for that setup: how facts are pulled out of finished conversations, scored, deduplicated, stored as vectors and fetched again under a strict token cap.

The window is a budget, not a bucket

Pour everything known about a customer into the prompt and the other inputs get squeezed. Knowledge retrieval is truncated, the conversation history is clipped, and the system prompt can slide so far back that the model loses track of its own instructions.

Include nothing, and the agent treats every chat like a first meeting. A customer asking about their appointment time gets "I don't have that information", even though they gave it last Tuesday.

The way out is a fixed allowance for customer memory. In the design described here it is 2,048 tokens. Within that allowance the system fits the facts that matter most for this customer in this conversation. Not every fact, not random ones, just the right ones.

Mining facts from finished conversations

Once a conversation ends, a consolidation pipeline goes through it looking for facts worth keeping. Extraction uses an LLM (GPT-4o-mini) with a structured prompt that asks what factual information about the customer can be drawn out, and to rate each fact from 0 to 1 for specificity, actionability and recurrence.

"Specific" is the key word. The facts must describe this customer and not offer generic observations. "Prefers morning appointments" is a fact. "Asked a question" is not.

The model replies with a JSON array, each item holding a text string and an importance score. The pipeline only passes along facts scoring at least 0.5. That cut-off drops throwaway observations, which tend to land around 0.2 to 0.3, and keeps anything that could help later.

A typical ten-message exchange yields two to four qualifying facts, and a long troubleshooting session might give six to eight. Most conversations produce at least one.

What the importance score captures

The single 0.0 to 1.0 number blends three qualities.

• Specificity. How particular is the fact to this customer? "Prefers email over phone" is. "Is a customer" is not.
• Actionability. Could the agent use it to answer better next time? "Has a recurring problem with invoice #4821" qualifies, whereas "said hello" doesn't.
• Recurrence. Will it matter in several conversations? A timezone of UTC+3 comes up every time scheduling does, but a question about today's weather happens once.

The team behind the design tried separate scores per dimension and found one blended score calibrated better and kept the prompt simpler. By their measurement the 0.5 threshold discards roughly 60% of extracted facts, leaving the high-signal ones.

Embeddings and duplicate control

Each fact that survives is embedded as a 3072-dimensional vector, which is simply the output size of the embedding model used, OpenAI's text-embedding-3-large. More dimensions allow finer similarity distinctions, an edge when you must tell a reworded repeat apart from a truly new piece of information.

Before a new fact is saved, it is compared against every existing fact for that customer. Cosine similarity above 0.92 means "duplicate, skip".

Why that number? People phrase one thing in many ways. "Prefers email communication", "Wants to be contacted by email" and "Said they don't check their phone much, use email instead" all describe a single underlying fact. At 0.92 the system catches such semantic repeats reliably. Lower cut-offs like 0.85 or 0.88 began merging facts that were really different, while higher ones like 0.95 or 0.97 let too many rephrasings pile up.

The check runs on every consolidation cycle. Without it, a customer who mentions their timezone in each chat would collect dozens of near-identical rows, burning storage and retrieval budget on repetition.

Why Postgres with pgvector

The facts live in PostgreSQL with the pgvector extension handling similarity search. A customer_memories row carries:

• content, the fact as plain text
• embedding, a vector(3072) column used for similarity search
• importance, the score assigned at extraction
• source_channel, where the fact was learned (webchat, whatsapp, voice and so on)
• created_at, the time of first insertion
• last_accessed_at, the latest time it was used in a chat

A dedicated store such as Qdrant or Pinecone would be the usual alternative, and three things argued against it.

1. The collections are tiny. At most 500 facts per customer. A separate service adds a network hop, connection handling and consistency worries that aren't justified at that size.
2. Transactions matter. Facts belong to a customer record. When a customer is erased under GDPR, their facts must vanish together with everything else. In pgvector that is one transaction, while a separate vector store would need distributed consistency.
3. The query is simple. Filter by customer_id with an exact match, then sort by vector similarity, which pgvector handles efficiently with a composite index.

The knowledge base is another matter. Product docs, FAQs and policy manuals sit in Qdrant, because that collection is large, with thousands of documents per tenant, serves heavy traffic and needs no transactional tie to relational data. Different problem, different tool.

Retrieval with a token ceiling

When a new message arrives and the agent has to respond, memory retrieval kicks in:

1. Embed the incoming message with the same 3072-dimension model used for storage.
2. Select the ten nearest rows from customer_memories for that customer_id, ordered by the <=> distance between each stored embedding and the query vector.
3. Walk through the results in similarity order, keeping a running token count.
4. Estimate each fact's tokens as len(content) / 4, a rough but conservative figure.
5. Stop as soon as adding the next fact would pass 2,048 tokens.
6. Update last_accessed_at on every fact that was retrieved.

The <=> operator gives cosine distance, so a smaller value means a closer match to the current message.

The ceiling is hard. The agent gets the most relevant facts that fit in 2,048 tokens, ranked by relevance. If the top fact runs 800 tokens, perhaps only two or three fit, and if every fact is short, at 50 to 80 tokens, fifteen to twenty might. The budget adapts to real content with no manual tuning. The default of ten rows rarely matters in practice, since the token cap usually bites first, but it keeps the database query itself from being unbounded.

The last_accessed_at write does two jobs. It tells the eviction policy which facts still earn their place, and it shows which memories the agent really draws on and which sit unused.

The 500-fact cap and who gets evicted

A customer can have at most 500 stored facts. The limit gives predictable storage cost per customer and keeps retrieval sharp, since cosine similarity over 500 vectors is fast and meaningful while over 50,000 it turns noisy.

When consolidation wants to save a fact and the customer is already at 500, eviction starts. The order of removal:

1. Facts untouched for 90 days or more, which are stale and probably irrelevant now.
2. Among what's left, the lowest importance score.
3. Ties broken by the oldest created_at.

Ninety days fits the business context. A fact learned three months ago and never used since is unlikely to matter, but one learned six months back and used weekly, like a timezone preference, stays indefinitely because its last_accessed_at keeps refreshing. Eviction runs after any consolidation that hits the cap and removes only what is needed, typically one to three facts, not a bulk purge.

Memory and knowledge are different things

The two systems solve different problems and compete for the same window.

Memory is per customer, small, personal and learned from interactions: a daughter's name is Amara, reports should be PDFs, an earlier delivery complaint was settled with a 15% credit. Knowledge is per tenant or global, large, shared and written by people: product documentation, pricing tables, return policies and troubleshooting guides.

Each gets its own token budget, 2,048 for memory and a larger one, typically 4,096 to 8,192, for knowledge. They share the model's context window without eating into each other. Even if memory uses its whole allowance, knowledge still gets its full one.

That split prevents a nasty case. Without separate budgets, a customer with 500 rich facts could crowd out all knowledge retrieval, and the agent would remember everything about the person while forgetting the product documentation it needs to help them.

How memory shapes reasoning depth

Memory also feeds the complexity classifier that decides how much reasoning a request deserves, in two ways.

The first is the "context dependency" signal. When retrieval returns highly relevant facts, meaning a low cosine distance to the current message, the context_dependency score goes up. The classifier reads this as "the right answer depends on customer-specific data" and may send the request to a higher reasoning tier. "Can you check on my order?" scores higher when stored facts mention a pending order than the same question from a customer with no history.

The second is competition with the thinking budget. In full chain-of-thought mode the model gives 4,096 or 16,384 tokens to thinking, depending on complexity. Memory facts injected into the prompt use tokens from the same finite window, so a customer carrying 2,048 tokens of memory leaves less room than one carrying 200. The budgets are managed independently and each is a hard ceiling, so the system never trades one for the other.

The parts that are still hard

Two difficulties don't show up in the architecture diagram.

Fact drift. Preferences change. "Prefers morning appointments" may turn into "prefers afternoon appointments" three months on. The system doesn't automatically detect contradictions between new and old facts. The 0.92 threshold sometimes catches them, when the new statement resembles the old one enough to match, but not dependably. For now old facts leave through the age and access policy, and better contradiction handling is on the roadmap.

Extraction hallucination. The extracting LLM occasionally infers things nobody said. A neutral question phrased a certain way might get stored as "customer seems frustrated with the product". The 0.5 threshold removes the weakest inferences but not all. Deduplication helps indirectly, since an invented inference rarely recurs, so it lingers as one rarely used row until eviction clears it.

Nothing here is a finished fix. Both issues are managed through the cap, the eviction policy and the access-frequency signal, which together favour recent, frequently used, high-importance facts. Stale guesses and outdated preferences sink to the bottom and go when space is needed.

Two flows in summary

After a conversation, the consolidation pipeline extracts facts with the LLM, scores importance, filters at 0.5, generates embeddings, removes duplicates at the 0.92 cosine threshold, stores what's new and evicts when the 500-fact cap is hit. Before a response, the incoming message is embedded, the top facts are retrieved by cosine similarity, collected within the 2,048-token budget and injected into the agent's context.