Searchable Long-Term Memory for Data Engineering Agents

Stateless agents solve problems well. They are poor at remembering that they solved them.

The amnesia problem

Picture an agent that spends almost an hour on a NoClassDefFoundError in a freshly generated Databricks notebook. It works through the stack trace, finds the library missing from the job cluster, applies a fix and confirms a clean run. Solid autonomous debugging.

A week later a different pipeline hits the identical error, and the agent has no idea it has ever seen it. The whole debugging session happens again at full token cost, with a real chance of settling on a different fix this time.

That's not a model failure. It's how agents behave by default, and they forget in three ways:

1. When a session ends, whatever was learned in it disappears.
2. Long sessions eventually get summarised or compacted, which keeps the broad storyline but often drops the implementation details that mattered.
3. In multi-agent systems, worker agents usually start from a clean context, inherit nothing from earlier workers and keep nothing after their task is over.

A team running a multi-agent system that generates Databricks lakehouse pipelines was paying this amnesia tax on all three counts. Their fix was a searchable long-term memory layer.

What "searchable memory" means

Two different things get called memory in agent systems, so the terms need pinning down.

The context window works as the agent's working memory. Whatever sits in it is instantly available to the model, but only for the length of the session. Once information drops out, as far as the agent is concerned it's gone.

Long-term memory is a persistent store that outlives sessions, tasks and agents. Just storing things isn't enough, though. Stored information becomes useful memory only when it is captured automatically, can be searched effectively, and gets recalled into the context window when needed. The last step is the one that counts. A perfect lesson that never gets retrieved might as well not exist, and a retrieved lesson is worthless if it doesn't reach the agent making the decision.

Cognitive science supplies a taxonomy that maps well onto agent systems:

• Episodic memory holds records of specific past experiences: archived session transcripts, earlier failures with their fixes, decisions from previous builds.
• Semantic memory holds structured facts, such as "entity X belongs to data product Y" or "this source system authenticates this way", ideally with validity intervals since facts change.
• Procedural memory is the know-how: instruction files, skills and the conventions every session loads.

Two tiers of memory

One decision removed much of the complexity: splitting memory into two tiers.

Tier 1, curated memory. A small, hand-maintained set of rules loaded into every session. Think architectural standards, security requirements, naming conventions and governance constraints. Prompt space is expensive, so this tier is reserved for what must always be available.

Tier 2, searchable memory. A large archive of experience captured automatically and retrieved only when relevant: validated fixes, troubleshooting discoveries, deployment lessons and recurring failure signatures. It stays out of the prompt by default.

The rule of thumb:

A fact that has to be obeyed every time belongs in the curated tier. A fact that's useful when a similar situation returns belongs in the searchable tier.

That split preserves knowledge without drowning the context window.

The multi-agent system that needed it

The system in question handles the whole data engineering lifecycle, turning a business requirement into a production-ready pipeline and dashboard. An orchestrator coordinates specialised workers, each with a single job and only the tools it needs:

• Functional specification reads the requirement ticket, drafts a spec and gets sign-off from the human requester.
• Technical design turns the approved spec into a blueprint of data contracts and source-to-target mappings.
• Ingestion, transformation and integration produce the actual pipeline code. That means Bronze notebooks pulling from Kafka, SAP, REST or files, Silver notebooks that reshape the data, and Gold dimensional models, all on a medallion layout.
• Validation is the only agent allowed near a live cluster. It runs each generated notebook against disposable scratch tables and checks the output.
• Repository manages branches, commits and pull requests.
• Monitoring keeps an eye on CI/CD once a change has merged.
• Dashboard builds analytics dashboards from the results.
• Cost tracking reports what all this costs.

The architecture is deliberately isolationist. Workers run in parallel with disjoint file ownership, can't talk to one another and begin every task with a completely clean context. The orchestrator writes a self-contained task prompt, the worker does bounded work, returns a result and vanishes.

Isolation makes the system safe and parallelisable. It also makes the amnesia worse than in a single-agent setup, since every worker is born yesterday, every time. Each role builds up its own kind of experience worth keeping, and none of it survives the end of a task unless something outside the agents' contexts holds it.

Designing memory for many agents

Four mechanisms address the problem.

1. Automatic capture

Hooks in the agent runtime fire at two moments, when a session closes and right before the context is compacted. Each one reads the transcript and pulls reusable lessons into a memory store. Nobody has to remember to write documentation or update a knowledge base.

Memory operations are also non-blocking. If the memory infrastructure goes down, the hooks quietly do nothing. The memory layer is there to help. It should never turn into something the real work can't proceed without.

2. Scoped access

The orchestrator owns the full memory toolset. Workers get just three capabilities: read their own diary, write their own diary, and search the store. The tooling enforces this, not the prompt, which keeps responsibilities clear and stops workers modifying memory outside their remit.

3. Role-specific diaries

Every worker gets a private memory namespace. Runtime failures and verified fixes pile up in the validation agent's diary, while the CI/CD monitor collects failure signatures and the remedies that worked. The other agents keep similarly narrow stores.

The diaries aren't communication channels. Workers don't pass messages through memory, and coordination stays with the orchestrator, which keeps each diary coherent and relevant.

4. Recall at the context level

This turned out to be the most important decision. Workers start with an empty context, so a retrieved memory only helps once it has been injected into that context. The working rule:

A memory that's retrieved but not placed in context doesn't exist for the worker.

Whenever relevant lessons are found, the orchestrator writes them straight into the task context before dispatching the work. Retrieval alone doesn't suffice. To be applied, a lesson has to be visible.

Building it on MemPalace

The layer is built on MemPalace, an open-source, local-first memory framework. The store, the embeddings and the search index all run locally, so no data leaves the environment, which matters when agents handle enterprise data and production systems.

MemPalace keeps memories as text chunks enriched with metadata such as the creating agent, a category and a timestamp, arranged in separate namespaces. That makes role-specific diaries workable: each agent writes to its own space and builds up expertise over time.

Capture runs through the lifecycle hooks above. At session end, or before compaction, the transcript is cleaned, turned into reusable lessons, embedded with a local ONNX model and indexed for later retrieval. All of it happens in the background, so agents never wait on memory operations.

Retrieval is hybrid, mixing vector similarity with keyword matching. That helps in data engineering, where some signals are semantic and others are exact strings. An error name such as NoClassDefFoundError or an exact Databricks runtime version has to match literally, yet the same problem phrased differently should still turn up.

Agents reach memory through MCP. The orchestrator sees the whole toolset, while workers see only diary_read, diary_write and search, with boundaries enforced by tooling rather than prompts.

MemPalace was chosen because it's local-first, free and easy to integrate, but the architecture doesn't depend on it. Mem0, Zep or Letta could fill the same role. What matters isn't the framework but the feedback loop: capture experience, store it, and make it available when the same problem returns.

What changed in practice

The first visible gain was failure triage. A core rule for the validation agent is short: on every failure, search before debugging.

mempalace_search(query="NoClassDefFoundError DBR 14.3")

When a match comes back, the agent gets a lesson distilled from an earlier successful resolution:

The Bronze notebook for Orders crashed on DBR 14.3 with a NoClassDefFoundError linked to the Avro deserialiser. What fixed it: declare the serialiser library explicitly in the job cluster configuration.

The difference is large. Rather than rediscovering a solution from first principles, the agent starts from evidence of something that worked. And the value isn't only speed. It's consistency. Without memory, two agents facing the same problem might each pick a different, reasonable-looking fix. With memory the system settles on one that's known to work and keeps using it.

Similar gains showed up elsewhere. The CI/CD monitor now folds historical occurrences of a failure signature into its diagnostic reports, session-start hooks restore relevant recent context, and specialised agents build expertise over time instead of rebuilding it constantly.

No retraining is involved. The learning loop lives outside the model: experience, lesson, storage, retrieval, application.

Lessons from production

Be careful with historical backfills. The first attempt imported large volumes of old transcripts. Results were disappointing, because older material often overshadowed newer, better lessons and made retrieval noisy. The team dropped the bulk import and started fresh from the day capture went live. Recency and quality beat volume.

Write memories in plain language. The framework offers compact entry formats, and the team passed on them. Retrieval does best when a stored lesson includes the very words a later search will use: product names, entity names, error messages and exception classes. Findability matters more than small size in a memory store.

Choose carefully what to write down. Not everything seen deserves a lasting place. Worth keeping are confirmed fixes, validated findings and reusable lessons. Worth skipping are routine logs, transient observations, repetitive status updates and sensitive information. Poor memory piles up fast and can do more harm than having none.

Takeaways

Whenever agents do real work repeatedly, they're producing experience that's worth keeping. What the build suggests:

• Divide memory into hand-curated rules and searchable experience.
• Capture lessons automatically, especially before context compaction.
• Give workers narrow, role-based permissions.
• Retrieve before reasoning whenever a similar problem may already have been solved.
• Put recalled memories straight into the prompt.
• Treat memories as evidence, not unquestionable truth.
• Favour memory quality over memory volume.

The deepest lesson is architectural. A lot of effort goes into making agents smarter, but in practice long-term value comes from making systems remember.