Persistent AI Agents in Practice: Costs, Memory Drift and the 80% Problem

For about two years the standard way to use a model was a conversation. You type a message, read the answer, then use it or argue with it. You're in the loop and you check every output before it goes anywhere. Then the advice shifted: stop typing prompts, run agents that keep working on their own, with memory, tools and permission to finish tasks without being watched at every step. Try that for a week, with agents that read code, run commands and open pull requests, and the pitch holds up in places. Plenty of it doesn't. The failures show up in a fairly predictable order.

Why people left prompts

The shift away from prompting reflects a real limit. A prompt is one question and one answer, fine for a lookup. It's poor at work where the model has to gather information from several places, reason over what it knows, use a tool and revise based on what came back. Asking for all of that in one shot isn't how real work goes.

What's needed is a loop: think, act, see the result, think again. Add memory that survives between sessions and you get what people call a persistent agent. One distinction is worth pinning down early. A chat transcript isn't the agent's memory. Memory is closer to a file the agent can read and write to remember things. That difference comes back below.

So the move makes sense, and nobody should pretend otherwise. What matters is what happens once you actually make it, starting with what it costs and ending with who is still responsible for the result.

The bill

The first thing nobody mentions: every API call starts from scratch. The provider keeps no memory of earlier calls, so the program has to resend the whole conversation each time it asks for something. Step one is cheap. By step twenty, with tool outputs and extra detail piled in, each call can carry thousands of words, and you pay for all of them on every call.

The arithmetic gets uncomfortable. A review of thirty engineering teams found that one late step in a task cost around fifteen cents. That sounds trivial until you multiply it by fifty steps per task, dozens of tasks per developer per day and a whole team. The result lands in six figures per month, and agents consume far more tokens than a plain chat for the same job.

Runaway cost is the other risk. A loop that keeps retrying something that can't succeed just keeps billing. There's a story circulating about a company hit with a bill of half a billion dollars from exactly this pattern. Whether or not that figure is exact, a single misconfigured agent burning a full daily budget on a ten-minute manual job is entirely believable. The unnerving part is that everything looks fine while it happens. The agent is working. It's just going in circles.

The fix that held up in practice has three parts:

• A firm daily budget for each agent, enforced outside the agent itself
• A rolling context window, so earlier steps drop away and don't pile up
• Sending routine work to an inexpensive model and promoting only the difficult cases to a stronger one

Teams that put in this plain instrumentation said agent costs fell by over 50% inside a month. A smarter model was never the lever. A leash was, and it's worth being blunt about that, because the first instinct when the bill arrives is to look for a better model.

The first pull request is a trick

An agent's first pull request can look amazing. A whole feature appears in minutes and every test passes, and the natural reaction is relief. Then comes the suspicion that the real work hasn't been done, and the suspicion is usually right.

Teams keep reporting the same pattern: the first PR impresses, and by the fifth one something breaks in production that no test caught. A retry was missing. A null case went unhandled. An auth check covered three endpoints and skipped the fourth. The code worked, but it didn't survive contact with reality.

That's often called the 80% problem. The agent delivers the obvious 80% and consistently misses the 20% made of non-functional requirements, failure modes and architectural consistency, because that 20% needs context the agent doesn't have. It writes code that runs, not code that endures.

Here's a one-line version of the trap: the code compiled, passed lint, and still did the wrong thing, because attention to it had slipped. The output looks complete, so it gets a quick skim, the way you'd glance at a junior's finished work.

Fixing it later costs more than doing it right at the start. The gaps repeat across the codebase, so there's no single place to patch them.

Memory comes in two flavors

The biggest trouble was memory, in two forms that weren't obvious at first.

The first is forgetting within a task. When a job runs long, the agent loses the decisions it made early on. The industry's usual answer is to hand it more context, a million tokens or two million, on the theory that seeing everything leads to good decisions. That doesn't hold up. An agent staring at a large pile of complicated code gets confused. It may fix one problem and cause a bigger one, because it sees many small parts and understands none of the whole.

The second form is sneakier, and it's the one that causes real damage: context drift. The agent follows config and spec files. The codebase evolves while those files stay frozen. Soon an always-on worker is confidently writing code to a standard that expired months ago, and nothing looks wrong, because it's obeying instructions faithfully from a stale copy of the truth. Multiply that across several agents and repositories and the gap widens every single day without a sound.

That's where the blame moves off the model. The broken piece wasn't the model. It was the project's memory, the part that lives on disk and outlives any session, which hadn't been built.

The developer's job moved

The developer didn't shrink. The job relocated, and the outcome is less dramatic than either the hype or the backlash.

Prompt crafting stops mattering much once an agent runs for a long time. What counts is the context: what the agent can see, what it can remember, what it's allowed to do and how you'd even find out what it did. The teams that got results didn't have clever prompts. They wrote specs, kept them current, fed them to the agent as its memory and reviewed the architecture themselves.

Spec-driven work makes the developer more important and shifts where the effort goes: defining what the thing should do, approving how far the agent may take it and reviewing the architecture it proposes. When requirements change, the spec changes, not the prompt. The agent drafts, implements and tests against that spec, and doesn't own the direction. That split was the only setup that stopped things breaking.

Underneath all this sits a measured sting. One study timed developers working in real codebases and found them roughly 19% slower, even though they felt about 20% faster. Feeling quick and being quick had separated. Persistent agents widen that gap when allowed to, since they excel at producing the sensation of speed. What narrows it is dull but necessary: governance, review, a spec that matches reality and a cap on spending.

Going back to pure prompting isn't the answer, since the loop is more powerful. But the agent isn't a colleague. It's a fast, confident contractor who has never admitted to being unsure. You keep a contractor like that because they're useful, and you read everything they hand you, every time.