Fine-Tuning Got Cheap, but Agents Still Need a Gate for Weights

Each night the brain quietly runs a batch job. The day's scattered episodes sit in the hippocampus, and during sleep they are replayed and selectively written into cortex. Researchers have watched this replay happen in rats hours after the maze had been taken away. The next morning there are skills that were only borrowed the day before, and the training run itself is never seen.

Andrej Karpathy pointed at the missing phase in agents some time ago: no distillation step, no sleep. Roughly eighteen months on, the field is building one. A 2026 paper even titled itself "Language Models Need Sleep" and describes folding a model's in-context memories into newly added expert weights during an explicit sleep phase.

The wake and sleep split has therefore moved from metaphor to architecture diagram, which is both the good news and the trap. Consolidation has become cheap enough to put on a schedule, and that is why the interesting engineering has shifted elsewhere, to the filter that picks the handful of lessons, among ten thousand, that merit being baked into weights.

Four skills come out of this. First, keeping the fast loop (context, handled per request) apart from the slow loop (weights, changed per release), with a price on each. Second, making a privileged self-teacher out of the agent already in production, no larger model required. Third, scoring each lesson against four promotion criteria before any gradient touches it. Fourth, spotting the failure that single-round demos conceal by running the loop three times. The only entry requirement is having shipped something that calls an LLM API.

Two loops with different clocks

The first half of the story is the agent finishing a task and recording what it learned in a playbook, a plain text file of lessons read back at the start of every request, so the next run starts ahead of the last. The second half is consolidation: a separate offline process that replays stored traces, decides which are stable and valuable enough to train on, and writes only that residue into parameters.

A 2026 memory survey names the pair the Accumulate-Internalize cycle. Accumulate explicitly, in memory and playbooks, where a lesson can be inspected and edited cheaply. Internalize the stable remainder into parameters, where it becomes fast, general and free at inference time. Put simply, the agent works all day from a notebook, and something else, later and apart, picks which pages are worth memorizing.

The nightly pipeline has three steps: replay, select, distill. Only the middle one involves judgment, and it is the one that matters most.

• The fast clock ticks at request time. Context and memory adapt per conversation, and the changes are reversible and cheap.
• The slow clock ticks at release time. Consolidation updates weights every few weeks, behind a gate and at real expense.

Good setups use both clocks and keep track of which lesson lives where. Bad ones either never advance the slow clock and pay ever-growing context rent, or plug the fast clock straight into the weights and invite poisoning early.

The best teacher is the agent itself, holding the playbook

Classic distillation trains a student to imitate a teacher's outputs. The catch is where the data comes from: the student learns from the teacher's polished trajectories and never sees the messy states its own errors create. Its first wrong turn lands somewhere the training set never visited.

On-policy distillation corrects the aim. Samples come from the student's own behavior, and a teacher grades or guides those tokens, so the correction signal falls on the mistake distribution the student really has. It is dense feedback on one's own errors, not a highlight reel of somebody else's wins.

What makes this practical is that the teacher need not be a larger model. It can be the same model with an unfair advantage: the playbook in context, retrieved lessons, and hindsight about an episode that has already ended. A research system called SEED works on exactly this seam. It converts finished agent trajectories into skills written in plain language, then pushes the guidance those skills hold into the policy itself, leaving no skill prompt and no memory lookup at inference. The lesson becomes the behavior.

Think of the teacher as the student wearing a backpack. Distillation is the process of removing the backpack without losing what was inside it.

A small version fits in an afternoon. Pick fifty tasks the agent already handles, freeze the playbook so it stops changing, and set up both sides:

# teacher: same model, unfair advantage
teach = Agent(system=playbook)
# student: same model, empty context
bare = Agent()

tries = [bare.run(t) for t in tasks]

Then have the teacher's judgment grade the student's own attempts, and train on that. This is the whole of on-policy distillation in a few lines:

graded = [teach.grade(x) for x in tries]
finetune(bare, graded)   # one round only
# now serve it with NO playbook

The run produces three numbers worth recording. In one trial on a fifty-task slice of a support queue in June, the bare student had 61% of answers accepted, the student with the playbook in context reached 79%, and the internalized model running without any playbook got 77%.

Sixteen of the eighteen points bought by the playbook held up, and each request dropped from 4,270 tokens to 910. The backpack comes off, and a lease turns into ownership. The first run makes the problem look solved.

A gradient is a commitment

The industry resisted weight updates for about three years, and the resistance was rational. Context can be diffed, audited and deleted in an afternoon. A weight update is opaque, entangled, and undone only by rolling back to a checkpoint, which discards everything else the model learned since that snapshot.

Catastrophic forgetting is the textbook form of the risk: regressions in capabilities nobody was measuring, found later by a customer. Safety alignment also needs re-verification after each update, since properties confirmed on the base model do not automatically carry over to a fine-tune.

A useful comparison is a feature branch reaching main, where recurrence, review, provenance and a rollback path all apply. The comparison breaks in one expensive place. Reverting in git is surgical. Rolling weights back is all or nothing, so a single bad merge forfeits whatever else landed after the checkpoint.

Four criteria decide what graduates:

• The lesson recurred across enough distinct episodes to look like a pattern and not an accident.
• It stayed in the playbook without ever being contradicted.
• It shows measured, gap-closing value against a baseline that was actually run.
• It has a clean provenance trail, so the traces that produced it can be named.

Anything failing these checks stays in context, where a mistake costs a token bill and not a checkpoint. As a rule, weights hold the residue and context holds the churn.

The four gates run in sequence, and the sequence is part of the design: a lesson that fails the first gate never consumes a review at the fourth. A gate that promotes more than a sliver of candidates is not a gate.

According to a June 2026 survey of self-improving agents, the decision sits in the harness, not the model. It is the harness that picks the traces worth turning into training signal, pulls supervision out of them, keeps the conditions under which they were valid, and judges whether a given experience is even suitable for parameter-side learning.

Once there are two routes, something above them has to choose which improvement process fires, at what moment, and within what budget, evidence bar and rollback terms. That controlling layer is the real deliverable. When picking, read the rollback column first, because routes that undo with a one-line deletion deserve about nine of every ten choices.

Round one works; round three breaks

A fresh result should slow everyone down. The paper "Rethinking Continual Experience Internalization" reports that a single round of distilling experience into weights works well, while repeating the self-evolving loop makes performance collapse.

The authors pin the collapse on three things: the way experience is represented, the way that representation shapes what the teacher supervises, and the distribution of trajectories that carries the transfer. A one-shot win was only the demo. Staying stable across rounds is the actual requirement, and nobody has solved it.

The earlier support-queue trial bent exactly where the paper says it would. Round two scored 74%, a little under the 77% of round one, which was easy to dismiss as noise. Round three landed at 58%, below the bare student the experiment began with, and the model had quietly worsened on two ticket categories that were absent from the eval set. The shape to remember is the gap between rounds one and three, since round one is the only one that reaches most write-ups.

There is a second boundary. Agentic RL means applying reinforcement learning to whole multi-step agent runs, with rewards from outcomes a machine can check. It pays off where checking is cheap, such as code, math and structured work. Most valuable real-world agent tasks give soft, late or exploitable signals. The Early Experience paper says this openly: plenty of real environments have no verifiable reward, and those that do frequently need lengthy, costly rollouts. Its answer is to learn from the agent's own early interactions with no external reward, treating the consequences of actions as the supervision.

Ownership also brings a bill. Each consolidated variant becomes its own artifact, needing a regression suite, a fresh safety evaluation and a serving slot of its own, and variants per tenant multiply the whole list by the number of customers.

A sober view for mid-2026: parameter-side learning has entered early production for coding and tool-use agents, whose rewards are checkable, and remains research-grade where rewards are soft. The context camp keeps a fair rebuttal. When a lesson fits in the playbook and retrieval functions, baking it into weights gains speed and polish while giving up auditability, and for many products that is a poor exchange.

Build the gate before it is needed

A frozen model remains a deliberate design choice, with one wrinkle. Current frontier models are themselves trained as agents, through reinforcement learning on long-horizon tool use in executable environments. The frozen artifact rented through an API is increasingly a graduate of the very loops described here, so the era of experience is arriving first as a change in how the underlying models get made.

A practical first step is instrumenting the accumulate loop well. Then try the inexpensive audit: open last month's playbook and tally entries that showed up in five or more separate episodes without being contradicted. One such pass turned forty-three entries into six candidates, and only one earned approval.

That lone entry is the first gradient. The rest is churn, which should stay in context, where an error costs tokens and not a checkpoint. The question to carry away: which lesson in the playbook has recurred often enough, without contradiction, to be signed off for release?