This article is published in English.
AutoSaddler: agents that rewrite their own harness
Let agents propose harness improvements under eval gates so self-modification stays measurable.
Use this as an operator-facing rebuild of the ideas in “AutoSaddler: Teaching AI Agents to Improve Their Own Harness”: clear stages, ordered code slots, and recovery notes that survive a handoff. Overview works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
The problem: agents fail for more than model reasons
For The problem: agents fail for more than model reasons, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.
AI Agent
│
┌──────────┴──────────┐
│ │
Model Harness
│
┌─────────────────┼─────────────────┐
│ │ │
Prompts Tools Middleware
│ │ │
└─────────────────┼─────────────────┘
│
Agent Loop Logic
│
▼
Execution
│
▼
Trace
From prompt engineering to harness engineering
For From prompt engineering to harness engineering, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.
Steering patches
For Steering patches, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary. For Steering patches, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
Capability patches
When working through Capability patches, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.
Execution traces become the training signal
When working through Execution traces become the training signal, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.
Task
↓
Model reasoning / response
↓
Tool selection
↓
Tool arguments
↓
Tool result
↓
Middleware
↓
Next model action
↓
Final answer
↓
Evaluation
Task failed
│
▼
Agent never inspected repository metadata
│
▼
Why?
│
▼
Tool existed but description didn't expose its purpose
│
▼
Diagnosis
│
▼
Update tool description
│
▼
Evaluate again
AutoSaddler’s optimisation loop
When working through AutoSaddler’s optimisation loop, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours. When working through AutoSaddler’s optimisation loop, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
Training Cases
│
▼
Run Agent
│
▼
Execution Traces
│
▼
┌───────────────────┐
│ Diagnosis-Patch │
│ │
│ Find root cause │
│ Propose patch │
└─────────┬─────────┘
│
▼
New Candidate
│
▼
Evaluate
│
┌─────────┴─────────┐
│ │
Improved Regressed
│ │
└─────────┬─────────┘
▼
Reflection
│
▼
Reusable Lessons
│
▼
EvoDAG
│
▼
Candidate Evolution
│
▼
Development Gate
│
▼
Best Generalizing Harness
1. Diagnosis-Patch: find the actual problem
- Diagnosis-Patch: find the actual problem works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
2. Reflection: learn from the result
- Reflection: learn from the result works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
Patch:
Add stronger instruction to inspect repository metadata.
Observed:
✓ Fixed cases A, B, C
✓ Existing cases remain stable
✗ Case D still failsLesson:
Instruction improves metadata discovery,
but does not address downstream tool selection.
3. Evolution: don’t forget what worked
- Evolution: don’t forget what worked works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
- Evolution: don’t forget what worked works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
Patch 1 → Patch 2 → Patch 3
Base
/ | \
/ | \
P1 P2 P3
│ / \ │
│ / \ │
P4 P5 P6 P7
\ /
\ /
P8
The most important safeguard: generalization
For The most important safeguard: generalization, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.
Training cases
│
▼
Generate candidate
│
▼
Evaluate
│
▼
Development split
│
▼
Does the improvement generalize?
│
┌──┴──┐
│ │
Yes No
│ │
▼ ▼
Keep Reject
Why durable execution matters
For Why durable execution matters, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.
Run
│
▼
Append-only Events
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Candidates Evaluations Sessions
│ │ │
└─────────────┼─────────────┘
▼
Snapshot
│
▼
Final Result
Reproducibility is treated as a first-class concern
For Reproducibility is treated as a first-class concern, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary. For Reproducibility is treated as a first-class concern, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
Candidate A
│
├── Prompt version X
├── Harness commit Y
├── Dataset revision Z
└── Model configuration M
V1 vs V2
When working through V1 vs V2, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.
V1
When working through V1, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.
V2
When working through V2, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours. When working through V2, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
AutoSaddler Core
│
Scenario Plugin
│
┌───────────────┼───────────────┐
▼ ▼ ▼
Harness Benchmark Evaluator
│ │ │
└───────────────┼───────────────┘
▼
Evidence Builder
│
▼
Optimizer Engine
Plugins make the idea extensible
Plugins make the idea extensible works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
AutoSaddler
│
┌───────────┼───────────┐
▼ ▼ ▼
Agent A Agent B Agent C
│ │ │
Plugin A Plugin B Plugin C
What makes AutoSaddler different?
What makes AutoSaddler different? works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
1. It optimizes the full harness
- It optimizes the full harness works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
- It optimizes the full harness works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
2. It diagnoses before modifying
For 2. It diagnoses before modifying, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.
3. It uses structured interventions
For 3. It uses structured interventions, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.
4. It learns from regressions
For 4. It learns from regressions, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary. For 4. It learns from regressions, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
5. It optimizes for generalization
When working through 5. It optimizes for generalization, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.
6. It retains evolutionary history
When working through 6. It retains evolutionary history, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.
7. It is durable
When working through 7. It is durable, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours. When working through 7. It is durable, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
What could come next?
What could come next? works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
Continuous harness optimization
Continuous harness optimization works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.
Automatic tool evolution
Automatic tool evolution works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.
Middleware optimization
Cross-agent learning
Cost-aware optimization
Quality
+
Reliability
+
Latency
+
Token Cost
+
Tool Cost