Loop Engineering: Four Nested Loops for Self-Improving AI Agents

For roughly two years, working with a model has meant taking turns. An engineer writes a prompt, reads what comes back, and hand-feeds the next piece of context. That routine carries a hidden cost, a kind of orchestration tax, because the speed of the whole process is capped by how fast a person can review. Getting past it means a change of focus: stop holding the tool and start designing the system that holds the tool.

Three ideas frame that change.

• Influence moves up a level. The role shifts from authoring exact instructions to designing loops that repeat, and eventually to something like a principal's seat. The human supplies the goal and the stopping condition, and the agent owns every step of the route between them.
• Autonomous loops have a recognizable anatomy. They discover work, hand pieces to sub-agents, check results with independent tests, and write their state down, which takes the person out of the middle of the triage, fix and verify cycle.
• Goal engineering sits at the top. Here the inner workings of the loop recede. AlphaEvolve, which beat matrix-multiplication records that had stood since 1969, and Amazon's Kiro, which turns high-level objectives into requirements and specs, show the pattern: get the scoring function right and the machine finds the answer.

To design a loop around an agent, you need a firm mental model of the agent inside it. So the sensible build order starts with a plain tool-calling worker, adds a second model that scores its output (so the system stops grading its own homework), connects the two in a loop, and writes the gains to disk, since models forget and a repository doesn't. After that come defenses against an agent gaming the metric, the sort of cheating reported when OpenAI's o3 was tested by METR, then a meter that tracks cost in an environment that may be token-rich or token-poor, and finally the climb from a 70-line script to something that survives production.

What follows is a working implementation of that whole idea, using an insurance underwriting pipeline as the testbed.

Why this is an architecture problem

Going from a model that answers questions to an agent that reliably takes actions is not a prompt-writing exercise. It is a design problem. Most tutorials show one agent calling one tool. Real systems need fault tolerance, self-correction, spending limits, tamper detection and steady improvement, all at once.

The approach here is called loop engineering: nest several control loops, each with a distinct job, so that the whole system holds up better than any single loop could.

Insurance underwriting is a convenient domain. Decisions come from a small fixed set (accept, modify, decline, refer), the rules are explicit, and historical data supplies right and wrong answers. The same patterns carry over to code review, document analysis, support triage and medical record processing.

Four loops, one inside another

Ordinary software tames complexity with layers of abstraction. Loop engineering tames the unreliability of an agent with layers of feedback:

┌──────────────────────────────────────────────────────┐
│  Loop 4 — Hill Climbing (self-improvement over time) │
│  ┌────────────────────────────────────────────────┐  │
│  │  Loop 3 — Event Queue Poller (orchestration)   │  │
│  │  ┌──────────────────────────────────────────┐  │  │
│  │  │  Loop 2 — Verification & Retry           │  │  │
│  │  │  ┌────────────────────────────────────┐  │  │  │
│  │  │  │  Loop 1 — ReAct Agent              │  │  │  │
│  │  │  │  (think → act → observe → think)   │  │  │  │
│  │  │  └────────────────────────────────────┘  │  │  │
│  │  └──────────────────────────────────────────┘  │  │
│  └────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────┘

Each loop has one responsibility, one failure it defends against and one feedback signal it reacts to. The discipline that matters is to keep concerns from leaking across loops.

The domain

Before the architecture, here is the data. An insurance application is a flat record:

{
  "id": "APP-001",
  "applicant_age": 34,
  "state": "FL",
  "property_type": "residential",
  "coverage_amount": 450000,
  "claim_history_5yr": 1,
  "credit_score_tier": "good",
  "flood_zone": true,
  "has_flood_rider": false,
  "business_use": false
}

Each of the 20 sample applications also has a ground-truth label:

"APP-001": {
  "split": "train",
  "expected_decision": "modify",
  "expected_flags": ["flood_rider_required", "windstorm_exclusion"]
}

A decision is one of four things:

• accept: standard risk, approve as it stands
• modify: approve with conditions or endorsements
• decline: the risk is beyond the acceptable threshold
• refer: a senior underwriter must look at it

A small knowledge base holds five hidden underwriting rules covering flood zones, claims history, credit score, coverage ceilings and state-specific restrictions. The agent isn't told them in its prompt. It has to find and apply them through tool calls. Sixteen applications form the training set and four are held back for testing.

Loop 1: a ReAct agent

Defends against: a model that invents decisions without evidence. Feedback signal: the tool-call graph, meaning whether the agent consulted everything relevant before it decided.

ReAct, short for reason plus act, is the base pattern. The agent thinks about what it needs, calls a tool, reads the result, thinks again, and repeats until it commits:

System Prompt
     │
     ▼
[Think] → What information do I need?
     │
     ▼
[Act]   → Call tool (lookup_application, retrieve_rules, ...)
     │
     ▼
[Observe] → Tool returns result
     │
     ▼
[Think] → What does this mean? What next?
     │
    ... (repeat until decision is made)
     │
     ▼
[Act]   → draft_decision (terminal tool call)

In LangGraph this becomes a state graph with two nodes and a conditional edge. The state carries the application ID, the message list, the decision fields, the lessons, the running spend and a step counter:

class UnderwritingState(TypedDict):
    application_id: str
    messages: Annotated[list[AnyMessage], add_messages]  # reducer accumulates
    decision: str
    rationale: str
    flags: list[str]
    recommended_premium_adj: float
    lessons: list[str]
    spend: float
    _steps: int

One detail deserves attention. The add_messages annotation makes LangGraph append to the message list at each step instead of replacing it, so the agent keeps the whole conversation across every tool call for one application.

The builder takes the system prompt and the shared cost meter and binds the tools to the model:

def build_loop1(system_prompt: str, meter: CostMeter):
    model = ChatAnthropic(model="claude-sonnet-4-6").bind_tools(TOOLS)

Inside it sit the agent node, the tools node, the routing function and the graph assembly:

    def agent_node(state: UnderwritingState) -> dict:
        prompt = system_prompt
        if state.get("lessons"):
            prompt += "\n\nUnderwriting lessons:\n" + "\n".join(
                f"- {l}" for l in state["lessons"]
            )
        msgs = [SystemMessage(content=prompt)] + list(state["messages"])
        response = model.invoke(msgs)
        meter.record_from_message(response)          # track spend per call
        return {
            "messages": [response],
            "_steps": state.get("_steps", 0) + 1,
            "spend": meter.spent,
        }

    def tools_node(state: UnderwritingState) -> dict:
        last = state["messages"][-1]
        tool_results, updates = _dispatch_tools(last.tool_calls)
        return {"messages": tool_results, **updates}  # updates captures draft_decision output

    def should_continue(state) -> Literal["tools", "__end__"]:
        if state.get("_steps", 0) >= MAX_STEPS:   # hard cap at 8 steps
            return END
        if getattr(state["messages"][-1], "tool_calls", None):
            return "tools"
        return END

    g = StateGraph(UnderwritingState)
    g.add_node("agent", agent_node)
    g.add_node("tools", tools_node)
    g.set_entry_point("agent")
    g.add_conditional_edges("agent", should_continue, {"tools": "tools", END: END})
    g.add_edge("tools", "agent")
    return g.compile()

The four tools

The agent gets exactly four tools, enough to cover everything it needs:

@tool
def lookup_application(app_id: str) -> dict:
    """Return the full application record for the given application ID."""
    # reads from applications.json — deterministic, no LLM call
    ...

@tool
def retrieve_rules(risk_factor: str) -> list[str]:
    """Return underwriting rules matching the given risk factor keyword.
    Use terms like: flood, claims, credit, coverage, state, business."""
    # keyword match against rules_kb.json - forces explicit rule lookup
    ...

@tool
def fetch_risk_signal(signal_type: str) -> dict:
    # simulated external data pull (credit bureaus, flood maps, etc.)
    ...

@tool
def draft_decision(
    decision: Literal["accept", "modify", "decline", "refer"],
    rationale: str,
    flags: list[str],
    recommended_premium_adj: float,
) -> dict:
    """Finalize the underwriting decision with structured output."""
    # terminal action - structured schema forces the agent to commit explicitly
    ...

lookup_application only reads data. retrieve_rules does a keyword search over a fixed rule set, which forces an explicit look-up. fetch_risk_signal simulates pulling outside data such as credit bureaus or flood maps. draft_decision is the terminal action, and it matters most. It makes the agent commit to a typed decision with a rationale and specific flags, so the structured result comes from the tool call itself and nobody has to parse free text.

Why the step cap matters

MAX_STEPS = 8 stops endless tool chains. A confused model might call the rules lookup for "flood" twenty times, hunting for an answer that isn't there, and burn tokens and time doing it. When the step count reaches the cap, the state gets the decision ran_out_of_steps, a sentinel Loop 2 can recognize and retry.

Loop 2: verify and retry

Defends against: a single wrong answer from a bad line of reasoning. Feedback signal: the deterministic judge, is_correct(), with no model involved.

Loop 1 can fail. The agent may misread a flag, skip a rule or hit the step cap. Loop 2 catches those failures and gives the agent another try with targeted feedback:

def run_loop2(
    app_id: str,
    system_prompt: str,
    meter: CostMeter,
    max_retries: int = 3,
) -> tuple[dict, bool]:
    state = run_loop1(app_id, system_prompt, meter)
    for attempt in range(max_retries):
        decision = state.get("decision", "ran_out_of_steps")
        if decision == "refer":
            _write_human_queue(app_id, state)   # human-in-the-loop path
            return state, True
        if is_correct(decision, app_id):         # deterministic check
            return state, True
        if attempt >= max_retries - 1:
            return state, False                  # exhausted retries
        # construct targeted feedback for next attempt
        expected = load_historical_decisions()[app_id]["expected_decision"]
        feedback = HumanMessage(content=(
            f"Your decision was '{decision}' but this application requires '{expected}'. "
            f"Review the risk factors carefully and call draft_decision again."
        ))
        extra_messages = list(state.get("messages", [])) + [feedback]
        state = run_loop1(app_id, system_prompt, meter, extra_messages=extra_messages)
    return state, False

A few design choices are worth spelling out.

The judge is deterministic. is_correct() reads the historical decisions file and compares strings. It never calls a model. Using a second model as judge would add a second source of unreliability, and here the ground truth simply is the ground truth.

Referral is a handled outcome. When the agent says refer, Loop 2 writes the case to human_queue.json and reports success. Escalating to a person counts as a valid end state and not as something to retry.

Feedback names the target. Instead of "try again", the message tells the agent which decision category is correct and asks it to re-examine the risk factors. That steers the next run toward the right answer, where blind re-sampling would wander.

History carries over. The extra messages include the entire earlier conversation plus the feedback, so on its next run the agent sees everything it already attempted and is less likely to repeat the same wrong path.

Loop 3: the event queue poller

Defends against: tight coupling between whoever produces work and whoever consumes it. Feedback signal: the contents of pending_applications.json, a plain file acting as the queue.

This loop is the orchestrator. It pulls from the pending queue, drives one application through Loops 1 and 2, updates the run state and, now and then, triggers Loop 4:

def run_event_loop(
    system_prompt: str,
    meter: CostMeter,
    pending_path: Path = DEFAULT_PENDING,
    state_path: Path = DEFAULT_STATE,
    once: bool = False,
) -> None:
    run_state = load_run_state(state_path)
    verify_judge_checksum(run_state.get("judge_checksum", ""))   # tamper check
    total = len(json.loads(pending_path.read_text(encoding="utf-8-sig"))
                if pending_path.exists() else [])
    processed = 0
    while True:
        pending = (json.loads(pending_path.read_text(encoding="utf-8-sig"))
                   if pending_path.exists() else [])
        if not pending:
            if once:
                print("[loop3] queue empty - done")
                break
            print("[loop3] queue empty - sleeping...")
            time.sleep(POLL_INTERVAL)
            continue
        app_id = pending[0]
        processed += 1
        print(f"[loop3] {processed}/{total}  {app_id} ...", flush=True)
        try:
            result_state, passed = run_loop2(app_id, prompt_with_lessons, meter)
        except BudgetExhaustedError:
            raise                                # budget exhaustion is fatal
        except Exception as exc:
            print(f"[loop3] {app_id} ERROR: {exc!r} - skipping")
            # remove from queue and continue - one bad app cannot block the rest
            remaining = [x for x in pending if x != app_id]
            pending_path.write_text(json.dumps(remaining), encoding="utf-8")
            continue
        # ... update state, distill lesson, trigger hill-climbing

Why a file for a queue

The queue is a JSON file rather than a Redis stream or an SQS queue, on purpose, because for local development it is easy to inspect (open the file and see what's pending), easy to resume (restart after a crash and the queue is intact) and decoupled (whatever fills it needs no knowledge of the consumer). A production system would swap in a proper message broker, and the interface contract, a list of IDs, would not change.

Tamper detection

Before processing anything, Loop 3 verifies the ground truth:

verify_judge_checksum(run_state.get("judge_checksum", ""))

The check works like this:

def compute_judge_checksum() -> str:
    content = _HIST.read_bytes()           # historical_decisions.json
    return hashlib.sha256(content).hexdigest()


def verify_judge_checksum(expected: str) -> None:
    actual = compute_judge_checksum()
    if actual != expected:
        raise RuntimeError(
            f"Judge checksum mismatch - historical_decisions.json was tampered with."
        )

A SHA-256 hash of the historical decisions file is stored in the run state at initialization and re-checked each time the event loop starts. If anyone edits the ground truth to flatter the agent, the system refuses to run. That closes a quiet loophole in self-improving systems: the cheapest way to raise a score is to change the scorer.

Skipping versus crashing

The try/except in Loop 3 draws a deliberate line. Budget exhaustion is re-raised, which stops everything. Any other exception is logged, the offending application is removed from the queue, and the next one is processed. A single malformed application therefore can't freeze the pipeline, while running out of money is fatal by design, because the system shouldn't keep going with nothing left to spend.

Loop 4: hill climbing

Defends against: a fixed prompt that never learns from experience. Feedback signal: training accuracy together with the audit verdict.

Every eighth decision (HILL_CLIMB_EVERY = 8), Loop 3 hands over to Loop 4, which runs a cycle of reflect, mutate, evaluate and gate to improve the system prompt automatically:

seed prompt
     │
     ▼
evaluate on training set → score, failures
     │
     ▼
┌────────────────────────────────┐
│  for each round:               │
│                                │
│  reflect(current, failures)    │  ← LLM analyzes failure patterns
│       │                        │
│       ▼                        │
│  candidate_prompt              │
│       │                        │
│       ▼                        │
│  evaluate(candidate, train)    │  ← full eval run
│       │                        │
│       ▼                        │
│  audit_prompt(candidate)       │  ← anti-cheating check
│       │                        │
│       ▼                        │
│  if score > best AND clean:    │
│      best = candidate          │  ← keep
│  else:                         │
│      discard                   │  ← revert
└────────────────────────────────┘
     │
     ▼
write best_prompt.txt

Reflect

REFLECT_TEMPLATE = """You are improving the system prompt of an insurance underwriting agent.

CURRENT PROMPT:
{prompt}
The agent got these decisions WRONG on training applications:
{failures}
Look for PATTERNS in the failures. Infer GENERAL underwriting rules that would fix them.
Do NOT memorize specific application IDs or applicant details.
Write an improved system prompt. Reply with ONLY the new prompt text."""


def reflect(current_prompt: str, failures: list[dict]) -> str:
    client = anthropic.Anthropic()
    failure_text = "\n".join(
        f"- APP {f['app_id']}: agent said '{f['got']}', expected '{f['expected']}'. "
        f"Application data: {f['application']}"
        for f in failures
    )
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1500,
        messages=[{"role": "user", "content": REFLECT_TEMPLATE.format(
            prompt=current_prompt,
            failures=failure_text,
        )}],
    )
    return response.content[0].text.strip()

The instruction not to memorize specific application IDs or applicant details carries real weight. Without it, the model could write a prompt such as "for APP-003, always decline", which would score perfectly on training data and fail completely on cases it hasn't seen. The new prompt has to generalize from patterns in the failures, not from individual cases.

Evaluate

def evaluate_prompt(system_prompt: str, split: str = "train", meter=None):
    tasks = load_tasks(split)
    failures = []
    for task in tasks:
        state = run_loop1(task["id"], system_prompt, meter)
        decision = state.get("decision", "ran_out_of_steps")
        if not is_correct(decision, task["id"]):
            failures.append({
                "app_id": task["id"],
                "got": decision,
                "expected": task["expected_decision"],
                "application": task["application"],
            })
    score = (len(tasks) - len(failures)) / len(tasks)
    return score, failures

Every candidate prompt runs through Loop 1 on all 16 training applications. That is expensive, 16 model runs per round and at least 80 over five rounds, but the signal is honest: a prompt must work across the full training distribution and not on a sample.

Gate

def improve(seed_prompt: str, lessons: list[str], meter: CostMeter, rounds: int = 5) -> str:
    best = with_lessons(seed_prompt, lessons)
    best_score, failures = evaluate_prompt(best, split="train", meter=meter)
    for r in range(1, rounds + 1):
        candidate = reflect(best, failures)
        score, cand_failures = evaluate_prompt(candidate, split="train", meter=meter)
        verdict = audit_prompt(candidate)        # anti-cheat check
        if score > best_score and verdict == "VERDICT: CLEAN":
            best, best_score, failures = candidate, score, cand_failures
            BEST_PROMPT_PATH.write_text(best, encoding="utf-8")
            print(f"round {r}: {score:.0%} KEPT  ({len(cand_failures)} failures)")
        else:
            reason = "cheating" if verdict != "VERDICT: CLEAN" else f"{score:.0%} <= {best_score:.0%}"
            print(f"round {r}: {score:.0%} REVERTED ({reason})")
    return best

Two conditions must both hold before a candidate is accepted. First, score > best_score, so it must strictly beat the current best. Ties are rejected, which prevents prompt drift. Second, the auditor must return VERDICT: CLEAN, confirming the candidate contains no hard-coded answers.

The prompt auditor

AUDIT_SYSTEM = (
    "You are a security auditor for AI systems. "
    "Check whether the prompt contains any hardcoded application IDs (e.g. APP-001) "
    "or hardcoded decisions for specific applications. "
    "These would constitute cheating — memorizing answers instead of learning rules. "
    "Reply with exactly one of:\n"
    "VERDICT: CLEAN\n"
    "VERDICT: CHEATING\n"
    "Followed by a one-sentence explanation."
)


def audit_prompt(evolved_prompt: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=200,
        system=AUDIT_SYSTEM,
        messages=[{"role": "user", "content": f"PROMPT TO AUDIT:\n{evolved_prompt}"}],
    )
    return response.content[0].text.strip().split("\n")[0]   # first line only

This is an LLM-as-judge setup, but restricted to one binary structural question: does the prompt contain specific IDs or memorized answers? That is the kind of classification where model judges are dependable. The auditor never rates quality, style or correctness. It only looks for one pattern.

Lesson memory

Apart from hill climbing, the system keeps a lessons list that persists between runs. When an application still fails after every retry, it calls distill_lesson():

DISTILL_PROMPT = """An insurance underwriting agent made a wrong decision.

Application details: {application}
Agent decided: {got}
Correct decision: {expected}
Write ONE short, general underwriting rule that would prevent this mistake.
Do NOT reference this specific application ID or any applicant names/details.
Reply with only the rule, as a single sentence."""


def distill_lesson(application: dict, got: str, expected: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=150,
        messages=[{"role": "user", "content": DISTILL_PROMPT.format(...)}],
    )
    return response.content[0].text.strip()

Each lesson is then added to every later Loop 1 run through with_lessons():

def with_lessons(base_prompt: str, lessons: list[str]) -> str:
    if not lessons:
        return base_prompt
    lessons_text = "\n".join(f"- {l}" for l in lessons)
    return base_prompt + f"\n\nUnderwriting lessons learned:\n{lessons_text}"

This is few-shot learning with no gradient descent. The system learns from mistakes at runtime by collecting compressed knowledge as natural-language rules. A function in the style of the CL-BENCH protocol checks whether those lessons actually help:

def measure_gain(base_prompt: str, lessons: list[str], split: str = "test", meter=None) -> float:
    stateless_score, _ = evaluate_prompt(base_prompt, split=split, meter=meter)
    augmented = with_lessons(base_prompt, lessons)
    stateful_score, _ = evaluate_prompt(augmented, split=split, meter=meter)
    return stateful_score - stateless_score   # positive = lessons help

Budget control with a cost meter

Every model call passes through one shared CostMeter, handed down from Loop 4 to Loop 3 to Loop 2 to Loop 1, so all spending accumulates in a single place:

BUDGET_USD = 2.00
INPUT_PRICE_PER_TOKEN  = 3.00  / 1_000_000
OUTPUT_PRICE_PER_TOKEN = 15.00 / 1_000_000


class CostMeter:
    def record(self, input_tokens: int, output_tokens: int) -> None:
        if self.spent >= self._budget:              # check BEFORE adding
            raise BudgetExhaustedError(
                f"Budget ${self._budget:.2f} exhausted at ${self.spent:.4f}"
            )
        self.spent += input_tokens * INPUT_PRICE_PER_TOKEN \
                    + output_tokens * OUTPUT_PRICE_PER_TOKEN

The ordering is the point: check the budget before adding the cost. That way the error fires the moment the limit is reached, instead of after an overspend has been booked. BudgetExhaustedError travels up through the loop hierarchy, and Loop 3 re-raises it rather than swallowing it, so exhausted budget always ends the whole run.

Run state with idempotent setup

run_state.json records everything that has happened: processed applications, passed and failed lists, accumulated lessons, decisions since the last hill-climb and the judge checksum. Its initialization is defensive:

def _init_run_state() -> dict:
    path = _STATE_DIR / "run_state.json"
    state = {}
    if path.exists():
        try:
            content = path.read_text(encoding="utf-8-sig").strip()  # handles Windows BOM
            if content:
                state = json.loads(content)
        except (json.JSONDecodeError, UnicodeDecodeError):
            pass                                 # corrupt file → start fresh
    if not state:
        state = dict(_DEFAULT_RUN_STATE)
    state["judge_checksum"] = compute_judge_checksum()   # always refresh
    path.write_text(json.dumps(state, indent=2), encoding="utf-8")
    return state

Three resilience habits show up here. Reading with utf-8-sig silently removes Windows byte-order marks. An empty or corrupt file falls back to default state instead of crashing. And the judge checksum is recomputed on every start, never trusted from a previous run.

Three ways to run it

There are three ways in, and each one exercises only part of the stack.

Single mode runs Loop 1 alone, which is handy for debugging one application and watching which tools the agent calls and why:

python -m src.main --mode single --app-id APP-001

Event-loop mode runs Loops 1, 2 and 3 over every pending application and may trigger Loop 4 after every eight decisions. The --once flag drains the queue and exits instead of polling forever:

python -m src.main --mode event-loop --once

Improve mode runs Loop 4 directly, with five rounds of hill climbing on the training split. It then scores the best prompt found against the held-out test split and prints the gain:

python -m src.main --mode improve --rounds 5

One application, start to finish

This is the complete data flow for a single application moving through the system:

pending_applications.json
          │
          │  [Loop 3 reads APP-009]
          ▼
    run_loop2("APP-009", prompt, meter)
          │
          │  [Loop 2 calls Loop 1]
          ▼
    run_loop1("APP-009", prompt, meter)
          │
          │  [LangGraph StateGraph]
          ▼
    agent_node → ChatAnthropic.invoke([SystemMsg, HumanMsg])
          │
          │  response: tool_call(lookup_application, {"app_id": "APP-009"})
          ▼
    tools_node → lookup_application.invoke({"app_id": "APP-009"})
          │
          │  returns: {id: APP-009, state: TX, coverage: 1200000, ...}
          ▼
    agent_node → ChatAnthropic.invoke([..., ToolMessage])
          │
          │  response: tool_call(retrieve_rules, {"risk_factor": "coverage"})
          ▼
    tools_node → retrieve_rules.invoke({"risk_factor": "coverage"})
          │
          │  returns: ["Coverage > $1M requires mandatory refer. Flag: high_coverage"]
          ▼
    agent_node → ChatAnthropic.invoke([..., ToolMessage])
          │
          │  response: tool_call(draft_decision, {decision: "refer", ...})
          ▼
    tools_node → draft_decision.invoke({...})
          │
          │  updates state: {decision: "refer", flags: ["high_coverage"], ...}
          ▼
    should_continue → END (no more tool calls)
          │
          ▼
    Loop 2: decision == "refer" → write human_queue.json → return (state, True)
          │
          ▼
    Loop 3: passed=True → update run_state.json → remove APP-009 from queue
          │
          │  [decisions_since_last_improvement becomes 9 >= 8]
          ▼
    Loop 4: improve(seed_prompt, lessons, meter, rounds=5)
          │
          │  reflect → evaluate → audit → gate → write best_prompt.txt
          ▼
    Loop 3: continue with APP-010 using new best_prompt

Lessons from building it

1. Keep the oracle apart from the agent. The judge (is_correct, evaluate_prompt) shares no code with the agent (run_loop1). The judge reads static files and the agent calls a model. Let the model grade its own output and you get a yes-man that never fails.

2. Make evaluation deterministic. Per-application correctness, training accuracy and test accuracy are all deterministic for the same inputs. The only randomness lives in the model's replies, so any score can be reproduced from the prompt and the seed.

3. Use a final-answer tool for structured output. A draft_decision tool beats parsing prose. Its schema fixes the vocabulary (accept | modify | decline | refer), the required fields (flags, rationale, recommended_premium_adj) and their types, so the agent has to be explicit.

4. Anti-cheating isn't optional in self-improving systems. Once a system can edit its own instructions, it will find shortcuts, and the obvious one is memorizing training answers. audit_prompt() catches this automatically. Without it, hill climbing dependably yields prompts with near-perfect training accuracy and near-random test accuracy.

5. Byte-order marks and encodings are production bugs. If you build on Windows or receive files from Windows users, encoding="utf-8-sig" on every read is mandatory. The BOM written by Windows Notepad triggers json.JSONDecodeError: Unexpected UTF-8 BOM, which is hard to diagnose because the file looks perfectly fine in an editor.

6. Check the budget before spending, not after.

# WRONG: allows one overspend before raising
self.spent += cost
if self.spent > self._budget:
    raise BudgetExhaustedError(...)
# CORRECT: raises before the overspend registers
if self.spent >= self._budget:
    raise BudgetExhaustedError(...)
self.spent += cost

The order matters when several model calls happen in quick succession. Add first and check later, and the budget can be exceeded by more than one call's worth before the error appears.

7. Flush progress lines in long loops.

print(f"[loop3] {processed}/{total}  {app_id} ...", flush=True)

On Windows stdout is buffered unless told otherwise. Skip flush=True and the "processing APP-009" line shows up after the API call returns, which defeats the point of a progress indicator. Always flush lines that come before blocking I/O.

8. Keep the hill-climbing budget separate from the event loop's. Budget exhaustion during hill climbing must be caught at that call site and not allowed to crash the event loop. After eight application evaluations have used most of a $2 budget, hill climbing has little room left. The event loop catches BudgetExhaustedError from hill climbing specifically and carries on with the remaining applications:

try:
    system_prompt = run_hill_climbing(system_prompt, lessons, meter)
    run_state["decisions_since_last_improvement"] = 0
except BudgetExhaustedError:
    print(f"[loop3] hill-climbing skipped — budget exhausted at ${meter.spent:.4f}")
    run_state["decisions_since_last_improvement"] = 0

Applying the pattern elsewhere

The four-loop design isn't tied to insurance. To adapt it, swap these parts:

• Loop 1 tools. Here: lookup, rules, signal, decide. Yours: retrieval and action tools for your domain.
• Ground truth. Here: the historical decisions file. Yours: your own labeled dataset.
• Judge. Here: string comparison. Yours: a correctness test specific to your task.
• Lessons. Here: underwriting rules. Yours: error patterns from your domain.
• Hill-climbing prompt. Here: the underwriting system prompt. Yours: your agent's system prompt.
• Budget. Here: $2.00. Yours: whatever each run should cost.

One constraint is firm: the judge must be deterministic and independent of the agent. If no deterministic correctness test can be defined for your task, the system can't improve itself reliably.

Trying it out

Setup and the four commands:

# 1. Install
cd underwriting_loop
pip install -e ".[dev]"
export ANTHROPIC_API_KEY=sk-ant-...
# 2. Run a single application (debug mode - cheapest)
python -m src.main --mode single --app-id APP-001
# 3. Drain the pending queue with event loop
python -m src.main --mode event-loop --once
# 4. Run standalone hill-climbing (5 rounds)
python -m src.main --mode improve --rounds 5
# 5. Run the full offline test suite (no API key needed)
pytest tests/ -v

Hill-climbing output looks like this:

seed score: 44%  (9 failures)
round 1: 56% KEPT  (7 failures)
round 2: 56% REVERTED (score 56% <= best 56%)
round 3: 69% KEPT  (5 failures)
round 4: 75% KEPT  (4 failures)
round 5: 69% REVERTED (score 69% <= best 75%)
Final test evaluation...
spend:              $1.8342
test score:         75%
gain (vs baseline): +25%

The seed prompt begins at 44% on training. Five rounds later the winning prompt scores 75% on the held-out test set, which is a 25-point jump from automatic prompt refinement with no manual prompt engineering.

What the loops add up to

Loop engineering isn't a framework. It is a way of splitting agent reliability into layers, each with a single responsibility and a single feedback mechanism. The four loops answer four separate failure modes:

• Loop 1 handles grounding: the agent must consult evidence before it decides.
• Loop 2 handles single-run failures: one bad reasoning path shouldn't permanently fail an application.
• Loop 3 handles throughput and resilience: one bad application shouldn't block the pipeline.
• Loop 4 answers recurring mistakes by rewriting the prompt itself from the failures that piled up.

Stack them, add a deterministic judge, protect everything with budget control and tamper detection, and the system gets measurably better over time with no human intervention.

What you gain

Adopt this design with open eyes. The sharpest caution from the wider writing on agent loops is Addy Osmani's "cognitive surrender": giving the loop full control and ceasing to read its output. The advantages are still real and measurable.

1. Self-improvement without a human. Hill climbing lifted training accuracy from 44% to 75% with no manual prompt work, and the system extracted the right underwriting rules from failure patterns by itself.
2. Fault isolation between layers. One malformed application doesn't bring the pipeline down. Loop 3 catches per-application errors, logs them and continues. BudgetExhaustedError is the only exception that climbs all the way up, and that is intentional.
3. Progress you can measure and audit. measure_gain() produces a concrete figure, stateful with lessons against stateless without, on the held-out split. The improvement can be quoted to a stakeholder as a percentage and a dollar cost.
4. Tamper-proof evaluation. The SHA-256 checksum on the ground truth shuts the commonest silent failure of self-improving systems, someone editing the answers to make scores look better. If the checksums don't match, nothing runs.
5. Knowledge that accumulates. Lessons distilled from failures persist across runs, so every new Loop 1 invocation starts smarter than the last and doesn't relearn from scratch.
6. Producer and consumer stay decoupled. With a file queue, whatever produces the pending list is separate from the evaluation pipeline. The queue can be paused, resumed, examined or replayed while the agent code stays untouched.
7. Human escalation is a first-class state. refer decisions go to human_queue.json and count as passed. The loop doesn't retry something a person should decide.
8. Retries are targeted. The feedback after a miss says what went wrong, along the lines of "you said modify but the right answer is refer, re-examine the risk factors". That converges faster than random resampling.

What it costs

1. Tokens multiply fast. Five hill-climbing rounds over 16 training applications need at least 80 Loop 1 runs, plus reflection and audit calls. One --mode improve run spent $1.83 of a $2.00 budget. Hitting the budget isn't an edge case. It is the expected way for a run to end.
2. Latency is real. One application through the full stack can take 30 to 60 seconds, and 16 applications in event-loop mode take minutes. That's fine for batch work and disqualifying for interactive use.
3. Non-determinism hides failures. Loop 1 uses a live model, so the same prompt on the same application can yield different tool sequences. A failure may not reproduce, and meaningful accuracy numbers need several evaluation runs.
4. Setup is heavy. Before the first run you need the run state, the pending queue, the historical decisions with checksums, a best-prompt file and the lesson list. That is an order of magnitude more infrastructure than a single agent.invoke() call.
5. Overfitting never goes away. Without the audit guard, hill climbing eventually produces a prompt that memorizes training answers, scoring perfectly on training and near chance on test. The audit is the only thing between self-improvement and self-cheating.
6. Cognitive surrender is a genuine failure mode. The more reliable the loop gets, the less anyone reads its output, and when you stop reading you stop noticing that the loop has begun to fail in a new way. Unattended loops need outside monitoring, since a loop can't audit itself for semantic correctness.
7. Debugging spans the whole stack. A wrong decision might come from a bad reflection in Loop 4, a missed rubric rule in Loop 2, a tool that returned nothing in Loop 1 or a corrupt run state in Loop 3. Tracing a failure through four nested loops is harder than tracing one in a single function.

Should you build a loop at all?

A review of public loop setups gives clear advice: loop tasks that are repetitive, reviewable and high-value, with a built-in checker, and for everything else simply prompt once. It reduces to two questions.

1. Does the work repeat, or need many steps nobody can predict? If a single prompt would do, a loop adds overhead and no architecture.
2. Can the agent tell for itself whether it finished? Without a test, an expected format or reference answers, nothing ends the loop and it wanders.

Both answers must be yes. If either is no, skip the loop.

Good fits

Repetitive, high-volume work. Sixteen insurance applications, 500 code files, 1,000 support tickets. The loop recovers its setup cost when the task runs many times and not once.

A deterministic done-check exists. Ground-truth labels, a test suite, a schema validator or a rubric of binary pass/fail rules. The judge should never call a model, because a model-based judge brings its own unreliability into the mix.

The agent can verify its own work. Anthropic's Agent SDK calls this the most underrated step. Here is_correct() provides it, and in a coding agent the test suite does. If the agent can't check its output, nothing closes the loop.

Labeled data or reusable success criteria. Hill climbing needs a training and test split with known answers. Without one, you can reflect on failures but can't tell whether the reflection helped.

Nobody watches each run. Think overnight underwriting batches, scheduled audits or document pipelines, where no person sits inside the inner loop and this design is meant to operate. If someone can review each output right away, direct prompting is quicker.

The cost per evaluation run is acceptable. One hill-climbing round is priced like a full pass over every training application. If that's unaffordable, Loop 4 is too, so run it less often or on a smaller training set.

You need numbers. measure_gain() exists because "the system got better" has to be quantified. If stakeholders want figures, loop engineering delivers them. If "looks better" is enough, it's overkill.

Poor fits

A single prompt handles it. Reformatting a CSV, pulling a date from a document, summarizing a paragraph. These have predictable step counts, so write a script or a one-shot prompt. A loop here is an architecture tax on a simple problem.

"Done" is pure taste. "Make this email more professional" has no binary checker, and a second model judging aesthetics adds variance without closing anything. Fix the goal first, for example under 100 words, professional tone, no exclamation marks, and then it can be checked.

Actions are irreversible without review. Sending emails, merging pull requests, deleting records, moving money. Scope permissions tightly and require a pause before anything gets deleted, sent or paid for. Loops able to take irreversible actions without gates are failures waiting to happen.

The budget can't cover evaluation. If $2.00 is tight, hill climbing eats it before all training applications are processed. The system handled that with BudgetExhaustedError, but for a budget-constrained setup the real answer is a smaller training set, fewer rounds, or no Loop 4 until you can pay for it.

There's no state persistence. If you can't write the run state, lessons and queue to disk or a database, you can't resume after a crash, accumulate lessons across sessions or protect against running out of budget mid-run. Stateless environments need another architecture.

You're still exploring. Calling the agent once with good tooling shows you where the agent goes wrong, while nested loops only show where the whole system goes wrong. Probe the single agent first, and add layers once its own failure modes are familiar.

A last rule of thumb comes from Rhys Sullivan, who warns against two extremes: "tokenmaxxing" loops that run just to inflate spend, and naive "promptchud" one-liners. A loop justifies itself when it reproduces real product failures against a realistic test distribution, and loses its case when it merely looks impressive.

Eight properties of a loop that works

Loops usually break in unglamorous ways. They never stop, drain the budget, or hand back junk labelled as done. A review of public loop setups found eight properties in every loop that works reliably, and each one maps to a design decision in this codebase.

1. A checkable goal

Define "done" so a machine can verify it, not so a human can feel good. "Produce a good underwriting decision" is unverifiable. is_correct(decision, app_id), a comparison against the historical decisions, either matches or it doesn't. Without a checkable goal the loop never knows when to stop and will hand over confident, polished, wrong output as finished work. The goal is a binary predicate and not a description of quality. The training and test split makes sure it generalizes past memorization.

2. A hard stop

Max tries, a budget or a time limit, always, with no exceptions. Never assume the model will stop on its own. Grassroots setups like a shell while :; do ... done loop contain nothing on the model side that ends them, so they run until somebody kills the process, and OpenAI's SDK ships max_turns because experience showed models don't reliably end themselves. This codebase has three stops at three levels: MAX_STEPS = 8 in Loop 1 prevents endless tool chains, max_retries = 3 in Loop 2 prevents endless retries, and BUDGET_USD = 2.00 through CostMeter is the ultimate backstop. At the step cap Loop 1 returns ran_out_of_steps, when retries run out Loop 2 returns passed=False, and when the budget is gone the error propagates up and halts everything.

3. Good tools

The actions available to the agent must be reliable, clearly described and tightly scoped. The loop itself is trivial, and the work happens in the tools. Anthropic's guidance is that toolsets and their documentation must be designed with care, since a vague description yields vague calls and an over-powered tool yields irreversible mistakes. Here there are four tools with one job each: pure retrieval with no side effects, keyword search against a static knowledge base, a simulated external data pull, and a structured terminal action that forces commitment to a typed schema (Literal["accept", "modify", "decline", "refer"]), a rationale and explicit flags. Without that last one, the agent returns prose that needs fragile parsing.

4. Memory

Keep the history but summarize it so context doesn't bloat, and for anything longer than one chat, write state to disk. Anthropic's long-running-agent harness has the agent read prior state from git and progress files, make incremental progress and leave the environment clean for the next iteration. A raw context window does not survive restarts, crashes or size limits. Two mechanisms work at different timescales here. Within a run, LangGraph's add_messages reducer accumulates the whole tool conversation for one application. Across runs, the lessons list in run_state.json accumulates distilled rules over all applications and all restarts. distill_lesson() is the compression step: it stores a single generalizable rule and not a raw failure transcript, keeping the list short and the injected context manageable.

5. A separate checker

Generate, judge, fix, repeat, and the maker never grades its own work. Anthropic's SDK calls verifying your work the most underrated step, but the verification has to be independent of generation. An agent that grades itself turns into a yes-man, and giving a second agent only the grading job prevents that. This codebase has three independent levels of checking:

• Rubric check (structure). RubricMiddleware tests nine rules against the parsed output with pure boolean predicates and no model.
• Judge check (correctness). is_correct() matches the answer to the stored label with an exact string comparison.
• Audit check (anti-cheating). audit_prompt() looks for memorized answers with a binary model classification, not a quality evaluation.

The rubric and judge are fully deterministic. The audit uses a model but only for a binary structural question ("does this contain application IDs?"), the category where model judges are most reliable.

6. A plan for complex tasks

For a big, multi-step job, have the agent write a plan before acting, and for a small one, skip it. Anthropic's long-running-agent research found that working on one feature at a time is critical, because a loop told to fix everything at once thrashes while one change per pass converges. In this codebase the prompt lays out four tool steps in order, which works as a built-in plan: get the data, get the rules, get external signals, commit to a decision. The order stops the agent from jumping straight to a decision without consulting evidence, and the cap of eight steps is sized to allow the full sequence plus one retry.

7. Logging

Record each thought, action and result so a 3am debugging session has something to read. If a step isn't logged, it didn't happen as far as debugging is concerned. Failures in nested loops are close to undiagnosable without logs: which loop, which application, which attempt, which tool returned the wrong thing? Three tiers cover it. Progress lines such as [loop3] 3/16 APP-009 ... are flushed, which is non-negotiable on Windows where stdout is buffered. Decision lines carry the decision, status and running spend in one line per application. Lesson lines confirm that extraction happened.

8. Cost sense

Loops burn tokens quickly, so start small and bounded and then scale, and watch the first few runs before leaving a loop unattended. One hill-climbing round across 16 training applications, with five rounds plus reflection and audit calls, comes close to $2.00. Only after you know the cost per iteration can you set a budget and walk away. CostMeter tracks spend to four decimals and raises BudgetExhaustedError before an overspend registers:

# The ordering matters:
if self.spent >= self._budget:          # check BEFORE adding
    raise BudgetExhaustedError(...)
self.spent += cost                       # add AFTER the check clears

The event loop also catches budget exhaustion during hill climbing separately from exhaustion while processing, so running dry in the middle of a self-improvement round doesn't throw away the rest of the queue. The loop degrades gracefully: skip the expensive improvement step and keep processing.

A checklist before leaving it alone

Before trusting any loop to run unattended, confirm each of these:

1. The goal is machine-verifiable (here, is_correct() against ground truth).
2. There is a hard stop at every level (MAX_STEPS, max_retries, BUDGET_USD).
3. Tools are scoped and documented (four tools, one job each).
4. State survives restarts (run_state.json plus the lesson list).
5. The checker is separate from the generator (rubric middleware plus a deterministic judge).
6. Complex tasks have an implicit or explicit plan (the four-step sequence in the prompt).
7. Key steps are logged and flushed (the [loop3] progress lines).
8. A budget is set and tracked from the first call (one CostMeter shared across all loops).

Miss any one and the loop will fail in production, not always right away but reliably in the end. Most production failures reduce to no verify step, no stop condition or no persisted state. Get those right first, then tune everything else.