Home / Articles

Agent Harness From Scratch: Six Stages and an Experiment That Isolates Gains

A harness can move benchmark scores more than the model. Build a small agent harness in six stages and test one change at a time to see what actually helped.

Publisher: REACTAPP.TOP7 min read1524 words

This article is published in English.

Sources: Build an Agent Harness, Then Prove Which Change Moved the Score

Take one frontier model and two evaluation harnesses. Through the first it landed at 62.7 percent on a reasoning benchmark, through the second at 99.9, all in the same week. That 37.2-point swing outweighed the 32.5 points between the top two entries on the leaderboard. So the machinery wrapped around a model, as distinct from the model, can shift outcomes further than the model itself. Below is a rundown of what that machinery holds, a small build in six stages, and the final step that shows which of your changes truly moved the score.

What sits inside a harness

Six pieces make up a harness, and none touches the model weights.

  • The system prompt, meaning what the model is told about its role and limits before the task arrives.
  • Tool access: can it run code, read files or hit an API, and what does it get back when it does?
  • Context handling, which decides whether reasoning persists across turns or restarts every time.
  • Attempt handling: the number of tries allowed and whether the model sees how the last one turned out.
  • Time and token budgets.
  • Output parsing, which turns a response into an answer and settles what happens to a right answer in an unexpected format.

Each of these is a decision someone made. Adopt a framework and you inherit its decisions without seeing them. That works fine until you have to explain why your figure differs from somebody else's.

Why it matters

The effect shows up at smaller scale in daily work. Picture two models 0.7 points apart on a coding benchmark where switching harness swings results by about 5. The gap you're being asked to read is then about a seventh of the noise in the measurement. A public example points the same way: an interactive benchmark score for one model reportedly tripled once its provider enabled two settings that let it retain reasoning from turn to turn. Same model, just permitted to keep its own working.

Cleanest of all is a secondhand story. An engineer reportedly took one coding task and ran it five times on the same model, adding a single harness component each time. An empty repository gave roughly a 20 percent success rate. An instructions file lifted it to about 60, verification commands to 80, and a progress file to nearly 100. Nobody has confirmed the figures, so read them as illustration only. What counts is the shape, because it separates the contribution of each addition, which a comparison of two whole harnesses cannot.

Published work points the same way. One technique evolves prompts by reflecting on past runs and is said to beat a reinforcement-learning baseline by 10 percent on average and as much as 20, with up to 35 times fewer rollouts. Another treats accumulated context as a playbook that evolves, claiming a 10.6 percent improvement on agent tasks. A persistent scratchpad moved one model from 10 to 99 percent on a puzzle task. Weights stayed untouched in all three.

One finding cuts the other way. A 2026 paper divided the harness into four parts that could each evolve and expected the structure to pay off. Results were 0.657 against 0.642 for evolving a single flat string, a gap the authors call insignificant. Their reading is that a fixed optimization budget spread over four parts gives each less signal. Extra structure doesn't help by default, and this null result is worth more than the wins since it rules out the free-lunch interpretation.

Building one in six stages

The whole thing is about a hundred lines with no framework. You supply a single function that sends text to a model and returns text.

The first stage is the smallest thing that runs: take a task, call the model with its prompt and time the call.

import time
def run_task(task, call_model):
    t0 = time.time()
    output = call_model(task["prompt"])
    return {"id": task["id"], "output": output,
            "seconds": round(time.time() - t0, 2)}

The second stage lets the model act and see. A helper runs a shell command in a working directory with a timeout.

import subprocess
def run_command(cmd, workdir, timeout=60):
    try:
        r = subprocess.run(cmd, shell=True, cwd=workdir, timeout=timeout,
                           capture_output=True, text=True)
        return {"exit": r.returncode, "stdout": r.stdout[-4000:],
                "stderr": r.stderr[-4000:]}
    except subprocess.TimeoutExpired:
        return {"exit": None, "stdout": "", "stderr": "timed out"}

Two decisions are already tucked in there. Output is cut to the last 4,000 characters and commands stop after 60 seconds. Both change what the model sees, so both belong in your notes as part of the harness and not the task.

The third stage decides what carries between turns, by building the message list from the system prompt, the task and the history.

def build_messages(task, history, carry_reasoning=True):
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": task["prompt"]}]
    for turn in history:
        if carry_reasoning:
            msgs.append({"role": "assistant", "content": turn["output"]})
        msgs.append({"role": "user", "content": format_result(turn["result"])})
    return msgs

Make carrying reasoning a flag rather than a hardcoded choice, because you'll want to measure both settings.

Stage four parses an action out of the model's output without failing silently.

import re
ACTION = re.compile(r"<action>(.*?)</action>", re.S)
def parse_action(output):
    m = ACTION.search(output)
    if m:
        return m.group(1).strip(), "matched"
    fence = chr(96) * 3
    alt = re.search(fence + r"(?:bash|sh)?\n(.*?)" + fence, output, re.S)
    if alt:
        return alt.group(1).strip(), "fallback_fenced"
    return None, "no_match"

Log the second return value, since it carries the most information. When a noticeable fraction of runs end on the fallback, the parser is measuring formatting more than ability and the score can't tell you which. A related cautionary tale: a detector for whether a model claimed to be finished passed five validation cases, then missed two obvious claims because a period inside a filename cut a sentence-based pattern short. The transcripts said the tests passed while the detector reported no claim at all.

Stage five is the loop, with a turn cap of eight.

MAX_TURNS = 8
def agent_loop(task, call_model, carry_reasoning=True):
    history, workdir = [], task["dir"]
    for turn in range(MAX_TURNS):
        out = call_model(build_messages(task, history, carry_reasoning))
        action, parse_status = parse_action(out)
        result = run_command(action, workdir) if action else {"note": "no action"}
        history.append({"output": out, "result": result, "parse": parse_status})
        if action and task["done"](workdir):
            return {"solved": True, "turns": turn + 1, "history": history}
    return {"solved": False, "turns": MAX_TURNS, "history": history}

Here task["done"] is a function you provide that inspects the world instead of trusting the model's claim. That one distinction turns a demonstration into a measurement.

Stage six is what makes the whole thing an instrument: an experiment function that runs every task under every variant, repeating each a couple of times.

def experiment(tasks, call_model, variants, runs_per=2):
    results = []
    for name, config in variants.items():
        for task in tasks:
            for r in range(runs_per):
                out = agent_loop(task, call_model, **config)
                results.append({"variant": name, "task": task["id"], "run": r,
                    "solved": out["solved"], "turns": out["turns"],
                    "fallbacks": sum(1 for h in out["history"]
                                     if h["parse"] != "matched")})
    return results
variants = {"carry_on":  {"carry_reasoning": True},
            "carry_off": {"carry_reasoning": False}}

Change one thing, hold the rest constant, run both and compare. That method is missing from almost every published comparison. Alter the model, the prompt and the turn limit together and you've learned nothing about which moved the number.

What to measure and what to expect

Four numbers are worth tracking. First, the solve rate for each variant. Second, turns to solve, because a variant that reaches the same score in fewer turns costs less. Third, how often parsing hits the fallback. Fourth, how far apart repeated runs on one task land, which is the most telling. If identical runs disagree, noise may be larger than the effect you're after.

Don't be surprised if the first result shows the two variants tied. In one trial of this design, two frontier models tackled ten small bug-fixing tasks twice each, and all runs passed, so nothing separated the arms. The tasks were too easy to tell them apart. That's the usual outcome and it speaks about the task set, not the models. Pick tasks that land the solve rate mid-range, well below the ceiling.

Limits

A harness can't give a model abilities its weights lack, so every gain tops out at what those weights support. A harness tuned to your tasks won't travel either. What wins at repairing code isn't what wins at open-ended exploration, since the failures differ.

So the point was never that the model doesn't matter. Comparisons treat the two as if only one varies, when both do and only one gets reported.