Harness, Loop, Graph: Three Layers Behind Most Production Agent Failures

An agent dies in production, and the team blames the model. They replace it with a smarter one, and the agent dies again. The prompt gets rewritten, and it still dies. Retry logic goes in, and nothing changes. A ticket lands with the LLM provider, who reports that the model is behaving as designed. Everyone is confused.

The pattern repeats. What looks like a model defect is really a flaw in the machinery wrapped around it, and that machinery divides into three layers that teams rarely keep apart: tools the agent can reach (the harness), the cycle that decides when work ends (the loop), and the map of places the agent may go (the graph). Once the layers blur, a loop bug or a tooling bug gets blamed on the model, and effort goes into the visible layer while the broken one sits underneath.

Two layers where three are needed

Most production agent stacks in 2026 run on two layers. They have a graph, built with LangGraph, CrewAI or the OpenAI Agents SDK, and a durability layer such as Temporal, or nothing at all. What gets skipped is the harness: the layer governing which tools the agent may use, which URLs it may call, what it may return and when it must stop.

Vendors are heading the same way. Claude Code gained a sandbox.network.deniedDomains setting. GitHub introduced firewall controls at organisation level for its Copilot agent, and VS Code gained group-policy keys that filter what the agent can reach on the network. A public catalogue of agent patterns lists egress policy as a pattern in its own right. Research points in the same direction: a 2026 arXiv study kept the model constant per run, replaced only the harness across six frontier models and measured 41% lower cost per task and 44% lower latency. Few teams have invested in that layer.

None of this exists in isolation. Around the three core layers sit a gateway that routes requests and enforces budgets, memory covering retrieval and a semantic cache, tracing for every step, governance that applies quotas and rate limits, and evaluation that gates a deploy on a golden dataset. Each is worthwhile. None is what typically takes an agent down, though. Muddled harness, loop and graph boundaries do that, since the resulting failure is easy to misread as a model fault.

How the three layers were found

The trigger was a sourcing agent stuck repeating a call to query_suppliers. The LLM kept announcing that it needed more data, the loop kept looping and the token bill kept growing. Instead of reading tutorials, the useful reading turned out to be post-mortems, architecture documents and incident reports, and across unrelated projects the same three words kept appearing: harness, loop, graph.

A frequently cited framing from the agent-architecture writing describes the split this way. The harness covers one agent's relationship with its surroundings. The loop covers how a single agent's execution is controlled across time. The graph covers how many agents, tools and checkpoints connect across a whole application. Every layer rests on the one under it, and each only shows up once the one below is fairly stable.

With the names in hand, the original bug became obvious: the stop condition lived inside the tool and not in the loop. Harness and loop had been mixed up. The model was not broken, the layer boundaries were. Others report the same pattern. One piece put it bluntly: an agent called unreliable because of the wrong model usually has the wrong layer to blame. A newsletter described a founder who ripped out one model, swapped in a supposedly smarter one and received the same broken agent back, because the failure sat in the machinery around it.

The three layers were not invented here. They were named by people who run agents at scale and repeated across independent sources. What follows is what happens when the names meet real code.

Harness: what the agent can touch

The harness is the capability boundary. It answers which tools the agent has, what their input and output schemas are, what permissions it holds, what memory it reads, which URLs it can call and what data it can see.

Think of the agent as a contractor. The harness is the badge hanging from the lanyard: it decides which doors open, which systems are visible and what is allowed once inside. It says nothing about intelligence. It defines surface area, and safety lives there too. Without it, an agent is a function call holding an API key, free to call, read and return anything. With it, the agent can only do what the harness permits.

Most agent projects never build this layer. Tutorials stop once the model has called a tool and received a result. They skip three situations: a tool returning a 10MB JSON blob that overflows the context window, a tool fetching an arbitrary URL because injected text told it to, and a retried side effect running twice because nothing carried an idempotency key. All three are harness problems, and handling them is what turns a demo into a production system.

The pattern catalogue spells out the principle for network egress. Domains that agent tools may contact should be limited by allow and deny lists that the harness enforces, which removes the model from the network trust boundary. When the model decides whether to connect, prompt injection can defeat the control. A harness-level check turns isolation into a structural property and not a probabilistic one.

That distinction is the point. Models are probabilistic and can be tricked, injected or confused. A harness is structural and enforces rules in code and not in prompts. Asking the model politely not to call arbitrary URLs is a request. Making it impossible is a guarantee.

Loop: when the agent stops

The loop is the engine. The agent observes the world, picks an action, performs it, checks the result and then either carries on or halts.

The observe-act-verify cycle is the easy part. The hard part is the stop condition, because without one an unattended agent burns tokens forever and does not fail loudly.

The sourcing agent had none. It called query_suppliers, inspected the results, decided it needed more and called again, over and over. The LLM kept saying it needed more data because language models lean toward one more tool call: they are helpful, they are trying, and they do not want to give up. That trait is an asset on creative tasks and a liability on production tasks.

The pattern catalogue labels this the unbounded loop. The implementation amounts to "iterate while the model reports unfinished work", with no counter, timer or spending cap outside the model able to cut it off. The unspoken assumption is that the model will announce completion. On difficult tasks that seldom happens: it wanders into adjacent questions, repeats failed actions and cycles through errors without recognising the cycle.

This is not hypothetical:

• In July 2025, a runaway recursion in Claude Code consumed somewhere between $16,000 and $50,000 within five hours. No crash or error occurred. The agents simply carried out instructions with no end point.
• Public bug reports describe subagents spawning descendants more than 50 levels deep. One session used 4 million tokens inside five minutes. Another user paid $27.60 over a plan limit when usage credits activated mid-session, with no way to halt the run. In the first case, the useful research had finished after three or four agents and the other 44 contributed nothing new. In the second, subagents read an instruction about handling several independent targets as a licence to spawn grandchildren of their own.
• A July 2026 paper labelled the problem Infinite Agentic Loops. Its static-analysis tool, IAL-SCAN, covered 6,549 LLM agent repositories and confirmed 68 failures in 47 projects. Sixty-eight agents that never terminate are less a bug count than a billing time bomb.

A design guide for bounded agentic loops makes the same demand: the bound is part of the design when shipping a loop, not an optimisation added once things go wrong. Otherwise the only stopping rule is the model deciding it is finished, which is a hope and not a control.

Hence the stop condition must be mechanical, not semantic. "The LLM thinks the work is finished" is semantic. "Three attempts have been made" or "50 cents have been spent" is mechanical. Mechanical conditions are deterministic, testable and stable under replay. They do not hinge on the model's judgement, which is probabilistic and can differ on a retry.

An analytics firm's on-call agent shows what this costs. It was meant to trace the root cause of a payment-gateway timeout in under two minutes. It searched the logs, got 4,200 rows back, compressed them, concluded that more context was needed and searched again. Twelve passes later the ticket still had no summary, while the run had used 380,000 tokens and $47 of API spend. Nothing in the model refused to continue and nothing in the runtime ordered a halt. After the firm added iteration caps, stagnation fingerprints (detecting repeated near-identical results) and a governor on token budget, the share of runaway loops fell by 91%, and the median time to resolve a ticket went from 4.1 down to 1.6 minutes. A bounded loop did the work, not a stronger model.

What a bounded loop contains

The analytics example hints at the ingredients of a loop that cannot run away. Each one is checked by code outside the model:

• An iteration cap. A hard count of passes, so a task that reaches the limit ends with an explicit failure and not a silent spin.
• A stagnation check. A fingerprint of recent results, so that several near-identical outputs in a row count as no progress even when the step count is still low.
• A spending ceiling. A budget in tokens or money that the run cannot exceed, with the governor sitting in the runtime and not in the prompt.
• A wall-clock timeout. A limit on elapsed time, which catches slow loops that never trip the other counters.
• Recursion limits for subagents. A depth cap and a total-spawn cap, so a delegated task cannot create grandchildren that create their own.

None of these needs the model's cooperation, and each can be unit-tested with a fake model that never says it is done. A failing run that ends loudly after three attempts costs cents. The same run left to the model's opinion costs whatever the meter allows.

Graph: where the agent can go

The graph is the topology: the nodes that exist, the edges linking them, the points where the path branches, the spots where a person approves, and the state handed from node to node. If the graph is the map, then the loop is the engine travelling across it and the harness limits what can happen at each stop.

The central design choice is explicitness. A ReAct loop (think, act, observe, repeat) is an implicit graph with one node feeding itself and the LLM acting as the edge function. For simple tasks that is fine. Once structure matters, it breaks. A comparison of ReAct with LangGraph sums it up: in the first the model chooses the next move from the prompt and history, in the second the developer chooses it through code and explicit routing.

The difference bites as soon as a human approval step is needed. In an implicit graph there is no "between": the agent moves from step to step on its own reasoning. Adding a checkpoint means breaking the loop, adding a pause mechanism, externalising the state and resuming on a signal. That is retrofitting a graph onto a loop, and the loop resists because it was built to drive itself. In an explicit graph, a new node with an interrupt() call is enough. The graph changes while the loop and the harness stay as they are.

One developer ran the same policy, deterministic model and three scenarios through a plain tool-calling loop and an explicit LangGraph state graph. Behaviour matched exactly, yet the graph version took 97 lines against 50, about 94% more code, so at three steps the plain loop wins on every measure. The developer then pointed to what such a comparison cannot capture. In the loop, behaviour is implied by statement order and must be learned by reading the body. In the graph, edges are declared data, which lets the topology be inspected, drawn and replayed without opening any node. The value is nil at three steps, large at thirty, and larger still when a different person takes over.

That is the production argument. Demos have three steps, and the person who wrote them owns them. Production has thirty steps and is owned by whoever is on call at 3 AM. At that scale an explicit topology is survival and not overhead.

A practical test for deciding between the two forms is to ask who will own the agent in six months. A short, single-purpose agent maintained by its author can stay a plain loop. An agent with human approvals, several branches or a different on-call owner earns the explicit graph, because the topology becomes data that can be drawn, reviewed and replayed.

What confusing the layers looks like

The failure modes below come from a build, which is the safe place to mix up layers. They are the same ones that show up in production.

The harness owning the loop

In the confused version, the check for "is this good enough" lives inside the tool, so the tool decides when the loop stops. A tool should be a capability, such as "can query suppliers", and not a decision-maker saying "enough, stop." Burying the stop condition there makes the loop untestable on its own, makes the tool impossible to swap without reimplementing the stop logic and leaves no place for a step ceiling because the tool knows nothing about attempts.

The repair is structural. Pull the stop condition out of the tool and put it in the graph node: the tool hands back data, the node decides whether another pass is warranted, the harness supplies the capability, and the loop lives in the graph. The confused version looks like this in code:

# CONFUSED LAYER — Harness owns Loop logic
async def query_suppliers(item: str, budget: float, quantity: int, attempts: int) -> dict:
    """Tool decides when the loop stops. Capability + control flow in one function."""
    results = query_mock_marketplace(item, budget, quantity)

    # The tool decides when to stop — couples execution logic with capability
    if len(results) > 5 or attempts >= 3:
        return {"status": "STOP", "suppliers": results}
    return {"status": "CONTINUE", "suppliers": results}


async def research_node(state: AgentState) -> dict:
    # The node just calls the tool and forwards whatever the tool decided.
    # It has no idea why the loop stopped — that logic is hidden inside the tool.
    result = await query_suppliers(
        state["brief"].item, state["brief"].budget, state["brief"].quantity,
        state.get("attempts", 0) + 1,
    )
    return {"suppliers": result["suppliers"], "status": result["status"].lower()}

# ✅ CLEAN SEPARATION — Harness is pure capability, Loop lives in the Node
MAX_RESEARCH_ATTEMPTS = 3

async def query_suppliers_impl(item: str, budget: float, quantity: int) -> dict:
    """Pure Harness capability: schema validation, circuit breaker, marketplace call.

    The tool has no idea a loop exists. It returns data. It does not decide
    when to stop. Circuit breaker + fault injection live here because they are
    capability concerns (is the downstream reachable?), not control-flow concerns.
    """
    breaker = get_circuit_breaker("query_suppliers")
    if not breaker.allow_call():
        raise CircuitBreakerOpenError("query_suppliers", breaker.state)

    maybe_inject("query_suppliers")  # fault injection for chaos tests
    try:
        results = query_mock_marketplace(item, budget, quantity)
        breaker.record_success()
        return {"suppliers": results}
    except Exception as e:
        breaker.record_failure()
        raise


async def research_node(state: AgentState) -> dict:
    """Graph Node / Loop: mechanical stop conditions and routing decisions.

    The node reads state, calls the tool through the registry, and decides
    whether to stop — based on counters in state, NOT on the LLM's judgment
    and NOT on the tool's opinion.
    """
    brief = state["brief"]
    attempts = state.get("attempts", 0) + 1

    # Harness call — schema validation, timeout, output cap all happen inside registry.call
    query_result = await registry.call("query_suppliers", {
        "item": brief.item,
        "budget": brief.budget,
        "quantity": brief.quantity,
    })
    suppliers = query_result.get("suppliers", [])

    # Mechanical stop condition — enforced OUTSIDE the tool, in code, on state.
    # Deterministic. Survives replay. Testable without the tool being live.
    if not suppliers and attempts >= MAX_RESEARCH_ATTEMPTS:
        logger.info(
            "STOP_CONDITION_FIRED path=no_matches item=%s attempts=%d max=%d",
            brief.item, attempts, MAX_RESEARCH_ATTEMPTS,
        )
        return {"suppliers": [], "attempts": attempts, "status": "no_matches"}

    if not suppliers:
        return {"suppliers": [], "attempts": attempts, "status": "running"}

    # Enrich, then mark completed
    enriched = []
    for s in suppliers:
        quote = await registry.call("get_price_quote", {
            "supplier_name": s["name"], "item": brief.item, "quantity": brief.quantity,
        })
        rating = await registry.call("check_seller_rating", {"supplier_name": s["name"]})
        enriched.append({**s, "quote": quote, "rating_info": rating})

    return {"suppliers": enriched, "attempts": attempts, "status": "completed"}


# The Graph owns WHERE the agent can go — conditional edges, not the node body.
def _should_retry_or_end(state: AgentState) -> str:
    """After Research: loop back if still running, else Score or END."""
    if state["status"] == "running":
        return "research"          # self-loop — the retry
    if state["status"] == "no_matches":
        return END                 # mechanical stop → graph terminates
    return "score"                 # success → next node


# Wiring (in build_sourcing_graph):
#   graph.add_conditional_edges("research", _should_retry_or_end,
#       {"research": "research", "score": "score", END: END})

The loop owning the graph

Many frameworks hand over a ReAct loop where the topology is implicit: the agent decides the next step from the model's output and no explicit map exists. This holds up until a human approval step must slot between deciding and confirming, and then the retrofit begins, with the loop fighting it.

The remedy is to design the graph explicitly from the beginning, with nodes as functions, edges as transitions, a few of them conditional and a few acting as interrupts. The loop still carries the agent across, but the graph defines the reachable places, not the loop. The monolithic ReAct form, in which the loop is the graph and the LLM is the edge function, looks like this:

# ❌ CONFUSED LAYER — Implicit topology inside a monolithic ReAct loop
# The loop IS the graph. The LLM is the edge function. There is no "between."
while not agent.is_done():
    action = llm.decide_next_step(history)
    if action == "query":
        data = query_suppliers()
    elif action == "approve":
        # Retrofitting human approval into an implicit loop requires messy pauses.
        # You have to break the loop, externalize state, park the worker, and
        # resume on a webhook — the loop fights you because it was designed to
        # be self-driving.
        pause_execution_and_wait_for_webhook()
    elif action == "stop":
        break


# ✅ CLEAN SEPARATION — Declarative graph with explicit nodes and human interrupt
from langgraph.graph import StateGraph, START, END
from langgraph.types import interrupt


def approve_node(state: AgentState) -> dict:
    """Human checkpoint via interrupt() — a STRUCTURAL pause, not a prompt.

    The graph genuinely suspends here. interrupt() pauses execution and waits
    for a human to call Command(resume=approval_data). The worker is freed.
    If the worker crashes while waiting, Temporal replays the activity, the
    graph re-runs from the last checkpoint, and interrupt() fires again —
    the human re-approves. No data loss.
    """
    selected = state.get("selected_supplier", {})
    brief = state.get("brief", {})

    approval_request = {
        "item": brief.item,
        "quantity": brief.quantity,
        "supplier": selected.get("name", ""),
        "price": selected.get("price", 0),
        "total_cost": selected.get("price", 0) * brief.quantity,
    }

    # interrupt() pauses the graph HERE. The human must call /approve
    # with a resume value to continue.
    approval = interrupt(approval_request)

    approved = approval.get("approved", False)
    comment = approval.get("comment", "")

    return {
        "approval_status": "approved" if approved else "rejected",
        "approval_comment": comment,
        "rejection_count": state.get("rejection_count", 0) + (0 if approved else 1),
    }


def _after_approve(state: AgentState) -> str:
    """After Approve: go to Confirm if approved, retry Research if rejected."""
    if state.get("approval_status") == "approved":
        return "confirm"
    if state.get("rejection_count", 0) + 1 >= 3:
        return END
    return "research"


def build_sourcing_graph(checkpointer=None):
    """Topology is DECLARED DATA, not statement order in a loop body.

        START → Research → (retry) → Score → Decide → Approve → Confirm → END
                                              ↓ (rejected, < 3)
                                             Research
                                              ↓ (rejected, = 3)
                                             END
    """
    graph = StateGraph(AgentState)

    graph.add_node("research", research_node)
    graph.add_node("score", score_node)
    graph.add_node("decide", decide_node)
    graph.add_node("approve", approve_node)      # explicit checkpoint node
    graph.add_node("confirm", confirm_node)

    graph.add_edge(START, "research")
    graph.add_conditional_edges("research", _should_retry_or_end,
        {"research": "research", "score": "score", END: END})
    graph.add_edge("score", "decide")
    graph.add_edge("decide", "approve")
    graph.add_conditional_edges("approve", _after_approve,
        {"confirm": "confirm", "research": "research", END: END})
    graph.add_edge("confirm", END)

    return graph.compile(checkpointer=checkpointer)

The graph owning the harness

This one is rarer, though it turns up in code reviews. A graph node calls an external API directly, such as an HTTP GET to a supplier endpoint inside the node function, so the graph owns the tool call.

It is wrong because the graph is topology and should know nothing about HTTP, authentication, retries or circuit breakers. With the call inside the node, adding a circuit breaker means editing the graph, swapping the implementation means editing the graph, and testing the graph needs a live tool.

The fix routes the call through the harness. The node asks for call_tool("query_suppliers", {...}) and receives a validated, typed result. The harness handles the HTTP request, the retries, the circuit breaker and schema validation. The graph does not know HTTP exists and the harness does not know a graph exists:

# ❌ CONFUSED LAYER — Graph node owns the tool call (Harness concern)
import httpx


def supplier_node(state: AgentState) -> dict:
    """The graph node reaches directly into HTTP, auth, retries, and parsing.

    Why that's wrong: the graph is topology. It should not know about HTTP,
    authentication, retries, circuit breakers, or schema validation. Those are
    harness concerns. When the graph owns the tool call:
      - You can't add a circuit breaker without modifying the graph.
      - You can't swap the tool implementation without modifying the graph.
      - You can't test the graph without the tool being live.
      - You can't enforce an egress allowlist — the URL is hardcoded here.
    """
    resp = httpx.get(
        f"https://supplier-api.example.com/suppliers",
        params={"item": state["brief"].item},
        headers={"Authorization": f"Bearer {SUPPLIER_API_KEY}"},
        timeout=10.0,
    )
    resp.raise_for_status()
    suppliers = resp.json()["suppliers"]  # no schema validation, no output cap
    return {"suppliers": suppliers}


# ✅ CLEAN SEPARATION — Graph node calls through the harness; harness owns the call
async def supplier_node(state: AgentState) -> dict:
    """The graph node says 'call this tool with these args' and gets back a
    validated, typed result. The graph doesn't know there's HTTP involved.
    The harness doesn't know there's a graph involved.
    """
    query_result = await registry.call("query_suppliers", {
        "item": state["brief"].item,
        "budget": state["brief"].budget,
        "quantity": state["brief"].quantity,
    })
    suppliers = query_result.get("suppliers", [])

    # Mechanical stop condition — the node's job, not the tool's
    if state.get("attempts", 0) + 1 >= MAX_RESEARCH_ATTEMPTS and not suppliers:
        return {"suppliers": [], "attempts": state.get("attempts", 0) + 1, "status": "no_matches"}

    return {"suppliers": suppliers, "attempts": state.get("attempts", 0) + 1, "status": "completed"}


# What the harness does — the graph never sees this:
#
#   registry.call("query_suppliers", {...})
#     │
#     ├─ 1. validate_input(QuerySuppliersInput, args)        # Pydantic v2
#     ├─ 2. EgressFilter(spec.allowed_egress)                # URL allowlist
#     ├─ 3. SandboxedHTTPClient(egress_filter)               # blocked domains = structural
#     ├─ 4. run_with_timeout(fn, kwargs, spec.timeout_seconds)
#     ├─ 5. enforce_output_cap(result, spec.max_output_bytes)  # 64KB cap
#     └─ 6. validate_output(QuerySuppliersOutput, result)    # garbage caught here
#
# The graph node gets back a validated dict. It never sees the HTTP call,
# the auth header, the retry, the circuit breaker, or the egress filter.
# Swap httpx for aiohttp, swap the supplier API for a new one, add a circuit
# breaker — the graph doesn't change. The harness doesn't know a graph exists.

Who owns which layer

Each layer needs an owning piece of infrastructure. When ownership is unclear, the layers bleed.

Temporal owns the loop's crash safety

When the engine dies partway through a cycle, Temporal handles recovery. It stores every step in an event history. After a worker crash it replays the workflow without re-running activities, injecting their recorded results instead, so the loop restarts from the last completed step and not from zero.

This is the reason a stop condition has to be deterministic. A condition like "the model judges the work done" is non-deterministic, and a replay may reach a different verdict. A step counter, a spending ceiling or a quality threshold derived from tool outputs works under replay. The model may weigh in, but the if attempts >= 3: stop check belongs in code.

LangGraph owns the graph's topology

In LangGraph, nodes are plain functions, edges are transitions, and conditional edges are functions that return the name of the next node. LangGraph compiles all this into a runnable graph. Since the graph is explicit, it can be viewed, each node can be tested alone, and a node can be added without disturbing the rest. A human approval point is just a node calling interrupt(), which changes the graph and leaves loop and harness unchanged.

LangGraph also owns the shape of the state, a TypedDict passed between nodes. Every node reads what it needs and writes what it produces, so the state acts as the contract between them. A node that never writes suppliers leaves the next one unable to read it, and the type checker flags that before runtime.

A hand-built tool harness owns everything else

This is the layer neither Temporal nor LangGraph provides, so it must be built from scratch. It handles:

• Registration. Each tool declares a name, a Pydantic input schema, an output schema, a timeout and whether it needs idempotency.
• Validation. Inputs are validated before a tool runs and outputs afterwards, so a garbage response is caught before the agent sees it.
• Egress filtering. Tools making HTTP calls go through a sandboxed client with a URL allowlist, so no tool can call an arbitrary address. The harness decides, not the model.
• Output capping. Results are truncated at 64KB before reaching the context window. A 10MB blob arrives cut short with a _truncated: true marker.
• Circuit breakers. Each tool has one, shared across every agent workflow. Five failures in a row open the circuit, and from then on agents fail fast instead of pounding a dead API.
• Bulkheads. Per-tool concurrency limits stop one slow tool from occupying every worker activity slot.
• Idempotency keys. For tools with side effects, the harness builds a key from the workflow ID and step name. A retried activity gets the existing result back and no duplicate is created.

None of this comes from Temporal or LangGraph. It is the layer that makes tools safe to call, and the one most agent projects omit.

Verification: "the agent said it worked" versus "it worked"

The loop's third step is verify, and it has to be external. It cannot mean the agent checking its own output, or the LLM grading its own homework.

The pattern catalogue names the fix the verification ledger: self-reported claims such as "build passed" give way to structured records, where each check is a database INSERT and each bundle of evidence a SELECT. Left alone, an agent may imagine that checks passed, skip steps without saying so or claim results from a tool it never ran.

Spotify's Honk team ran into exactly this. Their agents sometimes claimed "tests green, no issues found" and were wrong. They wired deterministic verifiers for format, build and test into the loop and blocked pull-request creation whenever any verifier failed. The agent's prose stopped being the success signal. The verifier's structured output replaced it.

Other projects go further. One open-source tool either holds a receipt for "done, all tests pass" or labels the claim unverified, and a passing command label by itself never upgrades a semantic claim. A research paper phrases the rule as: re-derive the claim and do not trust it, because the agent's word counts for nothing as input.

In the stack described here, verification is a separate Temporal activity, such as verify_po_exists or verify_payment_initiated, that queries the database directly. The agent says it created the purchase order. The activity checks the PO database, and if the order is absent the workflow fails whatever the agent reported.

Self-report is the signal most likely to be wrong in ways that are hardest to spot, which is why checking must be external. The agent called create_po, parsed the response and said it worked. Yet the response could have been a 200 OK carrying an error body, or the order could have been written and then rolled back on a transaction timeout, or the agent could have read the wrong field and reported an ID that never existed. The agent knows what the tool returned and nothing about database state. So the database gets asked.

Production evidence backs this up. A July 2026 paper examined Leni, an AI business analyst, whose verification loops execute, observe, compare and correct using small task-specialised models. The complete system beat its frontier base model by 11 percentage points on SpreadsheetBench, 91.25% against 80.25%. Verification alone added a modest 1.5 points, but those points clustered at the top of the score range, where they converted tasks that would otherwise fail. Most of the overall gain came from scaffolding, routing and specialist models. Verification does not rescue every case. It rescues the borderline ones, where the agent claims completion and reality disagrees.

The separation principle

The governing rule: every layer should be testable, replaceable and understandable without the others.

• The graph can be checked with tool calls mocked, confirming that the correct nodes fire in the correct order.
• The harness can be exercised by calling tools directly and examining schema validation, circuit breakers and idempotency.
• The loop can be tried with a trivial activity inside a Temporal workflow, confirming that it survives a crash.

Changing the graph leaves the harness alone, swapping a tool implementation leaves the graph alone, and altering the loop's retry policy touches neither.

The payoff is practical. The sourcing agent's graph changed three times: a scoring node, a rejection-aware retry loop and a human approval checkpoint, with no edit to harness code. The LLM provider moved from OpenAI direct to a routing layer without touching graph code. The retry policy went from aggressive to strictly non-retryable for side-effecting activities without changing graph or harness. Separation buys agility more than purity: the freedom to change one thing without breaking three others.

The same discipline pays off when something does go wrong. A failed run can be traced to one layer by asking three narrow questions. Did a tool return something it should never have been allowed to return, or reach somewhere it should never have been allowed to reach? That points to the harness. Did the agent keep going past any sensible limit? That points to the loop. Did the agent end up at a node it should not have been able to reach, or skip a checkpoint? That points to the graph. Swapping the model answers none of the three.

Three questions for any agent design

1. What can it touch? (Harness) Which tools exist, with which schemas, how much egress is allowed, where the circuit-breaker thresholds sit and which idempotency keys apply.
2. When does it stop? (Loop) The step ceiling, the cost ceiling, the timeout, and the mechanical stop condition that does not depend on the LLM's judgement.
3. Where can it go? (Graph) The nodes, the edges, the conditional branches, the human checkpoints and the state shape that flows between nodes.

When any answer is "the model decides", a problem exists. The LLM advises while code makes the decision, the harness enforces, the graph routes and the loop bounds.

In short, there are three layers, three owners and three questions. Mix them and the bugs look like model trouble without being so. Keep them apart and the model's unreliability stays contained, tools are safe to call and the agent halts when it should.