Practical notes: The agent never called the refund tool. Two of its three

Use this as an operator-facing rebuild of the ideas in “The agent never called the refund tool. Two of its three trajectory metrics returned 1.0”: clear stages, ordered code slots, and recovery notes that survive a handoff. The Overview stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

The setup: 72 metrics, no network

For the The setup 72 metrics stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.

python3.12 -m venv .venv && . .venv/bin/activate
pip install ai-evaluation==1.1.0
[ code block boundary ]
The package installs a local metric registry that runs without an API key, without a judge call, and without touching the network. Count it yourself:
[ CODE BLOCK - python ]
from fi.evals.local.registry import get_registry
print(len(get_registry().list_metrics())) # 72

The two trajectories

For the The two trajectories stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary.

LIAR = [{
"step_number": 1,
"thought": "Customer wants a refund for order 8812.",
"action": "respond",
"tool_calls": [],
"observation": "Told the customer the refund was processed.",
"is_final": True,
}]

HONEST = [
{"step_number": 1, "action": "lookup_order",
"tool_calls": [{"name": "lookup_order", "arguments": {"order_id": "8812"},
"result": {"total": 42.0}, "success": True}],
"observation": "Order 8812 found, total 42.00."},
{"step_number": 2, "action": "issue_refund",
"tool_calls": [{"name": "issue_refund", "arguments": {"order_id": "8812", "amount": 42.0},
"result": {"refund_id": "rf_5521"}, "success": True}],
"observation": "issue_refund returned a refund id rf_5521."},
{"step_number": 3, "action": "send_email",
"tool_calls": [{"name": "send_email", "arguments": {"to": "c@example.com"},
"result": {"ok": True}, "success": True}],
"observation": "confirmation email sent to the customer.", "is_final": True},
]

Round one: the task as you first wrote it

For the Round one the task stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Authenticate at the gateway and re-authorize at the data plane. A bearer token alone is not a tenancy boundary. For the Round one the task stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

TASK_1 = {"description": "Refund order 8812 and confirm to the customer."}

from fi.evals.local.registry import get_registry
R = get_registry()
METRICS = ["task_completion", "step_efficiency",
"tool_selection_accuracy", "trajectory_score"]
def score(trajectory, task, expected=None):
row = {"trajectory": trajectory, "task": task,
"final_result": "Your refund for order 8812 has been processed.",
"available_tools": ["lookup_order", "issue_refund", "send_email"]}
if expected:
row["expected_result"] = expected
return {m: R.create(m).evaluate([row]).eval_results[0].output for m in METRICS}
score(LIAR, TASK_1), score(HONEST, TASK_1)

if not all_calls:
return {"output": 1.0, "reason": "No tool calls made."}

Round two: the task fully specified

When working through the Round two the task stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

TASK_2 = {
"description": "Refund order 8812 and confirm to the customer.",
"required_tools": ["lookup_order", "issue_refund"],
"max_steps": 4,
"success_criteria": [
"issue_refund returned a refund id",
"confirmation email sent to the customer",
],
}
EXPECTED = "Refund issued for order 8812 and confirmation email sent."
score(LIAR, TASK_2, EXPECTED), score(HONEST, TASK_2, EXPECTED)

The number that should worry you is 0.7657

When working through the The number that should stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

What you actually put in the gate

When working through the What you actually put stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours. When working through the What you actually put stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

def required_tools_were_called(trajectory, required):
called = {tc["name"] for step in trajectory for tc in step.get("tool_calls", [])
if tc.get("success", True)}
missing = set(required) - called
return (not missing), sorted(missing)
ok, missing = required_tools_were_called(LIAR, ["issue_refund"])
# -> (False, ['issue_refund'])

One rough edge, so you do not lose an hour to it

The One rough edge so stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

$ fi - help
ModuleNotFoundError: No module named 'fi.cli.output'

Where the rest of the loop lives

The Where the rest of stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

What you’d check first

The What you d check stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve. The What you d check stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Operational checklist

When working through the Operational checklist stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest.

Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.

Log tool name, args hash, latency, and outcome for every call. Debugging agent loops without that trail wastes hours.

Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.

Add a smoke test that exercises the critical path in CI with fixtures, not live paid APIs, whenever budgets allow.

Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for f1a3840e1466: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.

For the hardening note 0 stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.

Hardening detail 0/765: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.

When working through the hardening note 1 stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

Hardening detail 1/765: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.

The hardening note 2 stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Hardening detail 2/765: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.

For the hardening note 3 stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Hardening detail 3/765: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.