This article is published in English.
Practical notes: A minimum viable experimentation platform for AI agents
Operable walkthrough of Practical notes: A minimum viable experimentation platform for AI agents: contracts, checks, and drop-in code slots for teams shipping this pattern.
The following notes reconstruct a practical path around “A minimum viable experimentation platform for AI agents”. Emphasis stays on contracts, checks, and drop-in code placeholders rather than motivational framing. When working through the Overview stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
agent runtime
memory and retrieval
system prompt
model
tools
guardrails
application logic
should version B replace version A?
Defining what B actually is
The Defining what B actually stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
A = agent-v17
B = agent-v18
pipeline:
name: support-agent
version: 18
components:
runtime:
artifact: agent-runtime@sha256:...
memory:
artifact: memory-service@sha256:...
configuration:
strategy: hybrid
top_k: 8
model:
route: support
model: provider/model-x
prompt:
artifact: sha256:...
tools:
- artifact: customer-lookup@sha256:...
- artifact: refund-tool@sha256:...
model X vs model Y
memory top_k=5 vs top_k=8
Assignment is not (another) production dependency
The Assignment is not another stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts.
request
user
session
workflow_instance
workflow_instance = migration-8291
variant = B
pipeline = agent-v18
Assignment is not exposure
The Assignment is not exposure stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts. The Assignment is not exposure stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
ASSIGNMENT
user-18271 → B
EXPOSURE
memory-v2 retrieval started
assigned 10,428
exposed 8,912
exposure rate 85.46%
assigned_variant = B
realized_model = X
fallback_reason = provider_unavailable
Traces explain what happened
For the Traces explain what happened stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
experiment.id
experiment.version
experiment.variant
experiment.assignment_id
pipeline.id
pipeline.version
execution.purpose
B has a lower task completion rate
show failed B traces
assignment
exposure
outcome
feedback
evaluation
Offline evaluation uses the same pipeline
For the Offline evaluation uses the stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
prompt
expected answer
scenario:
id: duplicate-charge-001
input:
message: >
I was charged twice for order 9811.
environment:
fixture: duplicate-charge-customer
expected:
refund_count: 1
ticket_status: resolved
limits:
max_turns: 15
max_tool_calls: 20
max_cost: 0.50
REPLAY_DIVERGED
Shadow execution bridges the gap between offline and production
For the Shadow execution bridges the stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness. For the Shadow execution bridges the stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
latency
cost
tool usage
model fallbacks
guardrail failures
judge scores
trajectory differences
Not everything needs an LLM judge
When working through the Not everything needs an stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.
refund_count == 1
ticket_status == resolved
helpfulness
clarity
tone
quality of explanation
evaluator
evaluator version
judge model
judge prompt
rubric
Oh! About the statistics
When working through the Oh About the statistics stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.
A/B allocation
binary metrics
continuous metrics
95% confidence intervals
sample ratio mismatch detection
fixed-horizon analysis
Experiment
support-agent-v18
Randomization
user
Primary metric
ticket resolution
A 81.4%
B 85.1%
Difference
+3.7 percentage points
95% CI
[...]
Experiment health
SRM PASS
4 model calls
47 model calls
max cost / execution
max tokens / execution
max agent turns
max tool calls
max variant spend / hour
Variant B
SRM PASS
Error rate PASS
Cost / request +312%
Circuit breaker TRIPPED
New exposure PAUSED
What to build first
When working through the What to build first stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node. When working through the What to build first stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
assignment
exposure
outcome
feedback
evaluation
what was randomized
what treatment was assigned
what treatment actually ran
what pipeline version produced the execution
what outcome was measured
how that outcome was evaluated
What exactly did we run?
What happened when we ran it?
Did assigning users to B improve the outcome we care about?
Operational checklist
For the Operational checklist stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state.
Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
Write a short runbook: how to rotate keys, how to drain the queue, how to roll back the last ingest.
Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.
Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.
Batch note for 35a4ad1bbf8c: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.
When working through the hardening note 0 stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
Hardening detail 0/952: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
The hardening note 1 stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
Hardening detail 1/952: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
For the hardening note 2 stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.
Hardening detail 2/952: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
When working through the hardening note 3 stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.
Hardening detail 3/952: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
The hardening note 4 stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.
Hardening detail 4/952: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.
For the hardening note 5 stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.
Hardening detail 5/952: measure wall time, error class, and token spend for this note, then decide whether to keep the change based on a fixed question set rather than anecdote.