A Minimum Viable Experimentation Platform for AI Agents: A/B Tests and Shadow Runs

Judging whether a new agent version beats the one in production is harder than it looks. In development the routine is manageable: hold a set of test cases, run both versions, read a few traces, let another model grade the outputs and decide whether the change deserves more work. Things get harder when the target is not a prompt or a single model call but a system whose components change on their own schedules and interact in ways that are not always obvious.

Picture a customer-support agent assembled from these parts:

agent runtime
memory and retrieval
system prompt
model
tools
guardrails
application logic

Suppose the retrieval strategy changes and task completion rises on the test set. The agent may really be better. It may also be making more model calls, costing 20 percent more, behaving differently with real users, or benefiting from some other component that happened to change in the same window. At that point the job looks less like model evaluation and more like experimentation.

What follows is a sketch of the smallest platform that would be comfortable to rely on for one fairly plain question:

should version B replace version A?

This only makes sense at a certain size. A single agent, twenty hand-reviewed test cases and a handful of internal users do not need most of it. The platform earns its keep once several teams alter models, prompts, memory, tools and agent implementations independently, experiments run non-stop, and somebody must eventually sign off on a production change using evidence that still reads clearly months afterward. A company-wide agent platform fits that description.

What B actually is

The first problem is basic. Say an experiment declares:

A = agent-v17
B = agent-v18

What exactly is agent-v18? If the prompt changes mid-experiment, the retrieval index is rebuilt and the model gateway swaps its default model, the label says almost nothing about what was evaluated. This bites agents especially hard, because the behavior seen at the end comes from the whole pipeline and not from one isolated model invocation.

So the immutable pipeline definition is the treatment under evaluation. It can look like this:

pipeline:
  name: support-agent
  version: 18

components:

  runtime:
    artifact: agent-runtime@sha256:...

  memory:
    artifact: memory-service@sha256:...
    configuration:
      strategy: hybrid
      top_k: 8

  model:
    route: support
    model: provider/model-x

  prompt:
    artifact: sha256:...

  tools:
    - artifact: customer-lookup@sha256:...
    - artifact: refund-tool@sha256:...

The precise artifact format hardly matters. What counts is that a variant can be reconstructed as a system configuration, so that months later someone can say what a user assigned to B was meant to run. It also keeps the experimentation platform independent of whatever happens to be changing. One experiment might compare:

model X vs model Y

another might compare:

memory top_k=5 vs top_k=8

and a third might pit two completely different agent implementations against each other. Seen from the experiment, each of these is just a pipeline variant. When an experiment starts, those definitions freeze. If the treatment must change, a new experiment iteration is created, so the historical meaning of B is never rewritten in silence.

Assignment should not become a production dependency

Next comes the question of who gets A and who gets B. A tempting design has every request ask a central experimentation service for its treatment. That pulls the experimentation platform into the availability and latency budget of the application itself, a poor trade for a system whose main purpose is measurement.

The first version uses deterministic hashing. The control plane pushes its current configuration to an SDK that lives inside the application, and the choice of arm is made right there, locally. That yields a helpful failure property: if the experiment service goes down, the platform temporarily cannot start new experiments or change allocations, yet the application is untouched and the SDK keeps working from its last valid configuration.

For the MVP, a few randomization units are enough:

request
user
session
workflow_instance

The last one exists because agents do not necessarily live as long as a web request. An agent may begin a repository migration today, wait for an outside approval and carry on tomorrow. If assignment were recomputed from tomorrow's configuration, the same workflow could start on one implementation and end on another, making both operations and analysis needlessly hard. The assignment therefore stays pinned:

workflow_instance = migration-8291
variant           = B
pipeline          = agent-v18

and it travels with the workflow each time it resumes.

Assignment differs from exposure

Another distinction becomes important when the treatment lives somewhere inside an agent pipeline. Landing in a treatment arm does not guarantee that the treatment ran.

Take two memory implementations under test. A user lands in B, the new memory version, but the agent answers without doing any memory lookup. The assignment exists, yet the component under test never had a chance to influence the result. The two should be separate events:

ASSIGNMENT
user-18271 → B

EXPOSURE
memory-v2 retrieval started

This also gives experiment-health information. Beyond the count of users randomized to each arm, the exposure picture can be inspected:

assigned          10,428
exposed            8,912
exposure rate      85.46%

If A and B show unexpectedly different exposure rates, that difference deserves attention first.

A related issue arises when the assigned treatment is not the one that finally runs. Say B chooses model Y, model Y returns a 503, and the gateway falls back to X. The record reads:

assigned_variant = B
realized_model   = X
fallback_reason  = provider_unavailable

and the execution stays in B.

This follows the intent-to-treat rule, ITT. Each subject is analyzed in the arm it was randomized to, even if what actually executed differs. The fallback here is part of the B experience, and removing such requests later would reshape the population using an event that happened after randomization. Exposure and realized-treatment data stay valuable, mostly as health and diagnostic signals, unless the analysis spells out something else.

Traces explain what happened

An experiment result says whether an outcome changed, and it says nothing about why. For the why, OpenTelemetry is the sensible choice, not a new tracing protocol. The experimentation layer only has to attach a little extra context to each span:

experiment.id
experiment.version
experiment.variant
experiment.assignment_id

pipeline.id
pipeline.version

execution.purpose

With that in place, an aggregate result such as:

B has a lower task completion rate

can turn into a far more useful request:

show failed B traces

and the traces reveal if the new pipeline pulls in weaker context, retries tool calls, switches to a different model, gets stuck in loops or simply walks a longer route to the same result.

A trace is still not the experiment record. It explains well how one execution unfolded, but experimental outcomes may outlive it. A customer might talk to an agent today and renew a subscription a week later, long after the original trace ended. So a smaller experiment event stream runs beside tracing:

assignment
exposure
outcome
feedback
evaluation

The trace records execution. The event stream does the bookkeeping that links randomization to outcomes arriving later.

Offline evaluation runs the same pipeline

Before B reaches production users, it should meet known scenarios. For agents, a scenario should not shrink to:

prompt
expected answer

since the interesting outcome often is not the returned text but the state of the environment afterward. A support scenario might be written like this:

scenario:
  id: duplicate-charge-001

input:
  message: >
    I was charged twice for order 9811.

environment:
  fixture: duplicate-charge-customer

expected:
  refund_count: 1
  ticket_status: resolved

limits:
  max_turns: 15
  max_tool_calls: 20
  max_cost: 0.50

The same scenario then runs against pipeline A and pipeline B, possibly several times, since agent executions are stochastic and two runs of one configuration can follow different trajectories from the same starting state. The MVP uses a resettable sandbox with fixture-backed services or test databases. Building general multi-turn trace replay is better left for later.

Replay holds up only while the candidate requests interactions that were already recorded. Suppose B decides differently on turn three and calls a tool the original trajectory never touched. Then no recorded world contains that branch. The honest outcome is:

REPLAY_DIVERGED

and a mocked-up response with a pretense that the run is still a replay is the wrong move. A stateful sandbox lets both agents branch naturally while controlling their starting conditions, which is a better first approximation of real agent evaluation.

Shadow execution fills the gap

One useful step sits between the controlled sandbox and exposing users to B: run B asynchronously on a copy of real production inputs without serving its output. Writes and outside side effects are disabled or redirected to an isolated environment. That supplies production-like inputs and lets the team observe:

latency
cost
tool usage
model fallbacks
guardrail failures
judge scores
trajectory differences

before any user sees the treatment.

Shadow runs cannot show whether B improves the outcome that matters most, since users never see B and so cannot react to it. They are a stepping stone: realistic inputs and real runtime behavior, with none of the product risk that comes with a live experiment.

Not every check needs an LLM judge

When the agent finishes a scenario, the result still needs evaluating, so the first version has two kinds of evaluator.

The first kind is deterministic. When a task means issuing a single refund and resolving a ticket, the environment can simply be inspected:

refund_count == 1
ticket_status == resolved

Asking another model whether the task was probably completed adds nothing when the resulting state can be checked.

The second kind is model-based, for properties that resist assertions:

helpfulness
clarity
tone
quality of explanation

An LLM judge helps there, but the judge becomes another versioned component and never an anonymous source of truth:

evaluator
evaluator version
judge model
judge prompt
rubric

Every generated score keeps that provenance. A small human-labeled calibration dataset is maintained too, and the judge is compared against it periodically. An LLM judge is a measuring instrument, and when the instrument changes, it matters whether its readings changed as well. Agentic judges that browse, call tools or dig up extra evidence are useful and probably unavoidable in some domains, but they have no place in the first version.

Statistics, kept small

Mature experimentation systems offer a great deal of sophistication. A better starting point is a small set of methods implemented correctly, in place of an impressive statistical menu that few users understand. The first version supports:

A/B allocation
binary metrics
continuous metrics
95% confidence intervals
sample ratio mismatch detection
fixed-horizon analysis

That is enough to answer many useful questions, and a result summary could read:

Experiment
support-agent-v18

Randomization
user

Primary metric
ticket resolution

A       81.4%
B       85.1%

Difference
+3.7 percentage points

95% CI
[...]

Experiment health
SRM PASS

Sample ratio mismatch deserves special care, because results should never look authoritative when the randomization itself seems broken. If the plan was an even split and the observed split clearly disagrees, the experiment is labeled unhealthy and that is the end of it, with no winner shown next to a warning icon.

Operational guardrails belong in that same health layer. A variant can fail on cost without failing technically. Imagine a prompt loop that pushes calls from:

4 model calls

into:

47 model calls

and the response may still be HTTP 200, even with a good answer. So hard caps are needed:

max cost / execution
max tokens / execution
max agent turns
max tool calls
max variant spend / hour

Crossing a cap can halt that execution or pause further exposure to the variant. A health report might then read:

Variant B

SRM                PASS
Error rate         PASS
Cost / request     +312%

Circuit breaker    TRIPPED
New exposure       PAUSED

A circuit breaker is no statistical conclusion. It is an operational safety mechanism that keeps a pathological treatment from eating traffic and budget while the experiment is still collecting enough observations to say anything statistically.

Fancier methods can wait: CUPED, sequential testing, holdouts, experiment layers, cluster randomization, adaptive allocation and more. The data model should leave space for them, but version one only has to run plain A/B experiments correctly.

What to build first

With most deferrable items removed, the platform is manageable. It needs a control service holding:

• an experiment registry
• a pipeline registry
• a dataset registry
• an evaluator registry
• metric definitions

On top sits the SDK, which downloads experiment configuration and assigns deterministically on the client side. An ingestion path is needed for:

assignment
exposure
outcome
feedback
evaluation

OpenTelemetry traces are ingested as well. Reasonable storage choices are PostgreSQL for control-plane state, NATS JetStream or Kafka for events, ClickHouse for experiment and trace analytics, and S3-compatible object storage for bulky artifacts. An offline runner plays a dataset against A and B inside environments that can be reset.

Much is deliberately absent. The MVP does not tackle arbitrary stateful environment virtualization, subtle interplay between experiments, evaluator drift, automated root-cause analysis, lasting treatment effects or causal attribution across several agents, and it can do without them. The things that matter from the start are those that are painful to fix once historical data depends on them:

what was randomized
what treatment was assigned
what treatment actually ran
what pipeline version produced the execution
what outcome was measured
how that outcome was evaluated

Provided those identities survive version one, the platform can become much richer without altering what its old experiments meant.

The helpful insight is that offline evaluation, shadow execution and live experimentation can share pipeline definitions, traces and evaluator tooling, though each answers its own question. Offline runs reveal how A and B behave in controlled scenarios. Shadow runs reveal how B copes with production inputs while users stay unaffected. Live experiments reveal what changes when real users are randomly split. Tracing records what went on inside each execution and helps interpret outcomes, but a trace-level diagnosis should never pass for a causal estimate. Together they support an MVP that can reliably answer three questions:

What exactly did we run?
What happened when we ran it?
Did assigning users to B improve the outcome we care about?

Everything else can grow around that.