When the Dashboard Is Green and the Agent Is Wrong: Tracing State

A green dashboard and a wrong answer

Debugging AI agents has an irritating quirk: the monitoring screen can swear that everything is fine. The model call returned, the tools finished, the APIs answered with 200s, latency looks ordinary and token usage sits where you'd expect. From an infrastructure viewpoint nothing is happening, and still the final result can be flat wrong.

That isn't the obvious flavor of wrong, where a model invents something absurd. Those failures are easier because at least you know where to start. The harder kind is where each step looks reasonable, nothing crashed technically, and the answer is believable enough that somebody may well act on it.

The problem grows as systems evolve from plain LLM calls into agents and multi-step workflows. Before a response appears, its underlying information may have crossed several models, tools, APIs, transformations and routers, so the final model might be doing perfect work on state that was corrupted five steps back. This corner of agent observability needs a lot more attention than it gets.

A schema change nobody notices

Consider a routine customer success job. The agent is asked to find customers whose renewals fall within 30 days, flag accounts that look likely to churn, and draft a report for the success team. Along the way it may query a CRM, compute a risk score, fetch recent support threads, check product usage, and finally have an LLM write the report.

Now suppose the CRM alters one of its response schemas. Yesterday the payload looked like this.

{
  "health_score": 42
}

Today it looks like this.

{
  "health": {
    "score": 42
  }
}

The API is still up, the request still succeeds and the JSON still parses. Depending on the integration code, no exception need be thrown anywhere. The damage comes later, when the normalization layer keeps searching for health_score and the next step is handed null.

Maybe risk scoring reads a missing value as neutral, or maybe the prompt says to use whatever is on hand. In both cases the workflow continues and ends with a believable report that leaves out an account the team needed to know about.

Staring at the final generation makes the model an easy culprit, and people end up rewriting prompts, switching models or adjusting temperature without touching the real cause. A better question asks when the state first became wrong, and with that question debugging an agent looks very different.

Success events don't prove understanding

One recent essay about the web turning into an API for agents describes software moving from reporting information to acting for a user. Its claim is that exposing tools to a model is only part of it, because intent, permissions, runtime controls and auditability matter once agents take real actions.

The same applies to debugging those actions. Noting that a tool executed is helpful, but it doesn't reveal whether the rest of the workflow understood what the tool produced. Imagine a trace containing this.

crm.search_accounts
status: success
duration: 184ms
records: 47

What you learn is that the CRM answered with 47 records. What stays unknown is whether all 47 arrived intact at the next step. Maybe six further pages existed and pagination ended after the first. Maybe an amount switched from dollars to cents, or a UTC timestamp was later read as local time. Maybe an enum changed spelling from "cancelled" to "canceled," or an API started reporting application-level errors inside a response whose HTTP status looked fine.

Ordinary software bugs, all of them. With agents the twist is that the following model typically proceeds regardless, taking in data that looks believable and reasoning over it. Hence a stronger model won't necessarily fix anything. If the input is wrong on arrival, the model just solves the wrong problem with more skill.

At the boundaries that matter, you should be able to look at the input, the output, the transformation applied to the response and what the following step actually got. Keeping every raw payload forever isn't necessary. Keeping enough to rebuild the run after a failure is.

Tracing a missing competitor

Tooling is getting better. OpenTelemetry's GenAI semantic conventions are beginning to standardize how model calls and tool execution are described, and frameworks like the OpenAI Agents SDK trace generations, tool calls, handoffs and guardrails. That's useful raw material, but making a trace helpful is still hard when the workflow completed and the actual failure is hidden in state passed between two healthy steps.

Consider a simplified run like the one pictured.

Suppose the final report lacks a competitor that obviously belongs. The old instinct was to open the final prompt and inspect the model's response. Maybe the instructions weren't strong enough, or the model ignored part of the context, or another model would do better. Starting there is no longer the right move.

Work backwards instead. Did the final model get the missing competitor? If it didn't, did the ranking step? If it vanished ahead of ranking, was it there after extraction? Present after extraction but absent after normalization means the failure sits at exactly one boundary.

{
  "company": "Example Corp",
  "website": {
    "url": "https://example.com"
  }
}

while the normalization step expected this.

{
  "company": "Example Corp",
  "website": "https://example.com"
}

Nothing about it was dramatic. Midway through, an object changed form and one step mishandled it, yet most of the operations nearby still looked fine. Bugs like this deserve tracing, since without it blame lands on whichever component is most visible, regardless of what happened upstream.

The practical lesson isn't that any specific observability product is required. It's that a trace should let you follow state through the workflow, and shouldn't just list which components ran. There's a big gap between knowing Step 4 ran and knowing what Step 5 received from it.

Plan drift looks sensible

Once you locate a tool-boundary failure it's fairly concrete. Plan drift is trickier, since an agent can leave the original task through steps that each look sensible. Imagine an agent told to research five suppliers, weigh their pricing and reliability and write up a recommendation, with an instruction not to contact them.

The agent finds the suppliers and begins collecting data. Two of them list no prices, so it hunts for another way around that. Eventually it spots an available email tool. Its logic isn't absurd: pricing is needed, pricing isn't public and a message to the supplier would settle it. It also breaks one of the task's most important rules.

When the rule is just a sentence near the top of a growing context window, everything depends on the model still treating it as vital after thousands of tokens of search results, tool output and intermediate state have piled on. Constraints with real consequences deserve an explicit form.

{
  "objective": "Compare suppliers and prepare a recommendation",
  "constraints": [
    "Do not contact suppliers"
  ],
  "completed_steps": [
    "discover_suppliers",
    "collect_public_pricing"
  ],
  "next_action": "request_missing_prices",
  "requested_tool": "send_email"
}

Now the application has something to check before an action happens. It can reject the tool call, demand human approval or send the agent off to find another route. The right response depends on the workflow, but the key point is that the constraint no longer lives entirely in prompt wording.

Rules like a spending cap, "draft but don't send" or "never touch production" work the same way. Plain language is good for conveying what a user intends, but the permissions and limits that matter should end up as runtime controls as well.

Software handles this everywhere else. Nobody would write "please don't let unauthorized users delete the database" in a comment and call it access control, and agents don't deserve an exemption because their instructions are in English.

Model routing adds a boundary

Routing creates a similar issue. Choosing a model stops being a simple configuration value once quality, cost, latency, tool use and differing model capabilities have to be balanced across a production system, and the routing decision becomes part of the trace you need for debugging.

Say Step 6 usually runs on Model A. During a live run the provider is rate limited, and the router switches to Model B. That's the fallback working as designed, since the workflow stays alive rather than failing. Yet Model B formats the structured arguments for the next tool slightly differently. They stay valid and the tool succeeds, but one field gets read in a way that alters the downstream result.

If observability reports only the model configured for the workflow, someone may spend hours trying to reproduce the failure on Model A when Model B served the request that broke. So a trace should record the requested model, the one that responded, any fallback, the active routing policy and whether the call was a retry. It isn't exciting information, but most helpful production debugging isn't. Beyond demos, much of the job is discovering which dull detail changed unnoticed.

Trace state, and not only events

Agent tooling now does a good job of showing the shape of a run, with model calls, tools, handoffs, retries and other events in increasingly handy trace views. The mistake would be to assume that recording more events automatically produces good observability. A run with ten beautifully instrumented spans still leaves you stuck if you can't tell what state moved between them.

A short list of questions helps with a bad run. What did the user originally want? Which constraints applied? What plan did the system follow? What did each tool return and what did the next step receive? Which model made each decision? Did anything in the outside world change? Where did the output stop matching expectations?

Answering these takes less than all the data, and capturing everything brings its own problems. Prompts, tool arguments and tool responses can carry private customer data, credentials or other sensitive content that shouldn't land in an observability platform without thought. OpenTelemetry's GenAI guidance flags this, and its detailed span specification makes capturing model inputs and outputs opt-in and not a default.

The aim, then, is to make the relevant parts of a run reconstructable, not to log all of it. Sometimes a redacted response will do, sometimes a schema version plus an object ID, and sometimes an intermediate result needs a snapshot because nothing else can recreate the run. Decide on purpose, so that an incident doesn't expose status: success as the only thing you kept.

What this means for evaluation

A trace explains what happened but not whether the outcome was good, so evaluation remains necessary. Anthropic argues the same in its article on agent evals. Agents are harder to grade than single replies because they run over several turns, call tools, change state and adapt to what came before.

Some results can be checked with no extra model. An agent told not to send an email that sends one has failed, and an extraction job meant to return 27 records that returns 11 is measurably off. Others are fuzzier: Was a research report thorough enough? Were the sources any good? Did a support answer fix the customer's problem? For those, human review, user feedback, datasets or model-based graders may be needed.

The real value comes from joining evaluation to the trace. A thumbs-down next to a stored final answer only says the user disliked it. Tied to the whole run, the same signal lets you look at the retrieved material, the tools that fired, the models involved and the spot where state changed.

Once the true failure is found, that production run keeps its value after the incident, because it can become an evaluation case guarding against a repeat after future prompt, model or workflow changes. It's far better than patching bad answers one at a time and hoping nothing else breaks.

Stop starting at the final prompt

Probably the largest shift in debugging AI systems is this one. In a simple LLM app that gives a bad answer, reading the prompt and response is reasonable, because there's little else where a problem could live. As systems grow, that habit pays off less. Picture a pipeline that has several models, retrieval, search tools, outside APIs, routers and a verification loop: its last prompt may simply be where an earlier error finally surfaced.

RAG has taught the lesson already. When retrieval drops the wrong document into the context window, a larger generation model won't repair it, and you trace back to fix retrieval. Agents follow the same logic with extra boundaries. Retrieval fine? Look at what became of the result. That fine too? Look at the next decision. Right tool picked? Look at arguments and output. State still right? Keep moving until you hit the first spot where it departs from what you expected.

It sounds plain on paper, yet people forget it because the final response is what's in front of them, and it's usually the simplest thing to blame.

Observability for the whole run

Agents are only going to get more complex. MCP exposes more tools to them, multi-agent designs add delegation and handoffs, routers pick models on the fly, and long-running workflows bring persistent state, retries and resumes. Human approval may sit in the middle of a run, and browser or computer-use agents can alter systems well beyond the chat. Each of these makes agents more capable and also hands state more chances to go wrong while every component reports success.

That's why observability has to go beyond LLM metrics. Latency, token counts, errors and cost matter, and a dashboard should certainly warn when an agent loops and burns API credit at 2am. Still, those figures mostly tell you whether the machinery is turning. A multi-step agent also needs a way to confirm that what flows through it continues to fit the user's task.

In practice that means tracing the initial intent, key constraints, the tools and models that really ran, the state handed between steps and the final result. Given a wrong outcome, you should be able to walk backwards to the first point where state parted from reality. With that ability a bad run becomes an ordinary engineering problem you can investigate, reproduce and test. Lacking it, you stare at the final answer and guess which piece of the system failed earlier.