When an AI Agent Fails, Who Do You Blame? Tracing the Failure Stack
Why 'the model hallucinated' falls short: a walk down the agent failure stack from intent and retrieval to tools, retries and traces, and what to observe.
This article is published in English.
Picture asking an AI agent to book a flight to São Paulo for tomorrow morning, around 9 a.m. A few seconds later it replies that the flight has been booked. Except it hasn't. No ticket, no confirmation email, no charge.
The tempting verdict is that the AI hallucinated. Maybe it did. But maybe the model behaved exactly as designed. The booking API could have returned an error the agent misread. The agent might have searched for flights without ever calling the booking tool. The API might have timed out after completing the transaction. The tool might have returned a quote and not a confirmed reservation. Or "tomorrow morning" might have been read in the wrong timezone.
Once AI systems start doing things and stop merely answering questions, failures get much harder to explain. That creates a fresh engineering problem: when an agent fails, who or what takes the blame?
Old software was easier to debug
Conventional software can be intricate, but its path of execution is usually familiar:
User
↓
Frontend
↓
Backend
↓
Database
When something breaks, the places to check are well known: whether the request got to the backend, whether the database went down, whether a downstream API stalled, whether an exception was raised. Logs, metrics and traces help piece the story together.
An agent looks quite different:
User Request
↓
Interpret Intent
↓
Retrieve Context
↓
Create a Plan
↓
Select a Tool
↓
Call the Tool
↓
Interpret the Result
↓
Reason Again
↓
Final Answer
Here a successful HTTP response says very little. The system can return
200 OK
and still do something entirely wrong. That's a core difference between classic monitoring and AI observability: a system can be healthy in operational terms and wrong in behaviour.
"The model hallucinated" is the new "the computer is broken"
Nearly every bad AI result gets labelled a hallucination, and the label wears thin fast. Say an assistant claims your company grants 30 days of paid vacation, when the real policy says 22. A hallucination? Possibly. But suppose the right policy sat in the document store and retrieval pulled an outdated version. The model may have answered flawlessly given the context it was handed. The model didn't fail. The context pipeline did, and swapping the model won't repair the system.
The agent failure stack
A helpful mental model treats an agentic system as a stack of places where things can go wrong:
┌───────────────────────────┐
│ User Intent │
├───────────────────────────┤
│ Prompt / Instructions │
├───────────────────────────┤
│ Context / Retrieval │
├───────────────────────────┤
│ Memory │
├───────────────────────────┤
│ Planning / Reasoning │
├───────────────────────────┤
│ Tool Selection │
├───────────────────────────┤
│ Tool Parameters │
├───────────────────────────┤
│ Tool Execution │
├───────────────────────────┤
│ External Systems │
├───────────────────────────┤
│ Orchestration │
├───────────────────────────┤
│ Infrastructure │
└───────────────────────────┘
Call it the agent failure stack. The point is that the final answer is merely the visible tip of a much longer execution path.
1. The agent understood the wrong thing
Tell an assistant to cancel Friday's meeting with John. It finds a recurring event called "Weekly Sync with John" and deletes the whole series. The calendar API worked, authentication worked, the deletion succeeded. The failure happened before any tool was called:
User Intent
↓
Wrong Interpretation
↓
Correct Tool Call
↓
Wrong Outcome
That's an intent failure, and it's dangerous because everything downstream of the initial mistake can run perfectly.
2. The information was there, but the agent never saw it
Take a legal assistant reading a contract that says either party may terminate after 90 days. Retrieval sends the model a neighbouring clause about a 30-day notice period, and the AI concludes the contract can end after 30 days. Again the model may be reasoning soundly from bad input:
Documents
↓
Retrieval
↓
Wrong Chunk
↓
Model
↓
Wrong Answer
The fault lies in retrieval and not necessarily in generation. Chasing a RAG bug often resembles tuning a search engine more than fixing a chatbot.
3. Memory can leave a smart system confidently out of date
Agents increasingly carry memory, which is useful right up until it goes stale. Say the agent remembers
Customer Plan = Basic
but the customer upgraded last week and the true state is
Customer Plan = Enterprise
The agent refuses a feature the user is entitled to. Nothing crashed, and the memory store did precisely what it was built for. It simply held onto something no longer true. Sometimes forgetting matters as much as remembering.
Tools raise the stakes
LLMs generate text. Agents generate actions, and that changes the risk. Give an agent two tools:
search_customer()
update_customer()
The user asks what address is on file for Sarah. The obvious tool is
search_customer()
yet the model picks
update_customer()
That's a tool-selection failure. Or it picks the right tool and passes
{
"currency": "USD"
}
where the correct value was
{
"currency": "BRL"
}
That's a tool-parameter failure. The API may run without a hitch, and the result is still wrong.
Sometimes the tool breaks
External systems are messy. APIs time out, databases drop offline, rate limits kick in, permissions expire. Consider this chain:
Agent
↓
Book Flight API
↓
TIMEOUT
How should the agent react: try again, ask the user, or first verify that a reservation exists? Now suppose the booking did go through but the confirmation was lost on the way back, so the agent tries a second time:
First booking → SUCCESS
Response → LOST
Retry → SUCCESS
Now the customer holds two tickets. This stopped being only an AI problem. It's a classic distributed-systems problem, and retries, idempotency, state management and transactions become part of AI engineering.
Multi-agent setups make blame murkier
Now imagine three agents in sequence:
Research Agent
↓
Analysis Agent
↓
Writing Agent
The research agent finds the right facts. The analysis agent reads them correctly. The writing agent drops an important qualification, and the final answer misleads. Which agent failed? Agentic systems increasingly look like distributed software whose components make probabilistic decisions, and that makes observability essential.
Traces matter more than logs
Suppose an agent produces a terrible answer while the infrastructure dashboard reads:
HTTP Status: 200
Latency: 3.8s
CPU: Normal
Memory: Normal
Errors: 0
Everything looks healthy, yet the answer is wrong. Classic monitoring tells you the system ran. Engineers need to know why it made the decision it did. That calls for something closer to an agent trace:
TRACE #84721
User Request
↓
Intent: "Book Flight"
↓
Retrieved user preferences
↓
Planner created 3 steps
↓
search_flights()
↓
Selected Flight #2
↓
book_flight()
↓
API response: timeout
↓
Agent assumed success
↓
Final response:
"Your flight has been booked."
Now the failure is actionable. The agent didn't just invent a flight. The system treated an unknown tool state as a completed transaction, and that's something engineers can fix.
What a trace should capture
A useful agent trace could contain:
trace_id
user_requestmodel_versionprompt_versionretrieved_contextmemory_usedtool_callstool_parameterstool_resultsretrieslatencytoken_usagecostfinal_answerevaluation_result
Not every application needs to store every field, since privacy, security and cost still count. The principle is plain, though. If an agent takes ten steps to reach an answer, recording only the first input and the last output is like debugging a microservice architecture while logging just the first request and the final response. Not much help.
Monitoring is not observability
The distinction is especially handy for AI systems. Monitoring asks whether something is wrong. Tracing asks what path the task took. Observability asks why the system behaved that way. Evaluation asks whether the behaviour was actually good.
Monitoring
"Is the system healthy?"
↓
Tracing
"What path did it follow?"
↓
Observability
"Why did it behave this way?"
↓
Evaluation
"Was the result correct?"
AI engineering needs all four.
Metrics have to evolve too
Conventional services are measured with figures like
Availability
Latency
Throughput
Error Rate
and those still matter. Agents add new questions:
Task Success Rate
Tool Success RateRetrieval Success RateTrajectory Success RateHuman Escalation RateAverage Steps per TaskTokens per TaskCost per TaskCost per Successful Task
The last of those is particularly telling. An agent that costs $0.01 per request but succeeds half the time may be a worse deal than one that costs $0.03 and succeeds 98% of the time. The cheapest AI system isn't necessarily the one on the cheapest model. It's the one that delivers the cheapest successful outcome.
So who gets the blame?
Back to the flight. The user asked for a booking to São Paulo tomorrow morning, the agent said "Done", and no ticket exists. The model? Maybe. The prompt? Possibly. Or retrieval, the booking tool, the airline API, the orchestrator, the infrastructure?
The uncomfortable answer is that you can't tell until you can reconstruct the execution. This marks a major turn for AI engineering. Once models turn into agents, and agents into whole systems, reliability stops being a property of the model alone and becomes a property of the whole execution path:
Model Quality
+
Context Quality
+
Tool Reliability
+
Orchestration
+
Infrastructure
+
Evaluation
+
Observability
↓
Reliable AI System
Stronger models will certainly help, but capability alone won't make AI systems dependable. The dull things software engineers have spent decades learning to respect still apply: logs, traces, timeouts, retries, permissions, state, testing, metrics and failure analysis. When an agent fails in production, "the model hallucinated" will eventually sound as useful as "the computer did something weird." Engineering begins at the point where that explanation stops being good enough.