Why Production AI Agents Fail Quietly and How to Catch Wrong Answers

The most dangerous production failure for an AI agent is not a crash or a timeout. It is an answer that looks right, passes every health check and is wrong. The case study below follows a small software company that put twelve agents into production in a single quarter and kept only three, and it explains what separated the survivors from the rest so you can design detection before you deploy.

The team, the tools and the ground rules

The company was a B2B SaaS business of about forty people, with nine engineers, not a research lab. Between 6 January and 27 March 2026 the team launched twelve agents. The ones that worked inside the code repository ran on Claude Code; the others were custom agents built on the Anthropic API and hosted behind a small internal service, so that every agent shared a single audit log and a single kill switch.

Two rules applied from the first day, and both proved worth keeping:

• Every agent logs every action to an audit channel, including read-only actions.
• Anyone on the team may switch off any agent at any time, with no approval and no ticket.

Success was defined in an intentionally demanding way. An agent counted as a success only if it was still running after thirty days and someone pushed back when its shutdown was proposed. Usage metrics are easily inflated by novelty; people defending a tool they depend on are a much harder signal to fake.

Scorecard: three survivors, nine shutdowns

Three agents were still running at the end of the quarter:

• a pull request description writer
• a drafting assistant for support ticket triage
• an incident context gatherer

Nine were switched off within a month: an automated code reviewer, an analytics question answerer, a dependency upgrade bot, a flaky test quarantiner, a customer email responder, a converter from meeting notes to tickets, a wiki documentation updater, a log anomaly responder and a changelog publisher.

The guardrail prompt that did not help

Like most teams, this one began with a guardrail written into every agent's system prompt, telling the model to admit uncertainty, to avoid unverifiable values and to cite a source for every number:

If you are not confident in an answer, say "I don't know" and stop.
Never state a value you cannot verify from the tools available to you.
Cite the source of every number you report.

In practice, this instruction had almost no effect, and the reason why is the most important lesson of the whole exercise. It is covered in detail in the analytics failure below.

Permissions also started conservatively: every agent was read-only at launch. Over the quarter, four agents were granted write access. Three of those four were later shut down.

What the surviving agents had in common

The pull request description writer

Running on Claude Code, this agent could read the diff and the linked issue and then wrote a description into the pull request body. An engineer edited it and merged. It handled roughly 35 pull requests per week, and the descriptions ended up better than the ones the engineers wrote by hand, largely because a worn-out developer at the end of the day tends to write "fix bug", while the agent does not get tired.

The support triage drafter

This agent read each incoming ticket, applied a label and drafted a reply as an internal note inside the helpdesk. It had no ability to send anything, ever. Roughly 60% of its drafts went out after light editing, and median first response time fell from a little over four hours to about eighty minutes.

The incident context gatherer

Whenever an alert paged someone, this agent posted exactly one message to the incident channel containing the three most recent deploys with timestamps, the change in error rate per service, and links to any similar incident from the previous ninety days. It took no action and offered no diagnosis; it simply collected the dashboards and links an on-call engineer would otherwise open by hand. It was the least sophisticated agent the team built and the one the team valued most.

Anatomy of a confidently wrong agent

On paper, the analytics agent was the best-engineered of the twelve. It had access to a read replica, a hand-written schema document in its context and a Slack interface, and its purpose was to answer metric questions so the data team would stop being interrupted twenty times a day.

On 3 February someone asked it for weekly revenue. It generated a query that joined orders to payments, and in doing so it treated refund rows as positive amounts. The result was about 12% too high, yet entirely believable: the magnitude was right, the week-over-week trend was right, and even the seasonal dip after a January promotion ended was visible.

That inflated figure appeared in the Monday metrics post, and then in the next two Monday posts as well. The error surfaced only on 24 February, during the January close, when the finance team's total and the agent's total would not reconcile.

Consider what did not happen during those three weeks. There was no exception, no alert and no latency spike. The SQL was valid, rows came back, and the agent cited its source exactly as its prompt demanded. Conventional monitoring exists to detect systems that stop working, and this one never stopped.

This is precisely where the guardrail prompt breaks down. An instruction to say "I don't know" when unsure only works if the model has some internal signal of its own uncertainty. Here the model was not uncertain; it was mistaken. From the outside, being confidently mistaken is indistinguishable from being correct, so a prompt cannot filter it out.

A useful general point follows: a join that silently changes the sign or multiplicity of rows is a classic analytics bug for humans too. The difference is that a human analyst usually has a sense of which numbers finance will check, while an agent has no stake in reconciliation unless you build one for it.

When a correct agent is ignored

The automated code reviewer failed in a way many teams do not anticipate: it was not wrong. Each pull request received around 40 of its comments; most could be justified, and many of them nitpicks about naming or error-handling style.

Within three weeks engineers were resolving its comments without reading them. Then it raised a genuinely serious issue, a query without its tenant filter, and that comment was buried among 38 style remarks. A person found the bug in staging two days later. The reviewer had been right, but excessive volume had destroyed its signal.

Why the nine were shut down

The failures broke down cleanly:

• six produced confident but wrong output
• two produced output nobody read
• one was switched off because nobody could tell whether it was doing anything at all

The design rule: stop one step before the human acts

The three surviving agents share one property, and it is not the model, the prompt or the framework. Each of them stops one step short of a human action. They produce a draft, a summary or a bundle of context, and a person performs the final step. Performing that step forces the person to read the work.

Every agent that was shut down either acted on its own or produced output that turned into an action without real review. The analytics agent is the telling example: it had no write access at all, yet its number flowed straight into business decisions through people who trusted it.

The resulting boundary is simple. Let agents assemble material; do not let them own the final step, where a mistake becomes a real consequence.

That boundary is not a judgement about model capability, and a stronger model will not remove it. It is about detection. An agent's typical failure is a plausible answer rather than a crash, and most teams have very little tooling for spotting plausible answers that are wrong.

Cost was never the bottleneck. All twelve agents together consumed a little under $900 in tokens for the quarter. The scarce resource was human attention.

Three changes you can make this week

• Before shipping, document how a confidently wrong answer would be spotted. Not how you would notice a crash, but how you would know the output is wrong. If you cannot write that sentence, the agent should draft rather than act.
• Reconcile every number an agent produces against a second source on a schedule. A weekly automated comparison against the finance figure would have caught the revenue error on its second day, and building it would have taken an afternoon.
• Deliver agent output where people already look, and limit how much of it there is. A brand-new channel tends to be muted, and a wall of forty comments is skimmed, while a handful of focused comments actually gets attention.

Reconcile the number against a second source

The revenue agent in this case study needed a check that does not ask the model whether it feels confident. Compare the figure it produced with the ledger and fail closed when the gap is larger than half a percent (or one currency unit, whichever is greater).

const reconcile = (agentRevenue: number, ledgerRevenue: number) => {
  const delta = Math.abs(agentRevenue - ledgerRevenue);
  const tolerance = Math.max(1, Math.abs(ledgerRevenue) * 0.005);
  return { ok: delta <= tolerance, delta };
};

Run that comparison on a schedule. A crash monitor stays green while this check is the one that pages a person.

Key takeaways

• Plausible wrong output, not downtime, is the failure mode to design for; standard monitoring will not see it.
• Prompt instructions about confidence cannot catch errors the model is not aware of.
• Read-only access is not the same as harmless: output that feeds decisions is effectively an action.
• Volume is a failure mode of its own; a correct signal drowned in noise is worth nothing.
• A useful test for any agent is whether anyone would object if it were switched off tomorrow.