A Production Readiness Checklist for AI Agents, Stage by Stage

Getting an AI agent from a working prototype to a live deployment follows a predictable path, and each stage has its own checks and its own ways to go wrong when skipped. This walk-through goes stage by stage.

A prototype has one job, which is to show an idea works: one machine, state in memory, no real users. Production asks for more. It has to survive restarts, serve several people at once, recover when a step fails halfway, and leave a record that can be traced when something breaks.

Little of that gap concerns the AI itself, since the model call is usually the easy bit. The harder work surrounds it: where state lives, what happens when a step runs twice, and how a change reaches a real user without harm. Use the lists below to see what is already covered and what still isn't before an agent goes live.

Stage 1: dev and local prototype

The aim here is deliberately narrow, which is to prove the core idea works at all. Don't make it production-shaped yet. Doing that too early usually means building the wrong infrastructure for a design that hasn't settled.

Get the core workflow working once, as small as you can. Take a single clean run from input to correct output before queues, databases or deployment enter the picture.

Why: Early infrastructure choices are usually wrong because the true shape of the problem is unknown: run time, state size and failure modes are all still guesses. A production-grade version built first is normally rebuilt anyway.

Example: A support-ticket triage agent begins life as a short script: it reads one ticket, asks a model for a category and prints the answer. There's no server and no database, and it's launched by hand from a terminal.

Make behaviour configurable instead of hardcoded. When the system must act differently for different inputs, such as customer segments, document types, languages or tones, handle that as configuration rather than as another branch in code.

Why: Variations pile up fast once users show up ("does it handle French tickets?", "can enterprise accounts get a stricter tone?"). When each one needs a code change and a release, new cases queue up behind engineers, whereas with configuration they're a quick edit.

Example: In place of an if/else per customer tier, the agent's instructions are built from small text templates, one base and one per tier. Adding a tier means adding a file.

Write down what the prototype doesn't solve yet. Before moving on, list the gaps on purpose: restarts in the middle of a run, several users at once, failure logging, running anywhere except your own machine.

Why: A gap nobody recorded turns into a surprise later, usually discovered during an incident and not in a planning conversation. A recorded gap becomes the checklist for the next stage.

Example: A short table with rows such as "state lives in a Python dictionary, so durable storage is needed," "only one job can run at a time, so concurrent runs must be supported," and "no record of what happened, so an audit log is needed." Nothing is fixed at this point. The items are just named.

Stage 2: lower environment and staging

Here the system first meets real infrastructure and the real ways it fails. It runs off the laptop now, gets restarted, and may be used by several people at once.

Choose the runtime to suit the real workload. Look at hard numbers before deciding where the agent will live: the longest run allowed, the cost of sitting idle, and whether a killed process can pick up where it left off.

Why: Agent workloads rarely look like ordinary web requests. One run can last minutes and has to carry plenty of context across many steps. A runtime designed for short requests will time out mid-run, and one designed for constant traffic will cost too much for something that runs a few times a day.

Example: A legal-document review agent goes through a document's sections one at a time in a single run, so that it keeps the full context, and a long document can take 10 to 15 minutes. That ruled out a serverless function with a hard time cap. It also ruled out a set of always-on containers, since the agent runs only a handful of times a day and idle capacity would just sit there. Long, bursty, infrequent runs pointed to a runtime that bills only for active processing time and has no timeout.

Move state out of memory into durable, shared storage. Anything that must survive a restart or be visible to more than one server, such as run status, conversation history and in-progress results, belongs in a real datastore and not in a variable inside the running process. Choose one system as the source of truth.

Why: State held only in memory vanishes when the process restarts, and in production that happens more often than most people expect, thanks to deploys, crashes and scaling. State scattered across several stores with no clear owner ends with two systems disagreeing about what's true.

Example: Here a document-editing agent keeps run state, chat history and proposed edits in object storage, with nothing held in the request-handling process. Object storage is the single source of truth, and a database is used for one narrow task: a claim lock that stops two workers from processing the same run. General state isn't split across both with no rule for which one wins.

Assume "this ran twice" and "these ran simultaneously." Most real messaging systems can deliver one trigger more than once. A plain status check ("start if not started") fails because two processes can each read "not started" before either records "started." Rely on a mechanism that lets exactly one of them win.

Why: Without it, two processes may repeat the same costly work or write conflicting results, and the bug appears only when things truly overlap, which a quiet dev environment seldom produces.

Example: An early run-claiming lock worked under normal conditions but couldn't release a run if the worker holding it died or hung. In production that appeared as a pile of runs frozen at "processing" for days, with nothing to rescue them. The fix added an expiring lease, a token that stops a recovered worker being mistaken for the original holder, and a periodic sweep that reclaims any run whose lease lapsed without finishing. That's much more complete than "claim it once."

Add tracing and logging before an incident demands them. When more than one service is involved, such as a web layer, a queue, a worker and an AI runtime, one user action passes through every one of them. Arrange a way to follow that path across services now, so you aren't piecing it together mid-outage.

Why: Reconstructing events from separate logs that don't connect is slow and error-prone, especially under pressure. A system that can be traced from the start turns a stressful investigation into a quick lookup.

Example: A system's AI runtime had detailed tracing from the beginning, but the web-facing service in front of it didn't, so a trace showed everything the agent did and not the request that triggered it, since nothing shared linked the two. The fix was to create one run ID at request time and pass it through every service's logs, so a single search could gather the full path of a particular run even though the two tracing systems stayed separate.

Try the real client early. When users reach the system through one particular browser, app or device, test on that target soon, and don't settle for a generic API client or an approximation.

Why: Real clients have quirks, such as length limits, silent cutoffs and odd encoding, that a generic tool won't catch. They tend to surface only with real users, by which time they're a production incident and not something found in staging.

Example: A desktop app's document-open feature passed every automated test but failed intermittently for users on one operating system. The app was silently truncating download links beyond a certain length, and the temporary security credentials used to sign the links pushed them well past it. The fix was to sign links with a separate identity that produces a much shorter signature, in place of the temporary credentials the rest of the system used.

Match the front door's network path to how clients connect. If users only reach the system over a private network, such as a VPN, a zero-trust proxy or an internal-only network, make sure the address they use really leads to something on that same private path from start to finish. An internal-sounding domain that quietly resolves to a public load balancer or CDN can work for some users and fail intermittently for others.

Why: A mismatch like this fails inconsistently: some users, some of the time, so it resembles unreliable infrastructure and not a bad setting. It slips through because it can pass from a developer's machine, which often takes a different network route than a staff laptop.

Example: An internal tool was meant to be reached only over the company network via a zero-trust proxy, which sends traffic through different connectors depending on what each laptop selects. The tool's domain sounded internal yet pointed at a public content delivery network. Certain connectors could get to that public address and others could reach only private ones, so failures appeared and disappeared with the connector in use. The remedy was to host the front door on a load balancer inside the private network, reachable the whole way by the route real users take, plus a separate signed path for one desktop app that can't use the proxy at all.

Create a stand-in client that can drive the entire pipeline whenever you want. It's a script that makes the same requests and follow-ups as a real user, through the same interface, so complete runs can be repeated without waiting for a human tester.

Why: Agent flows are seldom one request and one response. They involve several turns, tool calls and updates streamed over time, which is hard to verify thoroughly by hand. Unit and integration tests alone tend to test pieces separately and not a complete realistic session. With a fake client, the question of whether a complete multi-step conversation went right becomes a check you can run at any moment.

Example: A load-testing harness makes the same API calls a real client dashboard makes to start a run, then checks a status table to see when it's done, because the run finishes later and the first response says nothing about the result. The number of simultaneous runs rises in steps, from a handful to dozens to over a hundred, with a hard cap on total runs and spend, since every fake run is a real billed model call. It revealed that the true bottleneck under load was the model provider and not the surrounding infrastructure, which testing pieces separately wouldn't have shown.

Pick the point where a person looks at the agent's output before it counts. Provide a place where someone sees what the agent is about to do and can approve, change or refuse it, in place of letting the output take effect by itself from the first day.

Why: Agent mistakes look different from ordinary bugs. They look plausible but wrong, not like an obvious crash. Human review catches the mistakes that pass every automated check, and it builds the track record needed to decide later where the agent can safely act unreviewed.

Example: A document-editing agent never writes into the final document. It offers suggestions that a reviewer accepts, revises or dismisses individually, so nothing it produces counts until a person has acted.

Stage 3: pre-production

This is the last environment before real users. The question moves from "does the feature work?" to "does the whole pipeline still behave when conditions aren't ideal?" That includes repeated deploys, partial outages and drift when the model or prompt underneath is swapped.

Build a proper evaluation, since pass or fail isn't enough. Decide what "good" means for the agent's output, which is usually several things (acting on the right input, being substantively correct, staying consistent from run to run), and who or what checks each: a fixed rule, a human reviewer or a separate model acting as judge.

Why: Agent output is seldom simply right or wrong the way a unit-test assertion is. A single pass/fail signal hides which part got worse, and a scoring method nobody has looked at closely, especially an automated judge, can drift quietly away from what matters to the people using the output.

Example: A summarisation agent gets scored on three separate things: coverage of the key facts, any invented content that isn't in the source, and how much its output varies between two runs on the same input. A person reviews a sample for the "invented something" check, since getting that one wrong costs the most.

Re-run the agent on a frozen set of known-good examples. Keep a small, unchanging collection of inputs with approved outputs, and run it at every deploy, especially after any change to a prompt or model.

Why: Unlike regular code, an agent's behaviour can shift subtly after a small prompt change or a model update, and no test throws an error. A fixed example set is often the only thing that notices "it runs fine, yet the results have slowly degraded."

Example: A document-editing agent is checked on a fixed example set in three ways: whether it picked the right sections to edit (compared with a human-reviewed answer key), whether its proposed edits were correct (scored by a separate judge model), and how much its output varies between runs on identical input. Tracking all three across versions, and not one pass/fail figure, revealed that one change had helped a single metric while quietly hurting another.

Let a failing check stop the release. A test suite whose failure still allows the deploy is just a comment. Connect the results to the pipeline so a failure halts the release, and bypassing it isn't the default.

Why: If a gate can be skipped without friction, it will be, usually under deadline pressure, which is when broken changes are most likely to get through.

Example: A pipeline is set up so that if any test fails, the deploy step never runs. There's no "deploy anyway" button, and getting around it takes a conscious change to the pipeline configuration and not a click through a warning.

Break the fast path on purpose and confirm the system degrades gracefully. If results are normally delivered through a fast path such as a cache, a live connection or a queue, switch it off deliberately and check that a slower fallback still works, rather than the feature just going dark.

Why: Caches, live connections and similar shortcuts are the pieces most likely to fail in production. With no fallback, a minor outage in one small component takes the whole feature down.

Example: Updates usually arrive within a second over a push connection. In a test with that connection turned off, the system polls every few seconds instead, a slower route that still shows the user their results.

Let tooling enforce ordering when one thing must exist before another. If part A has to be in place before part B can be configured, put that sequence inside the deployment process and not in someone's memory.

Why: A rule like "deploy X before Y" survives only until the person who knows it is away or someone automates the process without knowing about it. Sequencing that matters belongs in tooling.

Example: An orchestration service requires the ID of an AI runtime that a different deployment stack creates, and having each stack reference the other would make a cycle. So the runtime's stack writes its ID to a shared parameter store when created, and the orchestration stack reads it there. The runtime is guaranteed to exist first, and the stacks never reference each other.

Stage 4: the production gate

The final checklist comes before real users rely on the system. Correctness should be settled by now. What's left is confirming that what ships is precisely what was tested, and that failures can be recovered from and explained.

Ship the exact artefact that was tested. Take the build that passed pre-production and relabel it for production, instead of compiling a fresh copy at the last step.

Why: A last-minute rebuild, even from the "same" source, can pick up a different dependency version, timestamp or environment detail. That gap means production runs something that was never actually tested.

Example: The packaged app that passed every pre-production check is simply relabelled as the production version and shipped as it stands, the same file and not a new build from the same code.

Decide beforehand where repeated failures go and how long they're kept. Set a firm retry limit, choose where anything failing past it should land, and decide how long to keep it.

Why: With no limit, a broken job can retry forever, quietly burning resources or blocking a queue. With no landing place, a failed job just disappears, and nobody can act on a failure they can't see.

Example: A job that fails three times is set aside automatically in a separate holding area and not retried forever. It's kept for a fortnight, which gives an engineer time to spot it and investigate.

Plan the audit trail now. If the system's decisions may need explaining later to a user, auditor or colleague, settle in advance what is recorded and how to find it.

Why: "We'll add proper logging later" almost never happens once real deadlines arrive, and by the time the need is obvious the missing history can't be rebuilt after the fact.

Example: Every automated decision is saved as its own timestamped record. If someone later asks why it did something in a particular case, there's a written answer to point at and not a guess.

Write up each real incident, not just the patch. After an unexpected failure, record the root cause and fix where the team will find it later, which means more than a closed ticket or a chat thread.

Why: An undocumented fix gets worked out again from nothing by someone else, usually under more stress. A brief note is often the cheapest guard against repeating a multi-hour investigation.

Example: A batch of runs once sat half-processed for days because nothing recovered them automatically. The cause, the remedy and the thinking behind it were written into their own document, beyond the code patch. Later, a follow-up look reopened an earlier conclusion, one blamed on the model making things up, and found on closer inspection that the true cause was subtler than first thought. That update was written up too, so the original, less accurate conclusion didn't stand.

The pattern beneath the checklist

Hardly any of this is specific to AI. These are the habits every distributed system needs: durable state, safe handling of repeats, visibility into what happened and why, and a release process that ships what was tested.

The agent changes where the risk sits. Runs are longer and carry more context, so the compute must suit that. A run can fail or drift in ways an ordinary test misses, so regression checks on known-good examples count for more. And sooner or later someone will ask "why did it do that?", which the audit trail must be able to answer.