Agent Workflow Design Principles for Production Engineering Teams
Design principles for production agent workflows: completion contracts, honest state, approval digests, idempotent outside actions, tracing and evals.
This article is published in English.
Sources: What building agents taught me about production engineering
Putting agents into real business processes changes what you ask before trusting an automation. Where did the evidence come from? What was the system permitted to modify? If the run dies midway, what then? These questions outweigh the choice of framework, particularly once go-to-market chores like account research, email, content and follow-ups are connected to business systems. The principles below come out of that kind of work and are meant for the next build.
Picture an ordinary ask: investigate an account, write a worthwhile message, and leave it somewhere a person can look it over. The model can assist with the investigating and the writing. Everything around it, meaning the software, has to retain the evidence, handle review, and take responsibility for whatever changes outside the workflow. Preparing a draft may finish well before anything external is authorised, so evidence, authorisation and outcome each want a separate check.
Decide what "done" means before picking a framework
Different workflows define finishing differently. An account brief is finished when a document is saved with its sources, a relevance judgment and a note on what is still doubtful. An outreach flow is finished when the email provider has taken the very message a person signed off. Those are separate promises, and neither shows that the recipient opened the mail.
Once that's written down, the code looks different. Research may legitimately return too little evidence. A draft can be done while its review is outstanding. An outside action can be accepted by the provider and its final delivery still be open.
Let the model handle what needs interpretation, such as choosing a good source, spotting relevant context or explaining a recommendation. Let code enforce what has explicit rules: required fields, permitted tools and execution limits. A schema can confirm that a source reference exists, but whether the source supports the claim still needs an evidence check.
Here the contrast between workflows and agents earns its keep. Anthropic frames a workflow as a path fixed in advance, and an agent as a setup where the model steers. A single product may mix both, so the real question is where freedom pays for its price.
Make state say what really happened
The data model needs thought before the workflow grows. One field called status turns ambiguous fast once it must describe research, review, execution and delivery together. Keeping those apart has practical value. If a draft is saved but the review notification fails, the next step can retry the notification without producing another draft, and anyone inspecting the run can see both facts.
Approval wants the same exactness. In this setup it is tied to the operation, the account, the recipient and the entire message body. A workable approach keeps the payload immutable and derives a digest from a canonical version. Before dispatch the executor compares what it's about to do with what was approved, checks expiry and current rights, and sends that same stored payload. The digest only spots alteration. Who may approve is a matter for access control.
A model must not be able to approve itself by returning approved: true. The application records authorisation through a trusted review path. Some low-impact tasks can use an explicit standing policy instead, and in either case the permission should stay scoped and checkable.
One logical owner per run
Frameworks like LangGraph help you think about state and transitions, while a workflow tool such as n8n handles triggers, routing and predictable data movement. A different question appears when work crosses runtime boundaries: which component is responsible for the run?
In this design one logical owner takes care of scheduling, the identity of a run and the authoritative state. A delegated runtime is handed a bounded job and answers with structured output, which the owner verifies before it moves the workflow ahead. That is an application pattern and says nothing about how any particular tools integrate.
For an account brief, the delegation could name the account, the permitted sources, the tool permissions, the output schema and a time or cost budget. What comes back should tell a finished brief apart from insufficient evidence, a tool failure or a hit limit, so the parent workflow can react to each without treating the closing paragraph of a model's answer as a control signal.
The surrounding software is the agent harness: code that supplies context, exposes tools, manages execution and keeps progress. Anthropic's work on long-running agents offers an example, showing how explicit progress artifacts help later sessions pick up unfinished work. Which artifacts to keep depends on the task.
State for a business workflow needs a lasting record kept outside the model's chat. LangGraph's docs draw a line between checkpointed thread state and data kept across threads, and caution that memory-only checkpoints are lost on a restart. Which storage backend to use is an ops call. Context is the right home for a remembered style preference, and the permission system is the right home for current authorisation. A fetched page or a reusable skill may inform a task, yet should never be able to enlarge what the task may do. Multiple workers can serve one logical owner, provided coordination blocks clashing state transitions.
Give each step its own context
Research needs access to sources and a place to log doubt. Writing needs the approved brief, samples of effective messages and the house rules. When every step gets context that fits it, its behaviour is simpler to assess. Different roles can be settings on one runtime and needn't mean extra agents or extra infrastructure.
A skill packages a procedure you repeat: sources to check, what output to make, how to confirm it. Enforcement still belongs to the runtime, through tool access and resource caps. Telling a worker in a skill file not to send is merely advice, whereas leaving send credentials out of its hands is a limit that actually holds.
Piles of skills and settings need periodic review as well. Two sensible instructions may collide when they're loaded together. After a run an agent can suggest a memory update, but its self-assessment doesn't prove an improvement. A correction should carry its origin, a suitable scope and an evaluation before it alters behaviour that others share.
Treat outside actions as their own execution problem
Describe tool effects precisely. Writing a local artifact, making a draft at the provider and sending a message are three operations, each with its own permission. Adapters should turn provider replies into explicit statuses, and integration tests should inspect the provider's resulting state.
For writes with consequences, route through an execution service that verifies authorisation at the last moment before dispatch. Credentials and network paths must back that up, because a check in one helper does little if a different script can reach the provider with the same credentials.
Recovery is hardest when no reply comes back. Imagine the provider takes a message and the connection fails before the worker stores the result. The worker knows it tried but can't tell whether trying again would duplicate it. Keep one identity for the action across retries and give each attempt its own. Where a provider supports idempotency, it relies on a stable key per intended operation, and the guarantee comes from that provider's contract. Stripe documents, for example, how parameters are compared and how long keys live, and reusing a key after it's gone can produce a fresh request.
If there's a dependable status lookup, use it. If the provider promises safe replays for the same key and payload, a retry may settle the matter. Failing both, record the outcome as unknown and reconcile or ask a human. A lookup that finds nothing isn't proof the action never ran when the provider's data can be delayed. AWS's write-up on safe retries makes the case for putting request identity in the API contract, so that client tracking and provider behaviour agree.
A transactional outbox keeps a committed action from disappearing between the database write and dispatch, but it can't by itself stop a provider from applying a repeated request. A checkpoint likewise lets local work resume yet can't resolve an unknown external result. After an ambiguous timeout the evidence might be a supported lookup, an event, or a documented idempotent replay. A locally generated action ID doesn't turn an API idempotent.
Measure at the boundaries
Tool calls are a good place to observe agent behaviour. The trace should link the run, the artifact revision, the authorisation decision and every external attempt. Capture tool arguments and results selectively, with redaction and access controls, since storing every raw payload creates problems of its own.
The record has to answer practical questions. Was the research evidence usable? Did validation turn the draft down? Is the run waiting for a reviewer, or did the notification break? Did the action get accepted, refused or left hanging?
These distinctions improve evaluation as well. For an account workflow, begin with ordinary examples plus a small set of hard cases:
- Sources disagree and the brief has to preserve the disagreement.
- The message is edited after approval, which means execution needs fresh authorisation.
- The same job arrives twice and must not create two logical actions.
- The provider takes the action and then the connection fails, so recovery has to honour what the provider truly guarantees on retries.
Reserve deterministic checks for things like valid schemas and matching approvals. For relevance and how well evidence supports a claim, use a defined rubric and reviewed examples. Grading by a model can help, provided you check how closely it tracks human judgment. Anthropic's evaluation advice separates an agent's recorded execution from the environment's end state and suggests repeating trials, since agents can act differently each time. Those ideas suit testing a whole workflow.
For a drafting workflow, follow how often drafts are accepted unedited, how long reviewers spend, and the total running cost per accepted artifact, with failed runs included in that cost. Coverage counts too: how many eligible requests yielded something useful. Skip it and a system can look steadier simply by refusing hard jobs.
Release the behaviour as one unit
Editing a prompt can change production behaviour while the container image stays identical, so a release needs a manifest tying the code version to the workflow, prompts, schemas, tool adapters, model settings and permission policy. Every run notes the release it ran under. Investigation then becomes feasible, though exact replay isn't promised, since models drift, providers change and source data may vanish. Keeping the relevant input versions or allowed snapshots helps explain results afterwards.
Work that runs for a long time brings one more snag: an old checkpoint may not match a newer state schema. A deployment must decide explicitly whether to finish those runs on the old version, migrate them or halt them for review. Rolling back only the worker image may not bring back compatible behaviour.
Exercise restarts at the points that matter: right after an artifact is saved, while a review is pending, and on both sides of dispatch. When workers are plural, the database can turn away stale writes via conditional updates or fencing tokens. Stopping duplicates outside the system still rests on how execution is designed and on the provider's contract.
A first deployment can be modest: a service, a durable database and a background worker may do. Separate credentials, capped retries and a kill switch for new external writes come ahead of a large cluster. Kubernetes becomes worthwhile when its scheduling or operations features answer a need you've measured.
Stick with one workflow until you understand it
Agent products keep changing what is cheap to build, and exploring them is worthwhile. Still, it pays to hold enough steady to grasp the work being done. For an account brief or an inbox triage, keep the input, the useful output and the review boundary fixed while the implementation gets better. A new model or runtime can then be judged against identical examples, permissions and recovery needs.
Adopting another tool becomes a concrete question. Can it express a handoff more clearly, cut reviewer effort, or recover interrupted work with fewer manual steps, and does that justify the migration and upkeep? That's the judgment worth carrying forward: decide where reasoning earns its cost, make authority explicit, and keep enough evidence to know what happened after the model stopped speaking.