Home / Articles

Multi-Agent Systems Are Backend Systems with Non-Deterministic Parts

Production multi-agent systems reward backend habits: contracts at every boundary, prompts as config, a tight context budget, one role per agent, retries.

Publisher: REACTAPP.TOP9 min read2167 words

This article is published in English.

Sources: Multi-Agent Systems Are Backend Systems with Non-Deterministic Components

Come to agentic AI from data science and machine learning, and the early work feels familiar. Stay long enough on a production multi-agent system and it stops feeling like data science at all. It begins to resemble backend engineering, with the twist that some of the components are non-deterministic. Seen that way, a lot of design decisions get easier to reason about.

The aim isn't to strip out the flexibility LLMs bring, since that flexibility is the reason to use them. The aim is to draw predictable boundaries around it.

Give every boundary a contract

LLMs speak text, so it's tempting to let the parts of an agent system talk to each other in text as well. One component writes something, the next reads it and works out what was meant.

That holds up until the interpretation becomes a source of doubt in its own right.

Clarification requests are a good example. A component can notice it lacks information and explain this fluently in prose, yet leave the rest of the workflow guessing. Is it a failure, or a request for clarification? Should the run halt? What exactly is missing?

The fix is to push boundaries like this toward explicit, structured contracts. Rather than a sentence like "I don't have enough information to continue, could you ask the user which dataset to use?", the components exchange a message such as:

{
  "status": "needs_clarification",
  "question": "Which dataset should be used?",
  "required_input": "dataset"
}

The JSON format is incidental. The contract is what counts. The next component receives a known state with a known shape and never has to infer intent from generated prose.

Human interaction can follow the same pattern. When a domain-specific component needs extra input, it emits a structured clarification request, and a separate interaction layer decides how to show that to the user. In outline:

Domain workflow
  ↓
Structured clarification request
  ↓
User interaction layer
  ↓
User

Completion deserves the same treatment. If finishing a task matters for orchestration, it should be a defined protocol event, and not something a second model must deduce from phrases like "the analysis is complete."

The idea reaches well past clarification and completion. Messages between components, tool calls and their responses, errors and orchestration events all benefit from explicit boundaries.

A workable rule of thumb: if another component has to interpret an LLM's output, that output probably deserves a schema.

Natural language belongs where natural language is actually needed. Inside the system, ambiguity tends to cause more trouble than it saves.

Treat prompts as configuration

As a system grows, hand-editing each system prompt as a stand-alone block of text gets harder to maintain. Shared rules end up copied into several places, capability descriptions change over time, and the prompt drifts away from what is really available at runtime.

A more declarative setup helps. Shared rules, role-specific instructions, capability descriptions and examples live in separate pieces and are assembled into the final prompt during the build.

The build step also gives you a place to validate the configuration before the model ever sees it. If a prompt claims a capability that isn't actually wired up, the build should catch that. The same goes for missing required sections, duplicated definitions, or settings that no longer match the role.

Nobody edits the final prompt directly. It's produced from smaller configuration pieces, validated, and emitted as a build artifact you can version and rebuild at will.

At that point prompt management looks far more like ordinary software configuration than like writing. You reuse shared parts, validate whatever can be validated before runtime, and keep the generated output consistent with the system around it.

Context is a budget

Big context windows make it easy to keep stuffing things in: instructions, examples, history, capability descriptions, artifacts, earlier outputs. Eventually the model has everything it could conceivably need, and sometimes it starts to perform worse.

The obvious explanation is length, as if the model simply forgot something. With current models, though, instruction dilution is often the more useful lens.

An agent's effective context is far bigger than its system prompt:

instructions
+ available capabilities
+ examples
+ conversation history
+ artifacts
+ previous outputs
+ workflow state

A model may be perfectly able to locate one particular instruction somewhere in all that. It doesn't follow that it will obey every instruction equally well once several signals are competing for its attention.

So the more useful question becomes how little context a component really needs, as opposed to how much the model can technically accept.

This goes beyond token savings. Smaller, more relevant context narrows the decision space and makes behaviour easier to predict. Each extra piece of context competes for the model's attention. It doesn't sit there for free.

One agent, one role

Tool count turns out to be a poor signal for when to split an agent. One agent can hold many closely related capabilities and still have a clear purpose, while another with only a few tools may already be doing several unrelated jobs.

Responsibility makes a much better dividing line. Different responsibilities usually call for different instructions, context, validation rules and failure policies. It's separation of concerns, applied to probabilistic components.

Split by responsibility and ignore tool count.

The graph holds state

Prompt optimisation gets plenty of attention in agent work. State gets less.

It's easy to fixate on prompt size and miss the graph state. A carefully trimmed prompt can still sit on top of a state that keeps hauling along outputs and artifacts from several steps back, unless that state is managed on purpose.

In graph-based orchestration, messages, intermediate results, artifacts and prior outputs pile up as execution moves through the graph. The graph remembers, which is handy, but it means state needs deliberate design.

A large artifact produced early can ride along through later steps that never need its contents. Big tool responses and intermediate results do the same.

For heavier artifacts it's usually better to keep the data outside the graph. Instead of this:

Large artifact
  ↓
Graph state
  ↓
Step
  ↓
Step
  ↓
Step

something closer to the following is normally enough:

Large artifact
      ↓
External storage
      ↓
Reference + metadata
      ↓
Graph state

Most components don't need everything produced earlier in the workflow. They need just enough to make the next decision.

Prompt size is only one side of the context problem, and state deserves equal attention. Design it actively, and don't let it become a dumping ground for everything execution generates.

Reliability is mostly engineering

Even with good contracts, narrow roles and carefully managed context, agents still do surprising things. An action fails, the model picks the wrong action or repeats one, a workflow ends too soon or runs far longer than expected.

There are also cases where each step looks sensible but the workflow as a whole is going nowhere. The model keeps making locally valid choices and effectively builds a loop.

The first instinct, especially for people who started with large LLMs in early 2023, is to patch such problems in the prompt. Sometimes that works. But most of the reliability mechanisms that end up being necessary look like plain backend engineering:

  • retries and timeouts
  • hard caps on how many iterations and tool calls a run may make
  • spotting loops by watching for the same action or state coming back
  • a gentle push when execution stalls yet could still recover
  • explicit failure states with recovery paths

When something must not happen more than a set number of times, enforcing a limit beats adding another line asking the model to behave. Prompts are part of reliability engineering, but they shouldn't be the entire reliability layer.

Observability follows the same logic. When an ordinary backend request fails, logs and a stack trace usually say what happened. With an agentic workflow, knowing where it failed often isn't enough.

You need to reconstruct how it got there: the route picked, the actions tried, the arguments produced and the results returned. In loop debugging, the interesting part is frequently not the final error but the chain of decisions that kept returning execution to the same path.

In agentic systems you debug decisions as well as code, so good observability has to make the decision path reconstructable.

Model choice is a system design decision

Viewing agents as software components also changes how models get picked. Nothing says each component must run on one shared model, and sending every one of them to the largest model on offer makes even less sense.

A multi-agent system naturally mixes tasks of very different difficulty. Some call for real reasoning, planning or synthesis. Others mostly route, extract, classify, transform structured data, or choose from a small set of actions.

A frontier model everywhere can work, but it can also leave the system needlessly slow and costly. What matters for each role is the balance of quality, reliability, latency and price.

The choice also goes deeper than parameter counts or leaderboard ranks. Models that look alike on benchmarks can act very differently inside the same workflow.

Architecture is one factor, since dense, mixture-of-experts and hybrid models differ in character. Post-training matters too: different supervised fine-tuning, preference optimisation and reinforcement-learning recipes shape instruction following, reasoning, tool use, structured output and behaviour off the beaten path.

The trade-offs in practice are concrete. One model may be fast, cheap and dependable on narrow jobs such as routing or classification, yet now and then fill gaps with plausible-sounding reasoning on open-ended steps. Another may be slower and pricier but clearly steadier on multi-step reasoning.

You don't need to train these models for this to matter. Knowing how they behave differently helps you decide where each one belongs in a larger system.

The benchmark that matters most is your real workload. For routing, test routing. Where output must match a schema, count how often it does. Components that lean on reasoning get a set of typical reasoning cases. Put latency and cost in the same evaluation.

Judge a model by the role it plays in your system and not by its overall ranking.

Engineering around non-determinism

One direction runs through all of this. Early agent prototypes maximise freedom: hand the model instructions, tools and context, then let it decide.

Moving toward production, much of the work goes the opposite way. Responsibilities narrow, interfaces become structured, context gets more relevant, state is handled on purpose, failure paths are explicit and model choice depends on the role.

Even prompts stop being isolated text and start acting like configuration that has to be assembled, validated, versioned and reproduced.

None of this removes the LLM's ability to decide. If every decision could be written deterministically, there would be little point building an agentic system. What matters is choosing where non-determinism is useful and where it isn't.

That's why these systems increasingly feel like backend engineering with a new kind of component. The LLM supplies flexibility inside the boundaries, and software engineering sets the boundaries.

The best agentic system isn't the one where agents can do the most. It's the one where agents have exactly as much freedom as they need.