AI Architecture Boundaries: Probabilistic Models in Deterministic Systems

Consider a layout planner built from three model-ish parts. An LLM reads the user's request, RAG fetches the applicable standards, and a graph-attention network suggests where each piece goes. Not one of them has the last word. A deterministic engine of rules and placement checks decides what ends up in the layout, and it vetoes the model's suggestions constantly.

That split wasn't a design principle chosen up front. It came from a layout the model returned with total confidence: four components wired to each other and overlapping, geometry that can't exist on a real playground. Nothing raised an error. The response was well-formed, the reasoning read fine, and the result was physically impossible.

Start from what's familiar

Take an ordinary e-commerce backend:

User → API Gateway → Backend Services → Database

Production systems have extras like caches, queues, auth, observability and outside APIs, yet the core picture holds, and its defining trait is control. The application chooses which operations run, what data gets touched, which business rules count, how failure is handled and what success means. When an order is paid, nobody asks a service to interpret whether that's true:

if order.status == "PAID":
    ship_order()

The program decides.

Now add a model

Say an AI support assistant joins the system. On day one, the architecture is:

User → Backend → LLM API → Response

It ships. Soon a customer asks about the status of an order, and the model needs data that isn't in the prompt:

User → Backend → Fetch order data → Build context → LLM → Response

Then another customer asks the assistant to cancel an order. Now the model has to do something and not merely say something:

User → LLM → Decide → Call cancellation API → DB → LLM → Response

Then comes a question that needs three separate operations, and the picture stops being a line at all.

The model is no longer just a dependency. It has become part of the control flow, and that's where the architecture truly shifts.

Probabilistic computation is the real difference

Traditional software has this shape:

Input → Program → Output

AI software looks like this:

Input → Context → Model → Probabilistic Output

A function that can be tested directly shows the contrast:

def calculate_discount(price):
    return price * 0.9

Calling calculate_discount(100) gives 90, and nobody needs a judge to say whether 90 is roughly right.

Take a request such as summarizing a contract. There's no one right string to compare against. Instead the checks are about behavior. Were all important clauses covered? Was anything invented? Did a termination condition vanish? Was a legal term misread? Does the result help anyone?

So the architecture needs something most systems lack: evaluation of behavior, and not only execution of code.

A 200 response can still be a failure

This one catches many teams.

Classic monitoring begins with a simple rule: 200 is healthy, 500 is broken. AI systems can return a 200 carrying a wrong answer. Or a 200 built on stale context. Or a 200 that leaked something sensitive. Or a 200 where the agent looped nineteen times and ran up a bill.

Every individual call succeeded, and the application still failed.

Status codes alone can't serve as the health signal. Teams need latency, error rates, token usage, cost, retrieval quality, output quality, tool calls and user feedback, tied together by one trace.

Context is a component of its own

A traditional app asks what data the program needs. An AI app asks a second question: what should the model see?

A question about the refund policy might need the question itself, the company policy, the customer's account and the last four turns of the conversation.

Adding a vector database is the easy part. Deciding what belongs in the window is harder, because the application must weigh how relevant and fresh each item is, who's permitted to see it, how much space it takes, what the history says, what happens when sources conflict, and how tenants stay separated. Context stops being plumbing and turns into architecture.

Business rules don't belong to the model

Suppose a user asks to delete their account. One option is writing a rule in the prompt that says to delete it when asked. That puts the boundary in the wrong place. A better flow:

User → Model → Intent: DELETE_ACCOUNT → Validation
     → Authorization → Business rules → Delete

The model works out what the user wants. Deterministic code decides whether that's allowed.

Let the model handle ambiguity. Let software handle authority.

A new trust boundary

The classic version has one line:

Untrusted user → Validation → Trusted application

Once a model sits in the flow, its output has to be treated as untrusted as well:

User → Application → Model → Model output → Validation → Real systems

Several things can steer that output: user input, retrieved documents, external data, tool responses, injected instructions, or context nobody planned for.

Validating and authorizing model output before executing it gives a completely different system from one that executes it directly.

Tools turn one request into a distributed workflow

A chat turn is a single model call. An agentic turn could be a chain: model, search, model, API call, model, another search, model, answer.

The user sees a single request while the backend performs ten operations.

That creates an observability problem that plain request logs can't solve. End-to-end traces are required, and not only for debugging. They answer why a request cost $3, why the agent hit one API five times, which retrieved chunk produced the wrong answer, and where the latency actually went.

Cost turns into a runtime constraint

Backend teams already budget CPU, memory, connections and bandwidth. AI brings new line items: tokens, model calls, context size, tool calls and inference spend.

A single agent run might break down like this:

Model    $0.02
Search
Model    $0.04
API call
Model    $0.03
Search
Model    $0.05
─────────────
Total    $0.14

That run costs fourteen cents. Multiply it by thousands of users a day and the number stops being bookkeeping and becomes a limit on the design, which explains why production systems pick up model routing, prompt caching, cost attribution and per-request budgets.

Distributed systems have always handled processor and memory limits this way. Here, dollars, tokens and seconds are the resources every request consumes.

Retries get riskier

Retries are standard practice. Now imagine the request is to send an invoice to the customer.

The model calls send_invoice(), which succeeds. The network drops the response, so the call is retried, and it succeeds again. The customer ends up with two invoices.

The underlying concepts are old ones: idempotency, timeouts, transaction boundaries, partial failure and compensation. AI didn't create them. It pulled them back into the middle of applications that had managed to ignore them.

Testing changes as well

A conventional pipeline goes from code to unit tests to integration tests to deployment. AI work needs another layer alongside it, where unit tests, evaluations and security checks exercise the AI workflow before release.

Behavior can shift whenever any of these change: code, prompt, model, context, retrieval or a tool definition. Zero lines of application code can ship, a provider can update a model, and by Monday the product is different.

What the architecture looks like

Picture a flow that runs from ingress through application logic, context assembly and the model layer, then through validation and policy, and finally out to tools. Underneath it sits a cross-cutting band covering evaluation, observability, cost, security, feedback and governance.

What matters most about that picture is simple: the LLM isn't the architecture. It's one component inside it.

Is AI architecture really different?

Partly. Everything older is still needed: APIs, databases, auth, caching, queues, load balancing, retries, transactions, monitoring and security. None of it disappeared.

What changed is the type of one component:

Traditional:  deterministic → deterministic → deterministic
AI:           deterministic → PROBABILISTIC → deterministic

That single swap changes everything around it.

The principle

The whole argument condenses into one line:

Put probabilistic intelligence inside deterministic boundaries.

The model can conclude that the user probably wants to cancel an order. The application decides whether this user is allowed to cancel it.

The model can claim that a refund will come to $42. The amount itself should come from deterministic code, such as refund = order.amount - fees.

The model can decide that the refund API should be called. Deterministic software decides whether the request has permission to call it.

That's the boundary.

The wider view

For decades, engineers learned how to make systems predictable, testable, observable, secure, recoverable and scalable. AI doesn't cancel that effort. It adds a part that lacks those properties by nature, and the rest of the system has to supply them.

Questions a demo never asks

The easiest mistake is believing that all it takes is calling an LLM. That's enough for a demo and not enough for a product.

Once AI sits in a real system, several questions that the deterministic version answered for free now need answers:

• What should the model know?
• Can its output be trusted?
• What happens when it's wrong?
• Who may execute its decisions?
• How is behavior tested when it changes underneath you?
• How is a failure detected when the call returned 200?
• How much may one request spend?
• What happens when the provider goes down?
• How does a half-finished workflow get recovered?

More logic in the prompt isn't the answer. Better boundaries around the model are.

Let AI deal with ambiguity and let software deal with certainty. The boundary between them deserves careful construction.

The model understood what the user wanted, but not whether the result could exist. Only the first is its job.