Typed Decision Models vs LLM Calls: Latency and Stability at the Boundary

Modern agent code is full of small yes-or-no judgments: is this ticket urgent, does this tool call fit the request, is this chunk worth retrieving. The default habit is to answer each one with yet another generative LLM call and parse the structured reply, which works but costs seconds per step and produces numbers of uncertain meaning. This walkthrough runs the same bounded decision through a purpose-built decision model and through a general LLM, 240 times in total, and then wires the resulting probability into a LangChain router. By the end you will know how to measure such a comparison fairly, what the latency and stability differences looked like, and how to keep routing policy in plain code instead of inside a prompt.

Bounded judgments are not generation tasks

Look at the decisions scattered through a typical LLM application and many of them are not really about producing language:

• Should this support ticket be treated as urgent?
• Does the proposed tool call actually match what the user asked for?
• Does a document meet a specific condition?
• Is a retrieved passage relevant enough to include in the context?

The usual implementation hands the current state to an LLM, asks for a structured answer, and converts that answer into a branch in the code. Nothing is wrong with that in principle. The open question is whether every one of these branch points deserves a full autoregressive model.

Jev is TypeSafe AI's first model in what the company calls its "System One" line, and it takes the opposite approach. It never produces free text. You give it some state plus a set of constrained questions, and it answers with typed decisions and probabilities. The vendor frames it as a building block for software, not a substitute for a chat or reasoning model. LangChain has published an integration that uses Jev inside agent harnesses for jobs such as picking a model and checking tool calls. Both products were new at the time of writing, so confirm current APIs before building on them. If you are mainly interested in the cost side of this pattern, the article on pricing a typed decision router in front of frontier LLM calls covers that angle; here the focus is behaviour and latency.

Designing the comparison

Both systems were given exactly one question to answer, paraphrased here: does the request call for real multi-step reasoning, or is it something simpler, such as a direct lookup, pulling out a value, a basic calculation, a rewrite, or a small local edit?

The prompt set intentionally leaned toward ambiguous cases. Obvious extremes tell you little; the interesting behaviour of a router lives near the line where "just answer it" turns into "think this through". Eight developer requests were used:

1. Dictionary lookup: why looking up a key in a Python dict is typically O(1).
2. List vs set: which of the two structures suits a collection of unique user IDs.
3. REST API design: a review of the endpoint POST /users/{id}/delete and whether a cleaner REST shape exists.
4. Redis vs PostgreSQL: whether an existing Redis cache should be swapped for PostgreSQL.
5. Async Python: whether moving a web scraper to async would help.
6. Queue vs pub/sub: which of the two to use for order notification messages.
7. SQL indexing: reviewing a query and judging whether an index could speed it up.
8. Microservices vs monolith: whether a Python service should be broken into microservices or kept whole.

Each prompt ran ten times in each of three setups:

• Jev, asked through a Noul that returns the probability that the request needs substantial reasoning.
• GPT-5.6 Sol with a probability, called via LangChain structured output and required to return a boolean plus a self-reported probability.
• GPT-5.6 Sol, binary only, the same model asked for nothing but True or False, with no mention of confidence at all.

The call budget works out as follows.

8 prompts
× 10 repetitions
× 3 conditions
= 240 calls

Every call logged its decision, its probability where one existed, the model name and the round-trip latency. It is important to be clear about what the experiment does not try to do: there is no ground-truth label for how much reasoning a given request "really" needs, so correctness is off the table. Instead the run targets three narrower questions:

1. Do the decision model and the generative model reach similar routing judgments?
2. How consistent is each one when the identical request is repeated?
3. What does a bounded semantic decision cost in latency when a specialised model makes it versus a generative one?

The findings that matter most came from the borderline prompts, not the clearly complex ones.

Three implementations of one decision

All three setups were written in Python with LangChain.

Jev with a single Noul

A Noul models a binary proposition, but instead of committing to True or False straight away it returns a probability. Jev received the developer request as its state and one Noul carrying the question. In outline, the question definition looks like this:

Noul(
    instructions="""
    Does this request require substantial multi-step reasoning
    rather than a direct lookup, extraction, simple calculation,
    rewriting task, or small localized edit?
    """
)

The response is simply a number, for example:

0.86

That value reads as the probability that the statement holds:

P(requires substantial reasoning) = 0.86

To compare Jev with the binary LLM setup, a probability of 0.5 or more counted as True.

GPT-5.6 Sol reporting its own probability

The generative baseline used ChatOpenAI through LangChain, with a Pydantic model constraining the output shape. The Field(ge=0.0, le=1.0) bounds ensure the number is at least in range; they say nothing about whether it is meaningful.

class ReasoningProbability(BaseModel):
    requires_reasoning: bool
    probability: float = Field(ge=0.0, le=1.0)

Here the model had to supply both a verdict and an estimate of how likely a yes was. That distinction deserves attention. Jev's output is the thing the model was built to produce. The LLM's number is text it generates because the schema demands a float, and there is no guarantee that it reflects any internal likelihood. Treating the two numbers as the same kind of quantity is an assumption, not a given.

GPT-5.6 Sol forced into a boolean

The third setup stripped out every reference to probability and asked the same model for a single field:

class ReasoningBinary(BaseModel):
    requires_reasoning: bool

That leaves three distinct signals: one native probability and two generative variants.

Jev
    → native probabilistic decision

GPT-5.6 Sol
    → generated probability + decisionGPT-5.6 Sol
    → forced binary decision

What the 240 calls showed

Three patterns emerged:

1. Jev reached the bounded decision far faster.
2. On requests that clearly needed reasoning the systems mostly agreed; they diverged near the boundary.
3. On one borderline prompt, the LLM's self-reported probability swung widely between identical calls.

Latency: tens of milliseconds versus seconds

Measured as mean round-trip time, Jev answered in 128.8 ms per decision. GPT-5.6 Sol took 2,828.1 ms when it returned a probability and 4,072.1 ms in the binary-only setup. That is roughly 22 times slower for the probability variant and 32 times slower for the boolean variant.

Treat those multiples as a snapshot rather than a benchmark. Hosted API latency moves with provider load, model version, region and network path. The order of magnitude is the useful part: when the job is one constrained judgment, spinning up a general generative model adds far more machinery than the question needs.

The binary-only result is a curiosity worth noting. Asking for fewer output fields did not make the call faster; if anything it was slower. One small run cannot explain why, but it is a reminder that end-to-end LLM latency is dominated by factors other than the count of visible output tokens, such as queueing, prefill and any hidden reasoning the model does before it answers.

Agreement at the extremes, divergence at the edge

On the plainly heavy prompts, the Redis replacement and the microservices split among them, the two systems agreed and scored the need for reasoning high. The gap opened on the easier-looking requests. For the dictionary lookup, list-versus-set and REST design questions, GPT-5.6 Sol reported probabilities near zero, whereas Jev placed them noticeably closer to the middle of the range.

The REST review was the sharpest example:

The request asks for a critique of POST /users/{id}/delete and a better RESTful alternative.

Jev's average probability was 0.551, just over the 0.5 cut-off. GPT-5.6 Sol averaged 0.071. This was not noise straddling the threshold: in all ten runs Jev classified the request as reasoning-heavy, and in all ten runs of both LLM setups Sol classified it as not.

That does not make Jev right. Without a labelled ground truth you cannot say which judgment is better. What it does establish is that the two signals are not interchangeable, and that they differ precisely in the zone where a router is most sensitive to how its boundary is defined. If you swap one decision source for another, expect routing to change on exactly the requests you are least sure about.

Two numbers between zero and one that behave differently

Repeating each prompt ten times made stability visible. Most prompts produced fairly consistent outputs from both systems. The exception was the request to pick between a queue and a pub/sub system for order notifications.

Across ten identical calls, GPT-5.6 Sol's self-reported probability ranged from roughly 0.28 to 0.90, while Jev's values stayed tightly clustered. The binary LLM setup answered True in nine of the ten runs, so the verdicts were fairly stable even though the stated confidence was not.

The explanation is structural. Both systems emit something you can write as a value in [0, 1], but the processes behind them differ. For Jev, the probability is the model's native output for a bounded decision. For the LLM, it is a token sequence produced because the prompt asked for an estimate, and sampling can land on quite different numbers from run to run. On this task, an LLM's self-reported probability did not reliably behave like the output of a model designed for probabilistic decisions. If you plan to threshold such a number, measure its variance on repeated calls before trusting it.

Turning the probability into a LangChain router

The comparison motivated a practical test: use Jev's probability to route real requests between two OpenAI models.

• GPT-5.6 Luna handles requests judged not to need substantial reasoning.
• GPT-5.6 Sol handles those judged to need it.

Jev never picks a model directly. It answers the same Noul question from the experiment, whether the request needs substantial multi-step reasoning, and returns a probability such as 0.54, read as P(requires substantial reasoning) = 0.54.

Application code then checks that value against a threshold defined in ordinary Python:

REASONING_THRESHOLD = 0.65

The branch itself is equally plain:

if reasoning_probability >= REASONING_THRESHOLD:
    model = powerful_model
else:
    model = fast_model

The division of labour is the point. Jev supplies the probability; the application decides what to do with it. With a threshold of 0.65, a score of 0.54 routes to Luna and a score of 0.86 routes to Sol. The policy is explicit, deterministic and reviewable in a code diff, while the semantic judgment stays with the model.

The complete flow looks like this:

User request
     ↓
    Jev
     ↓
P(requires substantial reasoning)
     ↓
Python threshold = 0.65
   ↙                  ↘
Luna                  Sol

A note on that threshold: 0.65 is illustrative, not tuned. It was picked only so that, in a tiny demo, the obviously simple requests landed on one side and the design and architecture questions, which scored higher, on the other. In production you would choose it empirically against a larger labelled set, weighing two different errors: paying for the expensive model on an easy request, and sending a genuinely hard request to the weaker one. Those costs are rarely symmetric, so the best threshold is rarely 0.5.

How the router behaved

Six requests went through the router, from a simple calculation up to a multi-region architecture problem. Most results matched intuition. The calculation and the rewriting task scored very low and went to Luna; the substantial architecture questions scored high and went to Sol.

The informative cases were in between:

• the dictionary lookup explanation scored 0.46
• the REST API review scored 0.54
• the queue-versus-pub/sub question scored 0.86

Because Jev returns a continuous score rather than a fixed easy or hard label, the aggressiveness of routing becomes a separate, adjustable decision. At 0.65, the REST review went to Luna and the queue-versus-pub/sub question went to Sol. Raising the threshold would send fewer requests to the larger model and save cost; lowering it would reduce the chance of under-serving a hard request. The model's judgment does not change when you move the dial, which means you can tune policy, even dynamically, without re-prompting anything.

Limits of this experiment

Keep the claims in proportion:

• Eight prompts cannot characterise either model in general, and the labelling question itself is subjective.
• Accuracy and calibration were not measured. The run compares behaviour, consistency and latency, not correctness.
• Jev's probabilities and the LLM's self-reported values should not be assumed to share a statistical meaning.
• Latency figures depend on the provider and conditions at the time of the run.

A stronger follow-up would use a larger labelled dataset and evaluate the router end to end: routing accuracy, calibration, downstream answer quality, cost, and sensitivity to different thresholds.

When a separate decision model is worth it

For a small application, letting an LLM do the routing is often perfectly fine; one extra call rarely matters. The picture changes as an agent accumulates routing, retrieval, validation and tool-checking steps. Many of those are bounded judgments, and if each becomes another multi-second generative call, the agent loop grows slow and expensive for reasons unrelated to the actual work.

A clean separation of responsibilities looks like this:

• A decision model handles bounded semantic judgments and returns a probability.
• Application code owns thresholds, policy and branching.
• The LLM does the open-ended thinking and writing.

A capable LLM could handle every one of these, but that does not oblige every part of the system to use the same kind of model, especially when latency matters. The broader case for sizing models to the task is made in right-sizing LLMs with routing, retrieval and evaluation.

Key takeaways

• In this run the decision model averaged about 129 ms per bounded judgment, against several seconds for the generative setups.
• Near ambiguous inputs the two systems disagreed consistently, not randomly, so replacing one signal with the other will change routing where it is most sensitive.
• An LLM's self-reported probability can vary widely across identical calls (roughly 0.28 to 0.90 on one prompt here); check its variance before thresholding it.
• Keep the threshold in code, choose it from labelled data and the asymmetric cost of each mistake, and treat a specialised decision model as a complement to the LLM, not a faster imitation of one.