Replay-Safe Agents: Keeping LLM Calls Out of the Workflow With Temporal

Picture an agent on step 7 of 10 when its worker dies: an OOM kill, a maintenance event, a kernel panic. If the LLM call lives inside the workflow, recovery becomes a silent rewrite of history. The workflow fires the model again, receives a different answer, and the agent ends up somewhere other than where it died. The same tokens get paid for twice, and the recovered run is a different run from the original.

It is not a thought experiment. At 02:14 one night a support agent closed a customer account that should have received a 30-day grace period. The tool call succeeded. When an engineer replayed the same customer message, the agent behaved correctly. The postmortem had a hole where the root cause belonged, filled by the words "could not reproduce." A non-reproducible failure has not gone away. It is a failure that can no longer be studied.

The rule that prevents this is short: the code that orchestrates an agent must be deterministic, the LLM is not, so the LLM must sit behind a boundary that does not have to be. Temporal is the engine used for the examples, but the pattern is framework-agnostic. The same contract applies to AWS Step Functions, to Prefect and Dagster, to a poller reading a database outbox, and to a homemade state table, and later sections map the idea onto those stacks.

The bug before the vocabulary

Take a sourcing agent with five steps: research, score, decide, approve, confirm. The decide step calls an LLM, and the natural, tempting place to make that call is directly in the workflow class:

# ❌ What I almost wrote — LLM call INSIDE the workflow.
@workflow.defn
class SourcingWorkflow:
    @workflow.run
    async def run(self, brief: SourcingBriefInput) -> SourcingResult:
        suppliers = await workflow.execute_activity(research_activity, brief)
        scored = await workflow.execute_activity(score_activity, suppliers)

        # The LLM call — right here, in the workflow. THIS IS THE BUG.
        # If worker dies here, replay will re-call the LLM and mutate state/history.
        response = await llm_client.complete(
            messages=build_decide_prompt(scored),
            temperature=0.0,
        )
        selected = parse_llm_decision(response.content, scored)

        approval = await workflow.wait_condition(...)
        # ...create PO, initiate payment

It reads fine. A senior engineer who had never seen Temporal would call it good. The trouble starts if the worker crashes once the LLM call has returned and while the human approval is still pending.

The fix moves the call into an activity, a function the workflow schedules and whose result gets recorded:

# ✅ The fix — LLM call moved into an activity.
@activity.defn
async def decide_activity(scored: list[dict], brief: SourcingBriefInput) -> dict:
    """The LLM call lives here — in the activity, not the workflow."""
    from app.agentmesh.llm import get_llm_client

    client = get_llm_client()
    response = await client.complete(
        messages=build_decide_prompt(scored),
        temperature=0.0,
    )
    selected, rationale = parse_llm_decision(response.content, scored)
    return {"selected_supplier": selected, "decision_reason": rationale}


@workflow.defn
class SourcingWorkflow:
    @workflow.run
    async def run(self, brief: SourcingBriefInput) -> SourcingResult:
        suppliers = await workflow.execute_activity(research_activity, brief)
        scored = await workflow.execute_activity(score_activity, suppliers)

        # The LLM call is NOWHERE in the workflow.
        # The activity result is recorded. On replay, it's injected.
        decision = await workflow.execute_activity(
            decide_activity,
            args=(scored, brief),
        )

One block of code moved. On replay the activity does not run again, because the workflow receives the recorded result and the model is never called a second time. Why that works is the real subject.

The law: replay must reproduce history

As a workflow runs, Temporal writes every step into an event history, a log that only ever grows, with entries such as workflow started, activity scheduled, activity completed, signal received and timer fired. Events are immutable, and the history is the source of truth.

After a crash, Temporal replays the workflow. A fresh worker runs the workflow function from its first line. At each call such as workflow.execute_activity(...) or workflow.wait_condition(...), Temporal consults the history. If a matching event exists, it injects the recorded result in place of executing anything. If the function makes a call that differs from what the history holds, Temporal raises a non-determinism error.

That is event sourcing. The history is the log, the workflow function is the fold over it, and the same events in the same order through the same function always give the same state. Temporal's documentation says it plainly: workflow code has to be deterministic to support replay, and non-deterministic operations such as API calls, LLM invocations and database queries belong in activities. The activity runs once, its result goes into the history, and from then on replays read it from there.

Anyone who has worked with Postgres has seen the same shape. The write-ahead log is the record, the table files are a materialised view, and recovery replays the log. An event-sourcing handbook states the identical law: same events, same order, same final state, with no clocks, random IDs or time-of-day reads inside the fold.

What the violation looks like

When the LLM sits inside the workflow, two things break at once.

The bill doubles. A call that cost three cents the first time costs three cents again on replay. One team measured uncontrolled retries burning about two dollars in tokens over thirty seconds for a single conversation, and circuit-breaking cut that to a cent, a 200x gap.

The run diverges. The original execution picked supplier B. The replay picked supplier A. The history no longer matches, Temporal raises its non-determinism error, and the workflow is dead.

These are not rare events. Roughly 0.8% of calls to Anthropic and OpenAI fail over a thirty-day window, and an agent making 12 model calls per task therefore faces a per-task failure rate near 9.2%. If the LLM is in the workflow, every retry re-fires it. If it is in an activity, the retry receives the recorded result. Temporal's own writing draws the line well: the workflow is the orchestration layer and must be deterministic so that agents survive process crashes, while activities are where the real work happens, including LLM calls, tool invocations and API requests, and they can be as unpredictable as they need to be.

Why temperature zero is not a shield

Someone always objects that temperature was set to 0. It does not help, for several reasons:

• Zero temperature is greedy decoding, not determinism. In one analysis, a thousand identical requests produced 80 different completions, caused by batch-dependent floating-point differences on GPUs.
• The same model name does not mean the same model. Providers update models silently, so the model called on Tuesday may not match Wednesday's.
• The provider might be down during replay. The workflow then sits stuck waiting.
• Even a byte-identical answer costs money. A second inference was still paid for.

A developer who writes under the name Tisha put the trap well: teams spend weeks forcing a model into bitwise determinism, fail, and conclude the system is unknowable, when the determinism that matters sits one layer up, in orchestration. They were aiming at the wrong layer.

The activity as a membrane

The change is structural. Inside an activity anything is allowed. Outside it, everything must be deterministic.

A workflow may call activities, wait for signals, set timers and branch, provided the branch conditions depend only on activity results and workflow inputs. Forbidden are model calls, HTTP requests, reads of the system clock, random number generation, and anything else able to give a different outcome on another machine at another moment. All of that goes into activities.

Think of it as anatomy. What the workflow runs is not the agent itself but the coordination around it, while the model is merely a service that the agent consults. The brain is unpredictable, the spine has to be steady, and the brain does not go inside the spine.

How it looks in a real codebase

The simplified snippets above hide a detail. In a real codebase the LLM call lives inside nodes of a graph, and the graph runs inside an activity, so there are two boundaries, not one:

Temporal Workflow (deterministic — replayed)
    └── Activity: run_graph_until_interrupt (non-deterministic — recorded once)
            └── LangGraph StateGraph
                    └── decide_node (async function)
                            └── get_llm_client().complete()  ← the LLM call

The workflow itself is pure orchestration, a sequence of execute_activity and wait_condition calls and nothing more. The execute_activity line is the boundary. What happens inside the activity is recorded, and what happens outside it is replayed:

@workflow.defn
class SourcingWorkflow:
    @workflow.run
    async def run(self, brief: SourcingBriefInput) -> SourcingResult:
        # ↓ This is the boundary. The activity runs once. Result is recorded.
        graph_result = await workflow.execute_activity(run_graph_until_interrupt, brief, ...)

        # Wait for human signal — a Temporal primitive, not an LLM call.
        # On replay, the signal is injected from history.
        await workflow.wait_condition(lambda: self._approval_received, timeout=timedelta(hours=24))

        # ↓ Another boundary. Resume activity runs once. Result is recorded.
        resume_result = await workflow.execute_activity(resume_graph, self._approval_data, ...)

        # ↓ Side effects — each is its own activity with its own retry policy.
        po_result = await workflow.execute_activity(create_po_activity, args=(...), ...)

Nothing in it calls an LLM client, performs an HTTP request, reads the current time or draws a random number. Same inputs, same history, same execution path, every time.

The activity builds the graph and invokes it, and that single graph.ainvoke() line is where the non-determinism lives: LLM calls, tool calls and memory queries. The activity returns a dictionary, which gets recorded, and replay skips the entire thing:

@activity.defn
async def run_graph_until_interrupt(brief: SourcingBriefInput) -> dict:
    graph = build_sourcing_graph(checkpointer=await get_checkpointer())
    config = {"configurable": {"thread_id": activity.info().workflow_id}}

    # ↓ Everything inside this call is non-deterministic. It runs ONCE.
    #   The result dict is recorded in the event history. On replay, it's injected.
    final_state = await graph.ainvoke({"brief": brief, "suppliers": [], "attempts": 0, ...}, config)

    return {"paused": True, "selected_supplier": selected, "suppliers": final_state.get("suppliers", [])}

At the bottom of the stack, the graph node is where the model actually fires. It builds the prompt from the scored suppliers and past decisions and then makes the call:

async def decide_node(state: AgentState) -> dict:
    scored = state.get("scored_suppliers", [])
    messages = build_decide_prompt(brief.item, brief.quantity, brief.budget, scored, past_decisions)

    # ↓ THE LLM CALL. This is the non-determinism that must never be in the workflow.
    response = await get_llm_client().complete(messages=messages, temperature=0.0, max_tokens=1000)

    selected, rationale = parse_llm_decision(response.content, scored)
    return {"selected_supplier": selected, "decision_reason": rationale, "cost_incurred": response.cost_usd}    # ↓ THE LLM CALL. This is the non-determinism that must never be in the workflow.
    response = await get_llm_client().complete(messages=messages, temperature=0.0, max_tokens=1000)

That is three layers below the workflow: activity, graph, node, then the LLM. On replay none of them runs, because the activity result is injected. The workflow is deterministic. The activity, the graph and the model are not, yet the non-determinism is contained at the activity boundary. The workflow only ever sees a recorded dictionary, the same one on every replay.

Where the boundary gets blurry

The pattern is clean on paper and blurs in a few places in practice. The multi-day case deserves attention because its failure is subtle and two persistence layers can drift apart without warning.

When the graph pauses at an approval node through LangGraph's interrupt(), the activity completes and hands a dictionary back to the workflow. The workflow then calls workflow.wait_condition() to wait for a human signal. At that moment, two copies of state exist in two systems:

Temporal event history             LangGraph checkpoint (Postgres)
  └── ActivityCompleted(result)      └── graph state at interrupt()
        paused: True                       node: "approve"
        selected: SupplierB                 selected: SupplierB
        suppliers: [A, B, C]                suppliers: [A, B, C]

Both sit in Postgres, yet they are separate tables fed by separate code paths at separate times. LangGraph's checkpoint is written by the activity as the interrupt fires; once the activity returns, Temporal adds its activity-completed event. That makes two writes, two systems and one logical state.

Trouble comes when one write survives and the other does not. One mitigation is operational: the thread_id must be the Temporal workflow_id, which stays stable across replays unlike run_id; the checkpoint table must keep data as long as Temporal's history does; and the resume activity must call graph.ainvoke(None, config), since passing None tells LangGraph to load from the checkpoint and not to rerun from scratch. Get any of them wrong and the layers drift. The workflow does not crash. It resumes from the wrong state.

The better answer: make the graph stateless

The intuitive design enables LangGraph's Postgres checkpointer to track state across agent nodes alongside Temporal's history. Tracing failure modes during multi-day pauses exposes the hidden trap, which is a classic dual-write problem across two databases. If a worker dies right after LangGraph writes its checkpoint but before Temporal records completion, the state engines disagree, and the workflow silently resumes from mismatched state.

Rather than keep adding synchronisation rules around thread mapping, retention and reloads, the cleaner decision is to remove the second state engine. Without its checkpointer, LangGraph becomes an in-memory, stateless reasoning unit inside the activity. Temporal owns all long-term state, execution history and human-in-the-loop pauses through workflow.wait_condition(). LangGraph simply works through its reasoning loop, hands a result dictionary back to the activity, and finishes. The result: no dual writes, no desync risk and a simpler operational surface.

Neither tool becomes redundant. LangGraph remains strong at fast in-memory graph routing, tool calling and node-level reasoning loops, while Temporal remains strong at durability across days, crash recovery and distributed orchestration. A stateless LangGraph inside a durable Temporal activity avoids making them compete over persistence and keeps the strengths of both.

One constraint in many systems

Orchestration frameworks are only one place the replay contract shows up. Any system that recovers by replaying has the same three parts: a record of events, a pure function that folds them into state, and everything unpredictable kept on the far side of a recorded edge.

Agents make this hardest. In a classic event-sourced ledger the apply function stays pure, since crediting ten dollars to an account always yields the same result. An agent's business logic is to ask the LLM what to do, and the answer differs each time. If the brain sits in the replay path, replay is impossible, so the brain has to sit outside it.

Temporal was chosen for the example platform because it enforces the contract: its SDK raises a non-determinism error on violation instead of quietly producing a divergent run. Step Functions enforces it for free, since its declarative state language makes it impossible to put an LLM call in the state machine by accident. With an outbox or custom polling, enforcement is manual. Nothing stops an LLM call inside the poller loop and nothing warns about it. The constraint is identical in all of them. The only question is who catches a violation: the framework, or whoever is on call at 3 AM.

The same split holds across other agent stacks. Some enforce it, namely Temporal and the Microsoft Agent Framework. Some supply the primitives and leave enforcement to the builder, as LangGraph and AutoGen do. Some ignore the matter altogether, like Celery and CrewAI. The practical question is how much of the replay contract has to be built by hand.

Figures from outside comparisons point the same way. A Celery-to-Temporal migration write-up flagged a high risk of duplicate execution on retries, especially for tasks that burn significant GPU time. A Temporal-with-CrewAI project reached the architecture described above, using the orchestration engine as the source of truth for state and the agent framework as a stateless reasoning unit invoked from inside an activity. And in one AutoGen versus CrewAI benchmark, AutoGen finished tasks at a p95 latency of 2.1 seconds against 4.7 seconds and succeeded 94% of the time against 83%, a reminder that how a framework handles failure shows up in numbers. The Microsoft Agent Framework ships file-based checkpoint storage and fault-tolerant supersteps, which is its way of providing the same guarantee.

What determinism costs

The activity boundary is not free. Inline simplicity goes away, since the wrong version has fewer lines and reads more easily. Inline access to state goes away too, because an activity sees only what it is passed, not the workflow's internals. And three edge cases appear that do not exist with an inline LLM. These are real engineering decisions, not philosophical trade-offs.

Edge case one: long activities and heartbeats

An LLM call may take 30 seconds. A multi-turn agent graph with tool calls may take five minutes. Temporal's default start_to_close_timeout is 10 seconds. If an activity exceeds it, Temporal marks it failed and retries, which repeats the model call, repays for the tokens and redoes the non-deterministic work that had been carefully isolated.

The remedy is a heartbeat timeout. Any activity that outlasts a handful of seconds ought to invoke activity.heartbeat() at intervals, which signals to Temporal that it is still alive. If the worker dies, Temporal uses the last heartbeat to judge how far the activity got, so a checkpointable activity can resume and any other restarts from zero:

@activity.defn
async def run_graph_until_interrupt(brief: SourcingBriefInput) -> dict:
    # Long-running: graph may run 5+ minutes with multiple LLM calls
    for node in graph.stream(initial_state, config):
        activity.heartbeat()  # ← "I'm alive, don't timeout me"
        # ... process node output

There is a catch. A heartbeat saves none of the activity's internal progress; all it does is stave off the timeout. Should the worker die halfway through the graph, the retry begins the activity again from the top and the LLM fires once more, because the result is recorded only when the activity completes. With a five-minute graph, a worker lost at the four-minute mark takes four minutes of unprotected work down with it.

That is why the example splits the graph across two activities, one running until the interrupt and one resuming after it. Each is short enough to finish within its timeout, and each result is recorded independently. The split exists for durability granularity, not code organisation.

Edge case two: streaming tokens

When the LLM is inline in a workflow, tokens can stream straight to the user's HTTP response. When it sits in an activity, tokens go to the activity's output, a single return value recorded in the history. Nothing reaches the user until the activity finishes.

Streaming draws the objection raised most often against this pattern, and it has merit. Three approaches exist, in rising complexity. The first accepts that the output is a decision, not a chat stream, which suits the sourcing agent. The second adds a side channel for multi-turn interactions such as an interview where the user must see each reply. The side channel can be a server-sent events endpoint backed by Redis: the activity publishes tokens to Redis, the endpoint streams them to the user, while Temporal keeps only the final result. A user who drops off can come back later and ask the workflow for its outcome. Tokens on the stream are disposable; the stored outcome is what lasts.

Edge case three: retry policies differ

Temporal retries failed activities automatically, yet failure means different things for different activities. A read-only lookup and a side effect should not retry alike. For a side effect such as creating a purchase order, the policy is strict and non-retryable, with an idempotency key providing safety:

# Side effect — strict, non-retryable. Idempotency key handles safety.
po_result = await workflow.execute_activity(
    create_po_activity,
    args=(supplier, item, quantity, price),
    retry_policy=workflow.RetryPolicy(
        initial_interval=timedelta(seconds=1),
        maximum_attempts=1,  # ← don't retry. Idempotency key prevents duplicates.
        non_retryable_error_types=["DuplicatePOError"],
    ),
)

# Verification — aggressive retry, but with backoff for eventual consistency
po_verification = await workflow.execute_activity(
    verify_po_exists,
    args=(po_result["po_id"],),
    retry_policy=workflow.RetryPolicy(
        initial_interval=timedelta(seconds=2),  # ← give the DB time to sync
        maximum_attempts=5,
    ),
)

The retry policy is part of the replay contract. When an activity fails and retries, the retries are recorded in the history, and on replay Temporal injects the final result without re-running failed attempts. But an activity that creates a purchase order with maximum_attempts=3 gets three attempts at creating it, and without an idempotency key that means three orders. The two mechanisms cooperate: the policy decides when to retry, and the key decides what happens when it does.

These are the costs of durability. The trade is not deterministic versus non-deterministic. It is durable with some indirection versus simple and unrecoverable, and a production agent should choose durability because production crashes. The reliability conversation about agents often stays at the wrong layer, concentrating on the model: better prompting, more evaluation coverage, a smarter checkpoint next quarter. All of that hits a hard ceiling.

The single rule to keep

Of everything in the system, only the LLM resists replay, and that is the reason it belongs where replay never touches it. Temporal's FAQ compresses it further: always wrap LLM calls in an activity. Always, not usually and not when time allows, because the one time it is skipped is the 3 AM crash where the replay yields a different run and somebody on call must account for an overnight agent sitting in a failed state.

Lacking the replay contract, the three layers of harness, loop and graph are only organisational. They help in thinking about a system but do not help it survive. With the contract they become structural: the loop resumes from the last completed step, with activities injected from history; the graph's state is held in one place and loaded by the resume activity; tools do not re-run, since results are recorded and the idempotency key is a second line of defence. The layers stay separate under failure, and not merely as a matter of principle.

One limit remains. Temporal guarantees that a workflow replays correctly after a crash. It does not guarantee that business logic is safe to replay, and that part stays with the builder. Idempotency is the next problem to solve.