LangGraph vs Pydantic AI: Why the Model, Not the Framework, Decided Accuracy

Choosing between agent frameworks is one of the most heated debates in AI engineering, yet a carefully controlled benchmark suggests it may be one of the least consequential decisions for correctness. When LangGraph and Pydantic AI were run on identical tasks, tools and model settings, they produced exactly the same score, while a change of model flipped a quarter of the outcomes. This article walks through that experiment, what its numbers do and do not show, and how to spend your own evaluation effort where it actually changes results.

How the benchmark isolated the framework

The comparison pitted LangGraph 1.2.9 against Pydantic AI 2.13.0 across four tasks, with 20 runs per task per framework: 160 runs in total, one model, temperature 0. The entire study cost $0.3767 in API spend. The methodology is documented and the test harness is open source, so the setup can be checked and rerun.

All four tasks are deterministic and scored by exact match, and each one probes a different agent skill:

• inventory-reorder: a single tool call followed by arithmetic on its result
• dependent-shipping-quote: a second tool call whose input depends on the output of the first
• recover-stale-revision: the agent must realize its data is outdated and fetch it again
• refund-policy-minimal-tools: date arithmetic against a refund policy window, with intentionally few tools available

Everything around the library was held constant: each framework saw identical prompts for every task, was handed tools described by identical JSON schemas and backed by one shared implementation, and was graded by a single scorer. The model was pinned to gpt-4o at temperature 0, with parallel tool calls disabled. The orchestration library was the only thing allowed to change.

That discipline is the whole point of the experiment. Many published framework comparisons change the prompt, the tools and the library all at once, then credit the library for any difference. If you want a trustworthy comparison of your own, holding everything else fixed is the first requirement.

An exact tie on correctness

Both frameworks completed all 80 of their runs correctly. The summary:

LangGraph 1.2.9    Pydantic AI 2.13.0
Completed           80 / 80            80 / 80
Wilson 95% CI       0.954 - 1.0        0.954 - 1.0
Total cost          $0.1881            $0.1886
Median wall time    3.863 s            5.526 s

This is not a case of "roughly comparable" or "within the noise". With the same tasks, schemas, model and payloads, the scores were identical. The Wilson interval of 0.954 to 1.0 is what a perfect 80 out of 80 gives you: it says that with this sample, the true success rate is very likely above about 95 percent for both. If one of the libraries made agents more likely to reach the right answer on these tasks, the experiment had enough runs to expose a sizable effect, and it found none. It could still miss very small differences, which is a normal limit of any sample this size.

The strongest sign that the comparison was fair lies in the input tokens. They matched exactly on both frameworks for every run: 311 for inventory-reorder, 791 for dependent-shipping-quote, 615 for recover-stale-revision and 926 for refund-policy-minimal-tools. In other words, both libraries turned the same schema into the same API request, byte for byte in terms of tokens. Output tokens differed by a few on a handful of runs, which is ordinary model variation even at temperature 0, and that accounts for the half-cent gap in total cost. Framework overhead does not.

A tie makes for an unexciting headline, but it answers the practical question: for tool-calling correctness, at this scale and with this model, the framework was not the deciding variable.

Latency: a consistent gap with a narrow explanation

The frameworks did differ on one axis, and consistently so. The list below gives, for each task, how much lower LangGraph's median wall time was than Pydantic AI's, followed by the bootstrap 95% confidence interval of that saving:

• inventory-reorder: LangGraph ahead by 1.669 s (interval from 1.481 to 1.918 s)
• dependent-shipping-quote: ahead by 1.430 s (interval from 1.241 to 1.686 s)
• recover-stale-revision: ahead by 1.842 s (interval from 1.658 to 2.101 s)
• refund-policy-minimal-tools: ahead by 1.645 s (interval from 1.427 to 1.911 s)

On all four tasks, the whole interval stays far from zero. LangGraph finished each run roughly 1.4 to 1.8 seconds sooner, about 1.4 times faster on the medians.

Do not turn that into a recommendation without reading the explanation. The gap comes from bridging async code into sync code: the harness is synchronous, and Pydantic AI was driven through that sync path. It does not show that Pydantic AI's agent loop is inherently slow. The number is real and reproducible for that specific integration style, but it is also the least transferable result in the study. In an application that is already asynchronous end to end, expect the difference to shrink or disappear.

One outlier cuts against the median

A detail worth noticing works against the headline latency result. The single slowest run in the study belonged to LangGraph: 16.196 seconds on dependent-shipping-quote, compared with a worst Pydantic AI run of 10.228 seconds. LangGraph's next-slowest run on that task took 5.232 seconds, so this looks like one isolated outlier rather than a heavy tail. With only 20 runs per task, though, those two possibilities cannot be told apart, and one data point is not a distribution.

The practical lesson: medians favor LangGraph here, but if you are setting a latency SLO, tail behavior is what matters, and you need to measure it on your own workload rather than rely on someone else's median.

Swapping the model changed a quarter of the outcomes

In a separate run on the same harness, the four tasks were executed against two models, with 40 runs each:

gpt-4o-mini    gpt-4o
Completed         30 / 40        40 / 40
Cost (40 runs)    $0.0057177     $0.094275

The frameworks, tasks and tools were unchanged. Only the model differed, and 25 percent of the outcomes flipped.

The way gpt-4o-mini failed is the most instructive part. Its tool calling was fine: with either framework it got every run right on all tasks but one. The exception was refund-policy-minimal-tools, which it failed in all 10 of its runs on that task, five per framework, and always identically. It calculated days_since_delivery = 19 by counting both the start and end dates, when the correct exclusive count is 18, and then concluded the customer was outside the refund window.

That is neither a framework failure nor a tool-use failure. It is a model that gets inclusive versus exclusive date counting wrong, inside an agent that faithfully acted on the wrong number. No orchestration library can catch an error like that on its own. What catches it is a deterministic check: computing date differences in a tool instead of asking the model to do the arithmetic, or validating the result before acting on it.

So the comparison the industry argues about ended in a tie, while the comparison hardly anyone debates produced a 25-point swing. Getting the higher score cost about 16.5 times more.

What to take from this for your own stack

Choose the framework for ergonomics

Base the decision on the things you will live with every day: type safety, whether a graph model fits your problem, the debugging experience, and how readable the code is to your team during an incident. Those differences are real. On this evidence, correctness is not among them. For a broader comparison of the options along those lines, see choosing a Python AI agent framework.

Put your evaluation budget into the model

Here, the model choice moved 25 percent of outcomes and changed cost by a factor of 16.5. If you have limited time to test one thing, test the model on your own tasks. Note also that both models in this study are specific, older OpenAI models; newer models may behave differently, so rerun the comparison with the models you actually plan to use.

Then study failure modes, not the overall success rate

The most informative part of this benchmark was not the score table but refund-policy-minimal-tools, the one task built to be difficult. Every model and framework passed the other three tasks, so those tasks revealed nothing. An evaluation suite only earns its keep where something fails. Design tasks that target the specific weaknesses you fear, such as off-by-one date logic, stale data or chained tool calls, and grow the suite from real production failures.

Distrust benchmarks that crown a winner

Treat any framework benchmark that declares a winner with suspicion, including this one. What makes this study checkable is that the raw JSONL results, a manifest with pinned versions and hashes, and the harness are all published, together with the full per-run data for all 160 runs. Hold any benchmark you rely on to the same standard.

Limits of the evidence

The scope is narrow: one model family, four tasks, one harness and a fixed date. The runs took place on 24 and 25 July 2026, with the models gpt-4o and gpt-4o-mini and the library versions LangGraph 1.2.9 and pydantic-ai-slim[openai] 2.13.0, and results may shift with later versions. Tool-calling correctness is also only one dimension of an agent framework, and possibly not the one you care about most; state management, persistence, streaming and observability are not measured here.

What the data supports is therefore more modest than a headline: on these tasks and at this scale, the framework did not predict whether the agent reached the right answer, and the model did.

Key takeaways

• Controlling everything except the library is what makes a framework comparison meaningful; identical input token counts are a good fairness check.
• LangGraph and Pydantic AI tied at 80 out of 80 on tool-calling correctness with gpt-4o.
• The latency gap reflected a sync-to-async bridge in the harness, not a slow agent loop, and a single outlier shows why tail latency must be measured separately.
• Changing the model flipped 25 percent of outcomes, driven by one consistent date arithmetic error that no framework could catch.
• Invest evaluation effort in model selection and in hard, failure-seeking tasks, and move deterministic logic such as date math out of the model.