Six Python AI Agent Frameworks Compared by Building the Same Agent

Ask a crowd which agent framework to use and you'll get a dozen confident, contradictory answers. One camp calls LangGraph overengineered. Another calls it the only serious option. A third says skip frameworks entirely and talk to the model API directly. A client project makes that noise expensive, so the practical way out is to stop arguing and run an experiment.

The experiment here was simple: build the identical agent in six Python frameworks over one weekend, with the same task, the same model, the same tools and the same scoring. The project behind it was a document analysis pipeline in 2026 terms, which means pulling data out of PDFs, cross-referencing it with a database, writing a summary and emailing the result. The results don't flatter either of the loud camps.

What got built and how it was scored

The test agent accepts a company name, looks up recent news on the web, pulls out key financial figures, checks them against a local SQLite database and returns a structured JSON summary. That's three tools, one agent and one output schema.

It's boring on purpose. Almost nobody ships an autonomous coding agent as their first production system. Most real agent work is an API orchestrator with a little reasoning on top, and every framework should handle that comfortably. GPT-4o served as the model across the board wherever a framework allowed it.

Five things were measured:

• lines of code needed
• tokens per run, averaged over 10 runs
• average latency
• hours from nothing to a working prototype
• a "2 AM debug score," meaning how easily you can work out what went wrong when the pipeline breaks late at night

That last metric matters more than it sounds. Things always break.

LangGraph: for people who like control

About 210 lines. Three hours to a working agent. 2,847 tokens per run.

LangGraph feels like it was designed by distributed systems engineers. You declare state, declare nodes, draw edges between them and compile the result. It reads less like building an AI agent and more like laying out a data pipeline, which is a compliment.

The first hour is rough. Seeing StateGraph, add_node and add_conditional_edges for a three-tool agent looks like far too much ceremony, and quitting for the next candidate is tempting.

Then a failure changes the picture. On the third test query the agent broke badly, and opening LangSmith showed the whole run step by step: which node chose wrongly, what state it held, and why it called the database tool ahead of search. Few frameworks make diagnosis that direct.

The feature people rarely mention: durable execution. A crashed run resumes from its last checkpoint. On a weekend experiment that's pleasant. On a pipeline chewing through 10,000 documents overnight, it decides whether you sleep or get paged at 3 AM.

LangGraph also used the fewest tokens in the test. Because the graph spells out control flow, the model doesn't burn tokens deciding what comes next. You've already told it.

The cost is up-front thinking. There's no pasting instructions in and hoping. A well-defined problem rewards the structure, but if you're still figuring out the workflow, you'll draw more graphs than you ship features.

2 AM debug score: 9/10. LangSmith earns its reputation, and issues were diagnosed faster here than anywhere else.

CrewAI: the quick prototype machine

About 340 lines. 45 minutes to a working agent. 4,216 tokens per run.

Going from pip install crewai to correct output took three quarters of an hour, the fastest start of the pack apart from Smolagents.

CrewAI thinks in teams, not single agents. Each member gets a role, a goal and a backstory, tasks go to the group, and the members work out who does what.

researcher = Agent(
    role="Financial Research Analyst",
    goal="Find and verify recent financial data",
    backstory="You're a senior analyst at a hedge fund...",
)

The backstory field looks gimmicky, and it was greeted with an eye roll. Yet outputs from the CrewAI agent came out noticeably more detailed than the same prompt sent through raw API calls, so the role framing does shift how the model approaches the job.

Speed has a price. CrewAI burned roughly 48% more tokens than LangGraph on the same work. The architecture assumes at least two or three agents collaborating, so even a researcher plus a writer, a job one agent could do, paid for delegation round trips. It's also synchronous underneath, and fitting it into an async FastAPI backend meant wrapping calls in run_in_executor. That's survivable, just irritating.

Costs scale quickly. One published comparison put CrewAI test spend at $1,088 against $390 for an equivalent PydanticAI build. Run the agent thousands of times daily and the difference snowballs.

On the debugging scale it scores 5/10 at 2 AM. Successes are lovely, but failures disappear into a delegation chain that behaves like a black box. One bug ate 40 minutes before turning out to be two agents handing the same task back and forth.

PydanticAI: the quiet overachiever

About 130 lines. 1.5 hours to a working agent. 2,912 tokens per run.

If LangGraph suits control seekers and CrewAI suits the impatient, PydanticAI suits the developer who wants the code to make sense in six months. The Pydantic team built it, and anyone who has used FastAPI already knows half the API. Agents are typed, outputs are validated and dependencies are injected, so whatever can be caught before runtime is.

agent = Agent(
    "openai:gpt-4o",
    result_type=CompanyAnalysis,  # Pydantic model
    system_prompt="You are a financial research assistant.",
)

The result_type argument does more than it appears to. Every model reply is checked against your schema automatically. When the model returns malformed JSON, which it eventually will, the framework catches it and retries, and no hand-written validation code is needed.

The full-stack-ai-agent-template project ran a controlled comparison that fits this pattern. The same chat application took about 160 effective lines in PydanticAI, around 280 in LangGraph and roughly 420 in CrewAI.

Shorter code isn't automatically better, yet here it tracked how fast each version could be understood. The PydanticAI version could be read and grasped line by line. The LangGraph one demanded tracing the graph mentally, and the CrewAI one required understanding its delegation model.

The limitation is that PydanticAI is a single-agent framework. It has no built-in multi-agent orchestration, handoffs or delegation, so a three-agent design means writing the coordination yourself. The pydantic-graph module exists but doesn't yet match LangGraph or CrewAI for complicated workflows.

Late-night debugging scores 8/10. Type errors show up early, the IDE flags problems before anything runs and stack traces point at the right line.

OpenAI Agents SDK: the sleeper hit

About 150 lines. One hour to a working agent. 2,791 tokens per run.

A framework from the model vendor sounds like lock-in with a bow on it, and it was nearly left out of the test. That expectation turned out wrong.

The SDK is OpenAI's production rebuild of its earlier Swarm experiment, and it's reduced to four primitives: Agents, Handoffs, Tools and Guardrails. Handoffs are the standout. You say which agents may pass work to which others, and the framework handles routing. It's less flexible than LangGraph's graph model, but for roughly 80% of multi-agent use cases it's enough and far simpler.

agent = Agent(
    name="Researcher",
    instructions="You are a financial research assistant.",
    tools=[search_tool, db_tool],
    handoffs=[summary_agent],
)

It's also no longer tied to one vendor. Support for more than 100 LLMs arrived through an any-llm adapter, and the same test ran against Claude with no code changes.

Token efficiency matched LangGraph, tying for the lowest in the test. Built-in tracing is solid, though a notch below LangSmith.

The drawback is a smaller ecosystem. LangGraph has years of community tools, tutorials and proven patterns. Here the product is newer, so at an edge case you're often alone.

2 AM debug score: 7/10. Tracing is built in and useful, and errors are clear. Documentation gaps, though, meant reading source code more often than anyone would like.

Smolagents: the minimalist's pick

About 95 lines. 30 minutes to a working agent. 3,340 tokens per run.

Those numbers aren't a typo. Hugging Face's entry takes minimalism seriously, with core logic around 1,000 lines of Python that you can read in an afternoon. Try that with LangGraph.

The difference isn't only size. Every other framework here uses JSON tool calling, where the agent emits a blob saying which function to run with which arguments. Smolagents' CodeAgent writes real Python instead.

agent = CodeAgent(
    tools=[search_tool, db_tool],
    model=InferenceClientModel(),
)
result = agent.run("Analyze recent financial news for Acme Corp")

Writing code lets an agent compose tools naturally, with loops, conditionals and chained operations that JSON calls can't express. Benchmark research backs this up, since code agents outperform JSON-based ones on most tests.

The trade-off is security, because the agent is generating and running Python. Sandboxing is available through E2B, Docker and Pyodide, but you set it up. The default LocalPythonExecutor is explicitly not a security boundary, and its docs say so.

Token usage ran higher than LangGraph and the OpenAI SDK. Generated Python is wordy next to compact JSON, and every import and variable declaration costs tokens.

With Gemini or GPT it works fine but feels like a square peg. It shines with Hugging Face models and infrastructure, and the Hub integration, which shares and loads agents the way you'd share a model, is excellent if you already live in that ecosystem.

2 AM debug score: 8/10. Reading the code an agent wrote beats decoding a chain of JSON tool calls, and a failure produces an ordinary Python traceback.

Google ADK: the enterprise dark horse

About 180 lines. Two hours to a working agent. 3,102 tokens per run.

Google's Agent Development Kit appeared quietly in 2025, and many Python developers have never tried it. That's a gap worth closing.

ADK is opinionated about structure and flexible about execution, the way Google's internal tools tend to be. Agents are defined with a model, a name, instructions and tools, much like the OpenAI SDK. On top sit workflow agents (Sequential, Parallel and Loop) plus the newer AgentTeam API for coordinating several agents.

from google.adk.agents import Agent

root_agent = Agent(
    model="gemini-2.5-flash",
    name="financial_analyst",
    instruction="You are a financial research assistant.",
    tools=[search_tool, db_tool],
)

The evaluation framework is what sets it apart. Test cases are written as JSON, adk eval runs them, and you get structured reports on whether behavior matches expectations. No other framework in the lineup ships evaluation tooling this mature.

Deployment is smooth too, provided you're on Google Cloud. Vertex AI Agent Engine handles scaling, monitoring and sessions, but that proviso carries a lot of weight.

ADK is tuned for Gemini. Other models work through LiteLLM, yet the docs, examples and tooling all assume Gemini. Running GPT-4o hit undocumented edge cases that took 45 minutes to untangle.

2 AM debug score: 6/10. The adk web dev UI helps. But error messages are sometimes cryptic, and the documentation is comprehensive in the typical Google way, which means hard to navigate.

Which one to pick

"It depends" frustrates everyone, so here's the decision logic that came out of building the agent six times:

• Production by Friday: pick CrewAI or Smolagents. CrewAI suits a team that isn't comfortable debugging Python code execution. Smolagents suits one that is and wants the smallest dependency footprint.
• Must run reliably for months: pick LangGraph or PydanticAI. LangGraph fits branching logic, human-in-the-loop steps and the need for durable execution. PydanticAI fits mostly single-agent work with structured outputs, where future readers should understand the code.
• Already committed to an ecosystem: pick the OpenAI Agents SDK on OpenAI's platform for its tracing and evaluation, or Google ADK on GCP for native Vertex AI integration with built-in evals.

The uncomfortable part is that for most projects, the framework matters less than prompt engineering and tool design. Output quality came out nearly identical across all six once the system prompt and tool descriptions had real attention. What differed was developer experience, debuggability and operational overhead.

The production setup that survived: PydanticAI for single-agent tasks, which is most work, and LangGraph when the workflow is truly complex. There's no need to apologize for running two frameworks.

All six side by side

| Metric            | LangGraph   | CrewAI          | PydanticAI    | OpenAI SDK       | Smolagents      | Google ADK      |
| ----------------- | ----------- | --------------- | ------------- | ---------------- | --------------- | --------------- |
| Lines of code     | ~210        | ~340            | ~130          | ~150             | ~95             | ~180            |
| Time to prototype | 3 hrs       | 45 min          | 1.5 hrs       | 1 hr             | 30 min          | 2 hrs           |
| Avg tokens/run    | 2,847       | 4,216           | 2,912         | 2,791            | 3,340           | 3,102           |
| Multi-agent       | Yes (graph) | Yes (teams)     | Manual        | Yes (handoffs)   | Yes (hierarchy) | Yes (AgentTeam) |
| Type safety       | TypedDict   | Pydantic config | Full generics | Generic context  | Minimal         | Standard        |
| MCP support       | Yes         | Limited         | Native + A2A  | Native           | Yes             | Yes             |
| Model-agnostic    | Yes         | Yes             | Yes (20+)     | Yes (100+)       | Yes (LiteLLM)   | Gemini-first    |
| Best debugger     | LangSmith   | Logs            | IDE/types     | Built-in tracing | Code output     | ADK Web UI      |
| GitHub stars      | ~48K        | ~44K            | ~15K          | ~16K             | ~26K            | ~23K            |
| 2 AM debug        | 9/10        | 5/10            | 8/10          | 7/10             | 8/10            | 6/10            |

None of these figures tells the whole story, but they make a more honest starting point than most comparison posts.

Expect the ground to move

These frameworks change faster than anyone can document them. LangGraph shipped a major update during the writing. CrewAI's enterprise tier gained integrations within a week. PydanticAI added A2A protocol support, and Google ADK released a SkillToolset feature days earlier.

Any comparison has a shelf life. In six months the performance gaps may close, missing features may land, and the framework you dismissed may become the obvious choice. So pick the one your team can debug at 2 AM, with the pipeline down and a client waiting, and not the one with the most GitHub stars.