Learning Agentic AI Engineering in the Order Failures Teach It

Developers moving into agent engineering often try to absorb every framework at once, cycling through LangChain, CrewAI, AutoGen and LangGraph in a single week while shipping nothing. The problem is sequencing: they study tools before understanding the failures those tools exist to prevent. This guide lays out a fourteen-step path in the order the skills actually build on each other, from basic Python through shipping an agent that runs unattended. Along the way you will learn the three failure modes that shape every design decision, the small stack that covers most production work, and the structural patterns (state files, maker-checker review, layered evaluation and permission scoping) that separate a demo from a system you can trust overnight.

Part 1: The mental model

1. Agent engineering is not prompt engineering with a new label

Prompt engineering is about crafting what you say to a model. Agent engineering is about building the system that decides what the model should work on, when it should stop, and how to react when its answer is wrong.

The working definition is concrete: you build software in which an LLM chooses the next action, invokes a tool to perform it, inspects the outcome, and repeats until the task is complete, with no human steering each step. A chatbot responds to a message. An agent picks actions, runs them, checks their results and iterates. That difference is the whole job.

It brings three responsibilities that prompt work never required:

• Treating errors as a core concern. Agents fail all the time: APIs time out, JSON comes back malformed, the model invents tool calls, and tool outputs violate their schemas. Code that assumes success will crash at the worst possible moment, typically during a demo.
• Managing state. An individual LLM call keeps nothing. An agent working through ten steps of tool calls, retries and subagents needs structured, durable state that survives beyond any one context window.
• Building evaluation into the infrastructure. There is no intuition that tells you an agent is right. You need automated gates, such as tests, rubrics or a judge model, that reject bad output without a person reviewing every run.

Job descriptions for these roles often read like a catalog. Orchestration frameworks (LangGraph, LangChain, LlamaIndex), protocols (MCP, A2A), model features (function calling, structured outputs, prompt caching), retrieval topics (RAG, RAGAS, hybrid search, reranking, embedding models, vector and graph databases) and operational skills (sandboxed execution, observability, evaluation) all appear, usually followed by a request for comfort with rapid iteration. The list looks overwhelming, but most entries are a handful of underlying ideas under different names. Learn the ideas and the product names become easy to place.

2. Three failure modes explain most of the work

Before writing code, understand why agentic systems break. Nearly every tool and pattern in this field exists to counter one of three behaviors.

Agentic laziness. Faced with a long, multi-part task, the model stops early and reports success after partial progress. It resolves 20 of 50 backlog tickets and describes the remainder as handled. The countermeasure is an explicit stopping condition that something other than the working model checks.

Self-preferential bias. Asked to review its own output, a model reliably approves it. A reviewer invested in the result cannot judge it fairly. The countermeasure is structural: the agent that produces the work must not be the one that reviews it.

Goal drift. Over many steps, and especially after context is summarized or compressed, the agent gradually loses track of the original objective. A constraint like "leave the payments module alone" can silently vanish by step 47. The countermeasure is a persistent specification file, re-read on every run, holding the constraints the model would otherwise lose.

When the ecosystem feels overwhelming, ask of any new tool or pattern which of these three problems it addresses. That question sorts most of the noise.

3. A small core stack, and four things to postpone

The requirements lists are long, but the bulk of production agent work rests on four layers, best learned in this order:

Core stack (learn these, in this order):
1. Python + async    : the bedrock; everything else builds on it
2. LLM APIs          : Anthropic, OpenAI; understand tokens, context, costs
3. Tool use / MCP    : function calling; how models act on the world
4. LangGraph         : stateful orchestration for multi-step, multi-agent work

Postpone the following until you have shipped at least one real agent:

• Fine-tuning. Early projects almost never need it. A capable foundation model with well-designed prompts generally beats a fine-tuned model with poor ones.
• Agonizing over vector databases. Something like Chroma is fine locally and a managed service like Pinecone is fine in production. Hold off on picking one until retrieval is a real bottleneck in something you are building.
• Switching frameworks. Choose one orchestration framework (LangGraph is a sound default), finish a project, and only then explore. Changing tools every week because each promises to be simpler guarantees nothing gets finished.
• Voice and browser agents. These are specializations built on the same foundations. Master text agents first; the patterns carry over.

Part 2: The building blocks

4. Python and asynchronous code

You do not need Python mastery, but you do need enough to debug failures, and agents fail often. Focus on:

• Classes and data models. Agents hand structured data from step to step, so you have to model it. A Pydantic schema acts as the contract between tool calls and agent logic; treat it as required, not optional.
• Async with asyncio. Agents spend much of their time waiting on tools: a database query, an HTTP call, a subprocess. Synchronous code blocks during every wait, while async code can do other work. Slow agent code is very often sync code waiting in series.
• HTTP and REST. Without APIs an agent can think but not act. Learn to read API documentation, respect rate limits, interpret error responses and retry sensibly. A tool that crashes on an HTTP 429 is a tool the agent cannot rely on.
• Error handling. Wrap every tool invocation in try/except. Agents run without supervision, and an unhandled exception that prints a stack trace and exits helps nobody in the middle of the night.

A practical threshold: if you could hand a task to a junior engineer with a checklist and trust a test suite to catch their mistakes, you know enough Python to start. More can come later.

5. LLM fundamentals: tokens, context and cost

Models such as Claude, GPT and Gemini are powerful but need direction, and you cannot direct what you do not understand.

Tokenization. Models read tokens, not words, and a single term can split into several tokens depending on the tokenizer. As a rough rule for English, a 100,000-token context holds about 75,000 words. Anything outside that window, whether last week's conversation or a file you forgot to include, simply does not exist for the model. If something matters, it must be in context.

Context limits and retrieval. Models do not remember; each session starts empty. Packing everything into the prompt is expensive and degrades quality as it grows. Retrieval-augmented generation exists to fetch only what is relevant.

Inference, not training. You will almost never train a model. You call inference on someone else's model and pay per token. Cost is input tokens times the input price plus output tokens times the output price, so a loop that makes 50 calls with a 20,000-token context has a real bill attached.

Prompting for agents. Agent prompts differ from chat prompts. Three patterns matter most: chain-of-thought, where the model reasons explicitly before acting; ReAct, a cycle of reasoning, acting and observing; and reflection, where the model critiques its own draft before returning it. Most other prompting techniques are variations on these. For a deeper look at the ReAct cycle, see how ReAct agents combine reasoning with real-world actions.

6. Tool use and MCP turn a chatbot into an agent

A model limited to generating text is a chatbot. A model that can invoke a function, inspect the result and choose what to do next is an agent, and tool use is the mechanism that enables it.

Mechanically, you describe each function with a name, a natural-language description and a JSON schema for its parameters, and send those definitions along with the user's message. The model decides whether a tool is needed and, if so, returns a structured tool call with arguments instead of plain text. Your code runs the function, sends the result back, and the model continues from there. The description matters as much as the schema, because it is what the model uses to decide when a tool applies.

Most practical agent tools fall into four categories. The signatures below illustrate them: tools that read (observe the world), tools that write (change state), tools that execute code, and tools that verify work:

# Category 1: Read  (agent observes the world)
def search_codebase(query: str, path: str) -> list[str]: ...
def fetch_url(url: str) -> str: ...
def read_file(path: str) -> str: ...

# Category 2: Write (agent changes state)
def create_file(path: str, content: str) -> None: ...
def open_pull_request(title: str, body: str, branch: str) -> str: ...
def send_slack_message(channel: str, text: str) -> None: ...

# Category 3: Execute (agent runs code)
def run_tests(test_path: str) -> dict: ...
def execute_sql(query: str, db: str) -> list[dict]: ...

# Category 4: Verify (agent checks its own work)
def lint_code(file_path: str) -> list[str]: ...
def run_type_checker(path: str) -> bool: ...

The categories are also a useful lens for risk. Read and verify tools are usually safe to call freely, while write and execute tools change things and deserve tighter permissions, a theme that returns in the security step.

The Model Context Protocol (MCP) is an emerging standard that replaces custom integration code with a protocol. A common analogy is USB-C for AI: rather than writing a bespoke adapter each time the agent needs GitHub, Slack or a database, you connect an existing MCP server, and the host application discovers the server's capabilities and uses them without extra glue code.

The integrations that pay off fastest are GitHub for branches, pull requests and issues; Slack for notifications and summaries; your database for queries and controlled writes; and the team's issue tracker. With those four connected, an agent can operate across most of an engineering workflow.

7. Retrieval, because context has a ceiling

Retrieval-augmented generation gives an agent knowledge that does not fit in its context. It is a response to a hard constraint rather than a fashion: the context window is finite, and your codebase and documentation are not.

A retrieval system has four main parts:

• Chunking, where beginners most often go wrong. Oversized chunks carry too much irrelevant material; undersized chunks lose meaning. The right size depends on the content, and source code usually needs different boundaries (such as whole functions) than prose documentation.
• Embeddings, which make similarity search possible. An embedding model turns text into a numeric vector so that similar passages produce nearby vectors.
• Search, which finds the stored vectors closest to the embedded query and returns their chunks.
• Evaluation, which distinguishes a system that works from one that only appears to. The key measures are retrieval precision (was what you fetched actually relevant?), faithfulness (did the answer stay within the retrieved material?) and answer relevance (did it address the question?). Tools such as RAGAS or a judge model can score these. Without measurement, improvement is guesswork.

Mature retrieval is rarely a straight line from query to answer. Production systems commonly rewrite the question before searching, rerank results after searching, and use a critic step to judge whether the retrieved material actually answers the question. In effect, the model reasons about what to fetch and whether the fetch succeeded.

8. Stateful orchestration with LangGraph

One LLM call is not an agent. An agent runs multiple steps, carries state between them, branches on what it observes and recovers from failures. LangGraph provides a structure for exactly that.

Structurally, a LangGraph app is a directed graph whose nodes are plain functions, such as agents, tools or processing steps. Edges define how control moves between them. A shared, typed state object is read and updated by every node.

The sketch below defines a state with a task, a plan, results, errors and a completion flag, then registers four nodes: a planner that splits the work, an executor that runs one step, a verifier that checks the result, and an error handler that retries or escalates. The conditional edge after the verifier is the heart of the loop: finish when the state says done, route to the handler when errors exist, and otherwise execute the next step. Note that this is a fragment; a runnable graph also needs an entry point, the remaining edges and a call to compile().

from langgraph.graph import StateGraph, END
from typing import TypedDict

class AgentState(TypedDict):
    task: str
    plan: list[str]
    results: list[str]
    errors: list[str]
    done: bool

graph = StateGraph(AgentState)

graph.add_node("planner", plan_task)       # breaks work into steps
graph.add_node("executor", execute_step)   # runs one step
graph.add_node("verifier", verify_output)  # checks the result
graph.add_node("handler", handle_error)    # retries or escalates

graph.add_conditional_edges(
    "verifier",
    lambda state: END if state["done"] else
                  "handler" if state["errors"] else
                  "executor"
)

Compared with a hand-written Python loop, the framework adds three capabilities:

• Checkpointing. When you compile the graph with a checkpointer, state is saved after each step, so an interrupted run (a crashed laptop, a restarted session) can resume where it stopped instead of starting over.
• Human-in-the-loop pauses. Setting interrupt_before for a high-stakes node makes the graph stop, surface the proposed action, and wait for approval before continuing. This is a large part of what separates a demo agent from a production one.
• Parallel branches. Independent steps can run concurrently while the framework handles merging their results into state, so you describe the structure instead of writing synchronization code.

A reasonable rule: reach for a graph framework when the agent has more than about three steps, branches on tool output, or loops until a condition holds. A single linear chain without branching is fine in plain Python. The trade-offs are covered in more depth in choosing between chains and stateful graphs.

Part 3: Building it properly

9. Five projects, in sequence

Reading about agents and building them are different skills, and the path only works with hands-on projects. These five, done in order, touch every concept production work requires.

1. An agent with one tool. Pick a single API, such as GitHub or a weather service, and write an agent that judges whether the API is needed, invokes it, and works its response into the reply. Use the raw Anthropic or OpenAI API with no framework, so you see the tool-use loop without any abstraction hiding it.
2. A ReAct agent with three tools. Add web search, a calculator and a code executor, and write the reason-act-observe loop yourself. This is typically where you first see an agent notice and correct its own mistake.
3. Retrieval over a codebase you know. Index a real repository, build retrieval over it, and ask questions that require understanding several files. Measure retrieval quality and fix the chunks that fail. This project shows why chunking matters more than anything else.
4. A multi-step graph agent. Tackle CI triage: read failing test logs, classify the failure, search the codebase for the likely cause, draft a fix and run the tests, with state flowing through five nodes. Make the verifier a separate node from the fixer; you will run straight into self-preferential bias otherwise.
5. A scheduled autonomous loop. Run project four on a cron schedule with nobody watching. Give it a state file so it resumes rather than restarts, a hard spending budget so it cannot run up API costs, and an audit log recording what it did. This is where "works in a demo" becomes "works unattended."

10. The state file: agents forget, files do not

This sounds too simple to matter, yet it is the backbone of every dependable autonomous agent: a markdown file, a JSON document or a database row that lives outside the conversation and records what has been done and what comes next.

Models carry nothing between sessions. Whatever an agent learned during one run disappears unless it is written down, so a loop without persistent state begins from zero each time, while a loop with state picks up where it left off. The example below tracks the last run time, counts of processed and remaining items, work in progress, completed work, items escalated to a human, and dated lessons such as environment quirks to avoid next time:

// STATE.md: what every working autonomous agent needs
{
  "last_run": "2026-07-01 03:00 UTC",
  "items_processed": 47,
  "items_remaining": 12,
  "in_progress": [
    "fix/auth-token-refresh: tests passing, awaiting CI"
  ],
  "completed": [
    "fix/null-check-in-billing: merged, CI green"
  ],
  "escalated_to_human": [
    "src/payments/refund.ts: root cause unclear after 3 theories"
  ],
  "lessons": [
    "2026-06-30: E2E tests require Stripe webhook secret in env. Skip if missing.",
    "2026-06-29: Windows runner has TLS 1.2 issue. Use bash, not PowerShell."
  ]
}

The lessons list deserves attention. It is how a loop stops repeating the same mistake, and it doubles as the persistent specification that counters goal drift. There are two common formats. A markdown file committed to the repository is version-controlled, easy to diff and simple, which suits individuals and small teams. For production loops that several people need to observe, an external system such as an issue tracker like Linear or a database works better. The principle is simple: the agent forgets, the repository remembers, so anything important belongs outside the context window.

11. Maker-checker: separate the author from the reviewer

One agent produces the work and a different agent, with its own context, checks it. This is the structural answer to self-preferential bias, and applying it consistently is one of the clearest markers of mature agent design.

A model grading its own output is far too generous with itself. Ask the agent that wrote a fix whether it is correct and it will find reasons to say yes. Give a separate reviewer the fix and a rubric, with no knowledge of who wrote it or why, and it finds real defects.

The contrast in code: the wrong version asks a single agent to fix a bug and confirm its own fix. The right version runs a fixer on one model, then passes only the resulting code and a specific rubric to a reviewer, instructed to ignore authorship and intent and to return a pass with reasoning or a fail with line references. The reviewer uses a stronger model because judgment is the harder task:

# Wrong: one agent does both
result = await agent("Fix the auth bug and verify your fix is correct")

# Right: maker and checker are separate agents, separate contexts
fix = await agent(
    "Fix the auth bug in src/auth/middleware.ts",
    model="sonnet"
)

review = await agent(
    f"""Review this fix against the rubric below.
    Do not consider who wrote it or their intent.

    Fix:
    {fix.code}

    Rubric:
    - Does it handle the null case on line 47?
    - Does it preserve the existing token expiry logic?
    - Does the test cover the regression case?

    Return: PASS with reasoning, or FAIL with specific line references.""",
    model="opus"   # harder model for the harder judgment task
)

The rule for pairing is that the reviewer receives just two inputs, the rubric and the artifact, and never the author's identity, the reasoning behind the change or the conversation that produced it. Any of those reintroduces self-preference through framing. The rubric also matters; specific, checkable questions like the ones above work far better than asking whether the change is good.

The same split applies well beyond code: authors and reviewers, writers and fact-checkers, generators and judges. Once you notice it, you will see how often default tooling quietly merges the two roles.

12. Evaluation: the gate that makes a loop trustworthy

An agent with no verifier is just a chatbot invoked repeatedly. Evaluation is what decides whether a result can be trusted enough to act on, merge or release. Use three levels, rising in cost and in the kinds of judgment they can make:

• Deterministic checks. Tests, linters, type checkers and builds give a binary result with no judgment involved. They are the first and cheapest gate; whenever a deterministic check can reject bad output, use it.
• An LLM judge. A second model scores the first model's output against a rubric. This works when the rubric is specific and fails when it is vague. The judge should be at least as capable as the producing model and should never be told who produced the output. Running a judge well is its own discipline, covered in managing LLM-as-a-judge as a production system.
• Human approval. Irreversible actions, such as production deployments, payment code, force pushes and architectural changes, need a person to approve before the agent proceeds. In LangGraph, interrupt_before provides this pause. Reserve it for actions that are costly to undo rather than gating everything.

To know whether evaluation is working, track the accepted-change rate. If an agent built to fix failing tests resolves 70% of its issues with changes that pass CI and human review, its rate is 70%. Below roughly 50%, humans are spending their time finishing work the agent started, and the loop costs more than it saves.

13. Security: an unattended agent is an unattended attack surface

Any autonomous agent that touches real infrastructure is a security exposure operating without supervision. The risk is concrete: with indirect prompt injection, an agent that reads a malicious email or web page can be manipulated into running an attacker's command. The main threats:

• Injection through tool outputs. Web pages, GitHub issues and support tickets can hide instructions inside ordinary content, for example a line telling the agent to ignore previous instructions and delete test files. Quarantine is the defense: any agent exposed to untrusted content gets read-only access. Keep reading agents and acting agents separate.
• Permission creep. An agent validated with read-only access gains one write permission "just for convenience," and nobody reviews it again. Re-audit permissions on a regular schedule (monthly is a sensible cadence) and grant only the minimum the task requires.
• Secrets in logs. Verbose logging in a long-running loop scatters credentials into outputs nobody monitors. Turn off verbose logging in production loops and sanitize what remains.
• Unreviewed generated code. Agents can open pull requests faster than people can read them. If CI lacks static analysis, dependency audits and secret scanning, vulnerable code can reach the main branch with nobody noticing. Automation makes these gates more important, not less.

A permission model makes these rules explicit. The example below auto-approves observation-only actions such as reading files, running tests and viewing git status or diffs, and requires a human for pushing, editing environment files, touching payment code and anything with a force flag:

# Safe agent permission model
permissions = {
    "auto_approve": [
        "Read(*)",           # read anything
        "Bash(npm test)",    # run tests
        "Bash(git status)",  # observe state
        "Bash(git diff*)",   # observe diffs
    ],
    "require_human": [
        "Bash(git push*)",   # never push without approval
        "Edit(.env*)",       # never touch secrets
        "Edit(src/payments/*)",  # never touch payments code
        "Bash(*--force*)",   # never force anything
    ]
}

The test for each rule is a single question: if this action turns out to be wrong, how expensive is it to undo? Cheap to reverse means auto-approve; expensive means a human decides. Deciding case by case in the middle is how permission creep starts.

14. Turning the skills into a career

A learning path should lead somewhere. Here is a grounded view of where these skills apply.

What to show. Skip tutorial clones. Three real projects that solved actual problems carry far more weight:

• A scheduled agent whose output you rely on in practice, like the loop built in step 9.
• A multi-agent system in which at least two agents hold different roles that structurally prevent one model instance from doing both, as in the maker-checker pattern from step 11.
• A retrieval system with documented evaluation metrics, showing the before and after of a retrieval fix rather than just the fact that it works.

How long it takes. As a rough estimate, someone who already writes solid Python and studies 10 to 15 hours a week can expect this path to take around eight months. Treat that as a planning figure, not a promise.

Where to start. Roles vary widely in how accessible they are from scratch:

• AI automation engineer at a company whose main business is not AI. The need is someone who can build, for example, a loop that fixes tests overnight. LangGraph, MCP and a working knowledge of CI are enough; dozens of frameworks are not required.
• AI engineer at a startup whose product is an agent. This demands the full stack of retrieval, evaluation, multi-agent design and deployment, with a higher bar and a higher ceiling.
• Agentic infrastructure engineer at a large technology company, which adds distributed systems expertise on top of everything else and is a senior position rather than an entry point.

The point most learning plans miss is that you do not have to know everything before you start. What counts is one deployed agent that handles a genuine problem, proof that you can quantify its success, and the ability to articulate which failure modes its design guards against. Few candidates bring all three, and no certificate substitutes for them.

Wrapping up

For a while, most of the leverage in applied AI sat in the prompt: better wording, better context, better single-shot output. As models have become capable enough to act, the leverage has moved up a level, to the system that decides what agents work on, when their output is verified, how their actions are recorded and what happens when they fail.

• Learn concepts before frameworks, and judge every tool by which failure mode it prevents: laziness, self-preference or drift.
• Keep state outside the model, in a file or database the agent re-reads each run.
• Never let the author of a change be its reviewer; give the reviewer only the artifact and a specific rubric.
• Layer evaluation from deterministic checks to judges to human approval, and measure the accepted-change rate.
• Scope permissions by reversibility, and keep agents that read untrusted content away from write access.

None of this requires a research background or fine-tuning expertise. It requires solid Python, a clear picture of the ways LLMs go wrong, and the habit of putting the verification gate in place before the loop itself. Build the first agent, let it run overnight, and review its diff in the morning.