Building an Agent Improvement Loop with LangGraph and LangSmith Evals

Getting an agent into production takes the least effort. Making it better afterward is what separates a hobby build from a system that holds up. Plenty of teams launch a first version and then spend months answering complaints one at a time. Each patch is a reaction to a single case, each fix stays local, and the agent never improves as a whole. It piles up workarounds.

The usual cause is treating an agent like conventional software. In ordinary programs the code explains what the system does, so a bug means reading the source until it turns up. With agents that assumption fails. Code only describes what the system is permitted to do, while the traces record what it really did. That gap is the basis of every serious evaluation pipeline, including those Anthropic describes in its engineering writing and LangChain describes in the LangSmith documentation.

What follows is a complete improvement loop for a LangGraph agent. It starts with a small ReAct agent, records its traces in LangSmith, adds three layers of scoring, finds recurring failure patterns, turns those failures into a lasting offline test set, and runs a regression evaluation before any fix ships. Each stage comes with code, an explanation and the output to expect.

The loop works like this:

1. The agent handles a real user query and produces a trace covering every LLM call, tool call and intermediate output.
2. Code-based checks score each trace for format and tool correctness, an LLM judge scores helpfulness and relevance, and humans review the subtle failures.
3. Traces with low scores are grouped into failure patterns such as the wrong tool, reasoning drift or a missed intent.
4. Every recurring failure becomes an offline test case with a known expected behavior.
5. Before a new agent version ships, it runs against the growing suite.
6. Only changes that raise the baseline reach production, and each fixed failure stays in the suite for good.

This is how an agent gets better over time instead of just accumulating patches. The six stages are:

• Stage 1, foundation: a LangGraph ReAct agent with LangSmith tracing.
• Stage 2, trace collection: pulling production runs out as structured records.
• Stage 3, scoring: code checks, an LLM judge and human review.
• Stage 4, pattern discovery: turning scored traces into failure categories.
• Stage 5, offline test suites: converting failures into a regression-safe dataset.
• Stage 6, closing the loop: evaluating before shipping and raising the baseline.

Stage 1: the foundation agent

The first stage builds a small customer support agent that answers questions about an account. It has two tools, one that looks up account details and one that checks recent orders, and it uses LangGraph's prebuilt ReAct agent, the standard pattern for tool-calling agents there. LangSmith tracing is switched on so every run, LLM call and tool call is recorded automatically.

Three pieces make this work:

1. The LangGraph StateGraph is the graph underneath, routing between a node that decides what to do and a node that runs a tool. The create_react_agent helper wraps that pattern.
2. LangSmith tracing turns on when the environment variable LANGSMITH_TRACING=true is set. After that, each LangChain or LangGraph call emits a trace with no code changes.
3. Tools are Python functions decorated with @tool from LangChain, which exposes each one to the agent with a name and a description that steer tool selection.

Environment setup

Installing the libraries comes first:

# Install the libraries we need
pip install langgraph langchain langchain-openai langsmith

Four packages get installed. langgraph supplies the graph runtime and the prebuilt ReAct agent. langchain provides the @tool decorator and message types. langchain-openai wraps the OpenAI chat model in LangChain's interface. langsmith is the client library used to reach the LangSmith backend for tracing and evals.

Environment variables come next, and they switch tracing on without touching the code:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=your_langsmith_key
export LANGSMITH_PROJECT=agent-improvement-loop
export OPENAI_API_KEY=your_openai_key

LANGSMITH_TRACING=true is the switch. LANGSMITH_PROJECT names the project that groups the traces, which is how they get filtered later. The other two are credentials for LangSmith and the model provider.

With those four variables set, any LangChain or LangGraph call in the process sends a trace to LangSmith. Nothing needs wrapping or decorating, because tracing runs in the background.

Building the agent

The imports first:

# Step 1: Import the pieces we need
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent

The imports bring in three pieces. ChatOpenAI wraps the OpenAI chat completion API for LangChain. The tool decorator converts a plain Python function into a callable the agent can use. create_react_agent is a ready-made LangGraph helper that assembles a ReAct-style graph for calling tools.

Then the two tools:

# Step 2: Define two tools the agent can call
@tool
def get_account_info(account_id: str) -> str:
    """Look up basic information for an account by account_id."""
    fake_accounts = {
        "A100": "Account A100: plan=pro, status=active, email=ada@example.com",
        "A200": "Account A200: plan=free, status=suspended, email=turing@example.com",
    }
    return fake_accounts.get(account_id, f"No account found with id {account_id}")

@tool
def get_recent_orders(account_id: str) -> str:
    """Return the three most recent orders for a given account_id."""
    fake_orders = {
        "A100": "Orders for A100: #9001 shipped, #9002 processing, #9003 returned",
        "A200": "Orders for A200: no orders in the last 90 days",
    }
    return fake_orders.get(account_id, f"No orders for {account_id}")

Each tool accepts an account_id and returns a string. The docstring on each function matters, since the LLM reads that text when deciding which tool to call. The decorator pulls the function name, parameters and docstring to build the schema the model sees. In a real system these bodies would query a database or API, but the demo returns hardcoded strings so it stays self-contained and reproducible.

Finally the agent gets assembled:

# Step 3: Wire the tools into a ReAct agent
model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
tools = [get_account_info, get_recent_orders]
agent = create_react_agent(model, tools)

The chat model is created with temperature zero for reproducibility. The tools go into a list and are passed with the model to create_react_agent, which returns a compiled LangGraph StateGraph that already has a reasoning step, a tool-running step and the conditional links joining them.

Testing the agent

A simple query exercises it:

# Step 4: Invoke the agent with a simple query
result = agent.invoke({
    "messages": [{"role": "user", "content": "What plan is account A100 on?"}]
})

for message in result["messages"]:
    print(message.type, ":", message.content)

The call passes a dict holding a messages key. That is because create_react_agent relies on the prebuilt MessagesState schema, whose state is simply a list of chat messages and gets extended by the agent as it thinks and calls tools. The loop afterwards prints every message type with its content, so the whole trace is visible.

The expected output:

human : What plan is account A100 on?
ai :
tool : Account A100: plan=pro, status=active, email=ada@example.com
ai : Account A100 is on the pro plan.

The agent got the user message, then emitted an AI message that triggered a tool call (its content is empty because it carries a tool call and no text). The tool returned the account string, and the agent finished with an answer. Opening LangSmith now shows one new trace in the agent-improvement-loop project, with each step recorded.

Nothing in that code captured the trace. As soon as LANGSMITH_TRACING=true is in the environment, every call sends inputs, outputs and intermediate steps to LangSmith. That ambient observability is what the rest of the loop depends on.

The first stage leaves a tool-calling agent running under LangGraph with every run emitting a trace. A trace sitting in a dashboard isn't an improvement signal yet, though. It has to be pulled out and acted on, which is the job of the next stage.

Stage 2: collecting traces

This stage pulls traces out of LangSmith and into analysis code. A trace is one complete agent run, with inputs, outputs, tool calls and latency. The LangSmith Python client lists runs from the project, filters by time window and status, and turns them into objects that the scoring stage can use.

A few terms make the code easier to read:

1. A run is a single piece of recorded work, either an entire agent call or one LLM call nested inside it. One agent invocation forms a tree of runs.
2. A root run is the top-level run of one agent call, which is what gets listed when you want one record per user query.
3. A project is the namespace a run belongs to, set by LANGSMITH_PROJECT. Filtering by project keeps production runs apart from local development runs.

Pulling recent runs

The client is created first:

# Step 1: Create a LangSmith client
from langsmith import Client
from datetime import datetime, timedelta

client = Client()

Client is imported from the langsmith package and created with no arguments. It reads LANGSMITH_API_KEY from the environment on its own, so no credential gets passed.

Then recent root runs are listed:

# Step 2: List recent root runs from our project
recent_runs = list(client.list_runs(
    project_name="agent-improvement-loop",
    is_root=True,
    start_time=datetime.utcnow() - timedelta(days=1),
))

print(f"Found {len(recent_runs)} root runs in the last 24 hours")

client.list_runs is called with three filters. project_name scopes the query to the project, is_root=True keeps only top-level agent calls and not every nested LLM and tool call, and start_time limits results to the last 24 hours. The iterator is lazy, so list() forces it into a real list.

The expected output:

Found 42 root runs in the last 24 hours

Keeping only the fields that matter

A raw LangSmith run carries plenty of fields, but scoring needs just five things: the input, the output, the latency, any error, and the child runs revealing which tools were used.

# Step 3: Build a compact trace record from each run
def compact_trace(run):
    return {
        "run_id": str(run.id),
        "inputs": run.inputs,
        "outputs": run.outputs,
        "error": run.error,
        "latency_ms": (run.end_time - run.start_time).total_seconds() * 1000
            if run.end_time else None,
        "tool_calls": [
            child.name for child in client.list_runs(
                trace_id=run.trace_id, run_type="tool"
            )
        ],
    }

traces = [compact_trace(run) for run in recent_runs]
print(f"Collected {len(traces)} compact traces")
print("Example tool calls:", traces[0]["tool_calls"])

This helper accepts a raw run and boils it down to a small dict. Inputs and outputs (run.inputs, run.outputs) contain the user query and the final reply, while run.error is None on success and a string on failure. Latency comes from subtracting start_time from end_time. Tool calls need a second LangSmith query using run_type="tool" plus the same trace_id, which yields only the tool executions of that run tree.

The expected output:

Collected 42 compact traces
Example tool calls: ['get_account_info']

The idea isn't to copy traces out of LangSmith and abandon it. The compact version is simply easy to score, while the full trace stays in LangSmith for deep debugging. Keeping a rich view in the dashboard and a lean one for analysis is a sensible way to separate observability from evaluation.

Checking the collector

# Step 4: Inspect the first trace end to end
import json
print(json.dumps(traces[0], indent=2, default=str))

The first trace is dumped as indented JSON with json.dumps. The default=str argument makes json stringify anything it can't serialize on its own, which covers the datetime and UUID types LangSmith returns.

The expected output:

{
  "run_id": "b1d2e3f4-...",
  "inputs": {"messages": [{"role": "user", "content": "What plan is account A100 on?"}]},
  "outputs": {"messages": [{"role": "ai", "content": "Account A100 is on the pro plan."}]},
  "error": null,
  "latency_ms": 1423.51,
  "tool_calls": ["get_account_info"]
}

Each production run now has a clean, structured record, which the scoring stage will enrich. Collecting traces costs little, and the effort lies in making them useful. Many teams stop at the dashboard and wonder why observability never turns into improvement. Traces without scores are noise, and a trace without a score is just a log line.

Stage 3: scoring the traces

Three scoring layers feed a single enriched trace. Anthropic's engineering post on evals describes this same arrangement: code-based graders that are fast, objective and reproducible, model-based graders that are flexible and scalable, and human graders that are the gold standard but expensive. One of each gets built here.

1. Code-based checks are fast deterministic functions that assert structural facts, such as whether the agent called the right tool or whether the output has the expected format. They catch obvious failures at no cost.
2. An LLM judge is a second model that rates subjective qualities like helpfulness, relevance and tone. It scales because it runs on every trace, but it has to be calibrated against human judgment before anyone trusts it.
3. Human review puts a domain expert in front of the traces that automation missed or scored inconsistently. It is the ground truth that the other two layers get calibrated against.

Layer 1: tool correctness

# Step 1: Define a rule that says which tool SHOULD be called for a given query
import re

def expected_tool_for_query(query: str) -> str:
    q = query.lower()
    if re.search(r"\b(order|shipment|shipped|return)\b", q):
        return "get_recent_orders"
    if re.search(r"\b(plan|account|status|email)\b", q):
        return "get_account_info"
    return "any"

def score_tool_correctness(trace):
    query = trace["inputs"]["messages"][0]["content"]
    expected = expected_tool_for_query(query)
    actual = trace["tool_calls"][0] if trace["tool_calls"] else None
    if expected == "any":
        return 1.0
    return 1.0 if actual == expected else 0.0

Two functions are defined. expected_tool_for_query applies a regex to the query text to guess which tool ought to have handled it. That's a crude heuristic, fine for spotting blatant routing errors but not for production. score_tool_correctness takes the user query from the trace, works out the expected tool and compares it with the first tool the agent really called. A mismatch scores 0.0 and a match scores 1.0.

A production version would swap the regex for a stronger classifier or a rule engine keyed to intents. The principle stays: a deterministic check goes before the expensive LLM-based evaluators.

Layer 2: an LLM judge for helpfulness

# Step 2: Score qualitative properties with an LLM judge
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

judge_model = ChatOpenAI(model="gpt-4o-mini", temperature=0)

judge_prompt = ChatPromptTemplate.from_messages([
    ("system", "You are an expert reviewer of AI agent responses. "
               "Rate the response on a scale of 1 to 5 for helpfulness. "
               "Respond with only a single integer, nothing else."),
    ("user", "User question: {question}\n\nAgent response: {response}\n\nScore:"),
])

def score_helpfulness(trace):
    question = trace["inputs"]["messages"][0]["content"]
    response = trace["outputs"]["messages"][-1]["content"]
    chain = judge_prompt | judge_model
    result = chain.invoke({"question": question, "response": response})
    try:
        return int(result.content.strip()) / 5.0
    except ValueError:
        return 0.0

A small LangChain chain is built by piping a prompt template into the judge model. The system message limits the judge to returning one integer from 1 to 5, so the result parses deterministically. Inside score_helpfulness, the user question and the final agent response are pulled from the trace, the chain runs, and the integer is scaled to a float between 0 and 1 so it combines neatly with the tool score. gpt-4o-mini serves as the judge because it's cheap and fast.

LLM judges are stochastic, so two runs over the same trace can return slightly different scores. Anthropic's guidance recommends calibrating a judge against a human-labeled set before relying on it, and the third layer builds exactly that set.

Layer 3: human review as an annotation queue

LangSmith has a native annotation queue where people label traces in the dashboard. To keep the example self-contained, the same shape is modeled in code.

# Step 3: Model a human review signal
def score_human(trace, human_labels: dict) -> float | None:
    run_id = trace["run_id"]
    return human_labels.get(run_id)

human_labels = {
    "b1d2e3f4-...": 1.0,
    "c2e3f4g5-...": 0.0,
}

The function looks a run id up in a dict of human-provided labels. If a label exists it comes back, and if not the function returns None, meaning not yet reviewed. In production that dict would sit behind LangSmith's annotation queue API. What matters is how the scores behave: they are sparse because only some traces get them, they are authoritative because they override the others when there's a disagreement, and they drive calibration of the LLM judge.

Putting the three layers together

# Step 4: Run all three scorers and attach scores to each trace
def enrich(trace):
    trace["scores"] = {
        "tool_correctness": score_tool_correctness(trace),
        "helpfulness": score_helpfulness(trace),
        "human": score_human(trace, human_labels),
    }
    return trace

enriched = [enrich(t) for t in traces]
print(json.dumps(enriched[0]["scores"], indent=2))

The enrich function adds all three scores to a trace and hands it back. It is applied to each trace gathered during collection, giving a list of enriched traces that each hold a structured scores dict. The rule-based regex produces tool_correctness, the LLM judge produces helpfulness, and human is a float or None.

The expected output:

{
  "tool_correctness": 1.0,
  "helpfulness": 0.8,
  "human": null
}

The three scores disagree on purpose. Tool correctness says the right tool was called, and the LLM judge says the answer deserves 4 out of 5. A human reviewer who joins later might say the tool was right but the phrasing didn't suit the brand's tone. Each layer catches what the others miss, so stacking them buys coverage and ground truth at different prices.

Now each trace holds three scores: a rule-based one, an LLM-judged one and, where available, a human one. Scores with no structure are nearly as unhelpful as having none, so patterns come next.

Stage 4: finding failure patterns

Once traces carry scores, the same failure tends to appear repeatedly. The agent picks the wrong tool for one particular phrasing. Its reasoning drifts on multi-step queries. Its answers are technically right yet miss what the user wanted. Each pattern calls for a different kind of fix, whether a rewritten tool description, a prompt update or a restructured workflow. Naming the pattern is the hard part, and fixing it afterward is usually simple.

LangChain offers an Insights Agent built for exactly this job at scale. It ingests traces, groups them by how they failed and helps teams rank what deserves attention first. A stripped-down version of the same concept is built in code below.

Filtering for failures

# Step 1: Keep only traces where at least one score is low
def is_low_scoring(trace):
    scores = trace["scores"]
    if scores["human"] is not None and scores["human"] < 0.5:
        return True
    if scores["tool_correctness"] < 0.5:
        return True
    if scores["helpfulness"] < 0.6:
        return True
    return False

failures = [t for t in enriched if is_low_scoring(t)]
print(f"{len(failures)} of {len(enriched)} traces flagged as low scoring")

is_low_scoring makes three checks. A human score below 0.5 marks the trace as a failure. Otherwise the automated scores decide: tool correctness below 0.5 means the agent routed wrongly, and helpfulness below 0.6 means the LLM judge disliked the answer. The thresholds can be adjusted, and these are sensible first-pass defaults. The filter runs over the enriched traces from the previous stage to produce the failure set.

The expected output:

9 of 42 traces flagged as low scoring

Grouping failures into categories

A small heuristic tags the failures here. In production, embeddings with clustering would do the job, or LangChain's Insights Agent, which does exactly that on top of LangSmith traces.

# Step 2: Tag each failure with a category based on its scores
def categorize(trace):
    scores = trace["scores"]
    if scores["tool_correctness"] < 0.5:
        return "wrong_tool"
    if scores["helpfulness"] < 0.6 and scores["tool_correctness"] >= 0.5:
        return "unhelpful_answer"
    if scores["human"] is not None and scores["human"] < 0.5:
        return "human_flagged"
    return "other"

from collections import Counter
categories = Counter(categorize(t) for t in failures)
print(categories.most_common())

The categorize function checks in priority order. A wrong tool is the most structural problem, so it is tagged first. An unhelpful answer that used the right tool suggests trouble in the prompt or response generation. A failure flagged by a human that the automation let through lands in its own bucket. Python's standard Counter tallies each category, and it is imported right there because this is its first use.

The expected output:

[('wrong_tool', 5), ('unhelpful_answer', 3), ('human_flagged', 1)]

The category hints at the remedy. A run of five wrong-tool failures points at the tool description or the routing prompt. Three unhelpful answers despite the correct tool point at the prompt that generates responses. A failure flagged only by a human means the automated evaluators overlooked something, so a new evaluator is probably due.

Looking at a single pattern

# Step 3: Print the inputs and outputs for every wrong_tool failure
wrong_tool = [t for t in failures if categorize(t) == "wrong_tool"]
for trace in wrong_tool[:3]:
    print("QUERY:", trace["inputs"]["messages"][0]["content"])
    print("TOOLS CALLED:", trace["tool_calls"])
    print("RESPONSE:", trace["outputs"]["messages"][-1]["content"])
    print("---")

The failure set is narrowed to wrong_tool cases, and the loop prints the user query, the tools actually called and the final response for the first three. This is where a person or a smarter LLM studies the failures and forms a hypothesis, for example that every query containing "subscription" triggers get_recent_orders instead of get_account_info. That hypothesis becomes the fix target.

The expected output:

QUERY: Is my subscription active?
TOOLS CALLED: ['get_recent_orders']
RESPONSE: I could not find any recent orders to confirm your subscription status.
---
QUERY: Tell me about my subscription to your product
TOOLS CALLED: ['get_recent_orders']
RESPONSE: There are no recent orders for this account.
---

Three examples on screen make the pattern plain. "Subscription" is semantically closer to "account status" than to "orders", yet the agent routes on shallow keyword overlap. The probable remedy is a single-line change to the get_account_info docstring adding subscription wording, or a short system prompt addition teaching the difference.

The stage ends with failure patterns that have counts, categories and examples. A pattern fixed today is worth little if a different engineer's change breaks it again next week, so the fix has to be pinned down with a test.

Stage 5: offline test suites

Here live failures turn into static tests. Every real production failure is added to a LangSmith dataset as an example holding the input (the user query), the behavior expected (the right tool, or an acceptable reply) and optional metadata like the failure category. Once stored, it never leaves, so a fixed failure cannot quietly return.

An offline evaluation in LangSmith has three parts:

1. A dataset is a named collection of examples, each with inputs and, optionally, expected outputs.
2. The target function is the code under test, which for an agent is usually its .invoke method.
3. Evaluators are functions that score the target's output against each example, shaped like the scorers from the scoring stage.

Creating the dataset

# Step 1: Create a LangSmith dataset for our agent
dataset_name = "agent-production-failures"

dataset = client.create_dataset(
    dataset_name=dataset_name,
    description="Production traces where the agent failed. Each example captures the user query and the expected tool call.",
)
print(f"Created dataset: {dataset.id}")

The same client from the collection stage calls create_dataset with a name and a description, and returns a dataset object whose .id can be referenced later. A real deployment would guard this with a create-if-not-exists check, but the demo assumes a first run.

The expected output:

Created dataset: 3f2a1b0c-...

Turning failures into examples

# Step 2: Convert each failure trace into a dataset example
examples = []
for trace in failures:
    query = trace["inputs"]["messages"][0]["content"]
    expected_tool = expected_tool_for_query(query)
    examples.append({
        "inputs": {"question": query},
        "outputs": {"expected_tool": expected_tool},
        "metadata": {"category": categorize(trace), "source_run_id": trace["run_id"]},
    })

client.create_examples(dataset_id=dataset.id, examples=examples)
print(f"Added {len(examples)} examples to {dataset_name}")

The code loops over every failure trace and builds an example dict with three fields. inputs carries the user question, outputs carries the behavior to test for, which here is the tool that should have been called, and metadata carries the category and the original run id so each example can be traced back to its source. A single batched client.create_examples call uploads everything.

The expected output:

Added 9 examples to agent-production-failures

Real deployments usually let a reviewer decide the expected output, since the regex heuristic is only a starting point. A person approving each example gives firmer ground truth, which is what LangSmith's annotation queue is for.

The regression evaluator

# Step 3: Write an evaluator that checks the agent's tool call against the expected tool
def tool_match_evaluator(inputs: dict, outputs: dict, reference_outputs: dict):
    actual_messages = outputs.get("messages", [])
    tool_called = None
    for m in actual_messages:
        if getattr(m, "tool_calls", None):
            tool_called = m.tool_calls[0]["name"]
            break
    expected = reference_outputs["expected_tool"]
    score = 1.0 if (expected == "any" or tool_called == expected) else 0.0
    return {"key": "tool_match", "score": score}

tool_match_evaluator follows the signature LangSmith expects: it receives the example's inputs, the agent's actual outputs and the example's reference_outputs. It walks the agent's output messages, finds the first one carrying a tool call and extracts the tool's name. It then compares that with the expected tool and returns a dict with a key and a score. The key becomes the metric name displayed in LangSmith. The shape follows the official LangSmith evaluation quickstart.

The target function

# Step 4: Wrap our agent in a function that takes a LangSmith example and returns its output
def run_agent(inputs: dict) -> dict:
    result = agent.invoke({
        "messages": [{"role": "user", "content": inputs["question"]}]
    })
    return result

run_agent is deliberately kept minimal. Given the inputs of a dataset example, it calls the agent and returns whatever raw dict comes back, which matches what LangSmith's evaluate function expects. The agent being called was created during the foundation stage.

Keeping the wrapper thin is deliberate. Anything placed inside it runs on every evaluation and inflates the cost, and a plain pass-through means the evaluator tests the agent and not the wrapper.

So the dataset exists, the evaluator is connected, and each production failure now sits there as a lasting test case. The final stage runs that evaluator ahead of any change going out.

Stage 6: closing the loop

This is the regression gate. The agent gets a change, for example an updated tool description aimed at the "subscription" routing bug found earlier. Before shipping, the whole dataset of past failures is run through the new version. LangSmith stores both runs as experiments and displays, example by example, where the new version improved and where it slipped. Shipping happens only when the new baseline beats the old one and nothing previously fixed regressed.

Running the baseline evaluation

# Step 1: Run the evaluator across the dataset with the current agent
from langsmith.evaluation import evaluate

experiment_results = evaluate(
    run_agent,
    data=dataset_name,
    evaluators=[tool_match_evaluator],
    experiment_prefix="agent-v1-baseline",
    max_concurrency=4,
)

evaluate is imported from langsmith.evaluation. It's given the run_agent target, the dataset name (which LangSmith resolves to the dataset from the previous stage), the list of evaluators, an experiment prefix that labels the run and a concurrency of four so the evaluation runs in parallel. It returns an experiment results object pointing to the run in LangSmith.

The expected output:

View the evaluation results for experiment: 'agent-v1-baseline-...' at:
https://smith.langchain.com/o/.../datasets/.../compare?selectedSessions=...

9/9 runs | avg tool_match: 0.33

Of nine failing examples, the current agent still routes most of them wrongly, and that's the baseline.

Shipping a fix and measuring it again

The fix suggested by the pattern stage is to rewrite the get_account_info docstring with subscription language.

# Step 2: Update the tool docstring to teach the agent about subscriptions
@tool
def get_account_info_v2(account_id: str) -> str:
    """Look up basic information for an account, including plan, status,
    subscription tier, and contact email. Use this tool for any question
    about the account itself, including subscription status."""
    fake_accounts = {
        "A100": "Account A100: plan=pro, status=active, email=ada@example.com",
        "A200": "Account A200: plan=free, status=suspended, email=turing@example.com",
    }
    return fake_accounts.get(account_id, f"No account found with id {account_id}")

tools_v2 = [get_account_info_v2, get_recent_orders]
agent_v2 = create_react_agent(model, tools_v2)

def run_agent_v2(inputs: dict) -> dict:
    result = agent_v2.invoke({
        "messages": [{"role": "user", "content": inputs["question"]}]
    })
    return result

The account info tool is redefined with a richer docstring that explicitly mentions subscription status. Since the docstring is what the LLM reads when picking a tool, widening it directly addresses the routing failure. The agent is rebuilt with the updated tool and wrapped in a new run_agent_v2 target. Everything else is unchanged.

# Step 3: Run the evaluator on the v2 agent
experiment_results_v2 = evaluate(
    run_agent_v2,
    data=dataset_name,
    evaluators=[tool_match_evaluator],
    experiment_prefix="agent-v2-docstring-fix",
    max_concurrency=4,
)

The identical evaluation runs again on the same dataset with the same evaluator, and only the target changes. A new experiment_prefix keeps the run apart in the LangSmith UI so it can be compared against the baseline. LangSmith pushes each example through the v2 agent, stores each full trace and scores every output with the evaluator.

The expected output:

View the evaluation results for experiment: 'agent-v2-docstring-fix-...' at:
https://smith.langchain.com/o/.../datasets/.../compare?selectedSessions=...

9/9 runs | avg tool_match: 0.89

The score rose from 0.33 to 0.89. Eight of the nine failures are fixed, and the ninth is a truly ambiguous query that might belong in the human annotation queue.

Before (v1):       After (v2):
wrong_tool: 5      wrong_tool: 1
unhelpful: 3       unhelpful: 0
human: 1           human: 0
avg score: 0.33    avg score: 0.89

Several things made this comparison trustworthy:

1. The baseline evaluation produced a concrete score before any code changed.
2. The fix was one surgical edit, a single docstring.
3. The re-evaluation measured that edit's exact effect against a curated set of real failures.
4. The test dataset stayed fixed, so the comparison was like for like.
5. Each original failure trace is now pinned as a test, so no later change can silently break it.

That's what makes the loop compound. Every fix adds examples to the dataset, and every future version is measured against every past failure. Shipping gets faster over time and not slower, because confidence grows along with the dataset.

The whole loop now runs end to end. Traces are collected and enriched with scores, patterns surface, failures become offline tests, evaluation precedes shipping, and a change is promoted only when the baseline climbs. Each cycle begins from a higher floor.

One full daily run

A single driver can run the whole loop once on a fresh day of production data.

# Full loop driver
def run_improvement_loop():
    # 1. Pull traces
    runs = list(client.list_runs(
        project_name="agent-improvement-loop",
        is_root=True,
        start_time=datetime.utcnow() - timedelta(days=1),
    ))
    traces = [compact_trace(r) for r in runs]

    # 2. Enrich with scores
    enriched = [enrich(t) for t in traces]

    # 3. Find failures and categorize them
    failures = [t for t in enriched if is_low_scoring(t)]
    categories = Counter(categorize(t) for t in failures)

    # 4. Add failures to the dataset
    new_examples = [{
        "inputs": {"question": t["inputs"]["messages"][0]["content"]},
        "outputs": {"expected_tool": expected_tool_for_query(
            t["inputs"]["messages"][0]["content"])},
        "metadata": {"category": categorize(t), "source_run_id": t["run_id"]},
    } for t in failures]

    if new_examples:
        client.create_examples(dataset_id=dataset.id, examples=new_examples)

    # 5. Re-evaluate the current agent against the full dataset
    result = evaluate(
        run_agent_v2,
        data=dataset_name,
        evaluators=[tool_match_evaluator],
        experiment_prefix="daily-regression",
    )

    return {
        "traces_collected": len(traces),
        "failures_found": len(failures),
        "by_category": dict(categories),
        "examples_added": len(new_examples),
    }

print(run_improvement_loop())

This function glues the six stages together. It pulls yesterday's traces, scores them, categorizes the failures, appends new failures as dataset examples and runs a regression evaluation. It returns a small dashboard dict that can be logged or posted to Slack each day.

The expected output:

{
  "traces_collected": 186,
  "failures_found": 12,
  "by_category": {"wrong_tool": 4, "unhelpful_answer": 6, "human_flagged": 2},
  "examples_added": 12,
  "regression_score": 0.87
}

What the output shows: the day yielded 186 production traces with no change to the agent code, and twelve of them drew at least one poor score. There are now more unhelpful answers than wrong tool calls, hinting that the routing fix paid off and that the following fix belongs in response generation. The regression score stayed at 0.87, so none of the week's changes broke an earlier failure. Twelve fresh examples joined the permanent dataset, where they will guard against repeats.

Five upgrades worth making next

Each of these maps onto a LangChain, LangGraph or LangSmith feature that can be turned on without rebuilding the architecture above.

1. Replace the regex classifier with embeddings. OpenAIEmbeddings can embed each query so failure traces cluster by semantic similarity and not by keyword rules, which exposes subtler problems like tone mismatches.
2. Swap the inline LLM judge for the OpenEvals library. The LangSmith quickstart uses openevals with ready-made prompts such as CORRECTNESS_PROMPT, which holds up better than a hand-rolled judge and comes with reference-free variants for online monitoring.
3. Add multi-turn evaluations. LangSmith's multi-turn evals run an LLM judge across entire conversations and not single replies, catching reasoning drift that single-turn evaluators miss in chat-based agents.
4. Bring in LangChain's Insights Agent for pattern discovery at scale. The regex categorizer copes with dozens of failures, but with thousands, pointing the Insights Agent at the project clusters the traces and surfaces failure modes automatically, feeding the same dataset structure.
5. Promote through a LangGraph deployment with a staged rollout. Keeping v2 in a canary project set with LANGSMITH_PROJECT=agent-canary for a day, then comparing its live scores with the control project, means it only replaces production if it's strictly better.

The loop then repeats from a higher baseline each time, with better traces, better evaluations and a better agent.