Agent and Skill Evals in Practice: Trajectories, Graders, pass^k and Tools

Sooner or later, every team shipping agents and skills runs into one wall: the demo impresses, and then real traffic uncovers failures nobody noticed. Evals replace guessing with measurement. What follows is a practical primer on what to measure, how to write cases and graders, and how to run an improvement loop that makes an agent genuinely better.

The mental model

An eval is a repeatable experiment. Give an agent an input and an environment, then watch two things: the trajectory, meaning what the agent did along the way, and the outcome, meaning the final artifact it produced.

Judge both:

• Outcome: does the finished artifact satisfy the task?
• Trajectory: did the agent choose the right skill and tools, steer clear of unsafe actions and work efficiently?

A polished final answer can conceal a poor trajectory, and a clumsy trajectory can still stumble into a passing result. Looking at only one tells half the story.

The layers of evaluation

No single number says an agent is good. Split the question into layers, each with its own check:

• Skill routing. Was the right skill picked? Look at precision, recall and false triggers.
• Skill adherence. Were the skill's instructions actually followed, including required steps and forbidden actions?
• Tool use. Were tools called correctly, with proper arguments, sensible ordering and no needless calls?
• Task outcome. Does the result work? Tests, schema checks and behavioral assertions answer that.
• Quality. Is the result maintainable and fitting? A rubric or a reviewer decides.
• Safety. Were boundaries respected around secrets, destructive operations and prompt injection?
• Efficiency. What did success cost in tokens, latency, calls and retries?

A skill needs two suites

A skill is more than a block of instructions. It embodies a decision about when to activate and a plan for what to do afterward, so it needs two separate evaluation suites.

Trigger evals check whether the skill fires for the right prompts. Each case pairs a prompt with the skill expected, or a skill that must not fire:

- prompt: "Patch the vulnerable npm dependencies"
  expected_skill: cve-remediation
- prompt: "Review authentication input validation"
  forbidden_skill: cve-remediation

Cover obvious positives, reworded positives, near-miss negatives, ambiguous prompts and clashes between two plausible skills. Report a confusion matrix instead of plain accuracy, because a false positive can hurt as much as a missed trigger.

Execution evals check whether using the skill improves the work. Pit an agent that lacks the skill against that same agent equipped with it, using identical cases, environment, model and tools, and repeat the runs because agents are nondeterministic. A skill earns its place only when the improvement justifies its added complexity and token cost.

What a good case looks like

Favor realistic repository snapshots over isolated trivia. A good case is self-contained and can be checked by a machine:

{
  "id": "fix-null-condition",
  "prompt": "Fix saving rules with a null condition",
  "fixture": "repos/null-condition",
  "setup": ["npm install"],
  "checks": [
    "npm test -- null-condition.test.js",
    "git diff --check"
  ],
  "rubric": [
    "Fixes the root cause",
    "Preserves existing behavior",
    "Adds a regression test",
    "Avoids unrelated changes"
  ],
  "forbidden": [
    "deleting existing tests",
    "hard-coded fixture-specific output"
  ]
}

Graders: use the strongest one you have

Prefer the most deterministic grader available, and fall back to judgment only when forced. Roughly from strongest to weakest:

1. exact assertions
2. compilation and unit or integration tests
3. schema or syntax-tree validation
4. behavioral checks through a browser or API
5. static inspection of the trajectory
6. an LLM judging against a rubric
7. human review

LLM judges suit qualities like clarity and architectural fit, yet they shouldn't stand in for executable checks. Hide from the judge which variant produced a result, and shuffle the order of comparisons.

Metrics worth tracking

• Success rate is the share of runs passing every critical check.
• Pass@k is the probability that at least one of k attempts works.
• Pass^k is the probability that all k attempts work, which is the truest signal of reliability.
• Routing precision and recall describe skill-selection quality.
• Regression rate counts cases that used to pass and now fail.
• Cost per successful task, median and tail latency, tool-call counts, retry rates and how often a human had to step in all belong on the dashboard too.

For production agents, steady reliability usually beats impressive best-case results.

Building the dataset

Begin with 20 to 50 cases drawn from actual work: earlier failures and support incidents, everyday tasks, edge cases, misleading or adversarial requests, underspecified prompts, tasks that ought to be refused or clarified, and situations where no skill should activate.

Then divide them into three sets:

• a development set you can see while improving the skill
• a holdout set left untouched until validation
• a regression set that receives every failure you discover, permanently

Avoid optimizing only against one fixed public set, since agents overfit to its wording quickly.

A working loop

Think of it as test-driven development for agent behavior:

1. Capture a real failure.
2. Shrink it into a reproducible fixture.
3. Write the grader before touching the skill or agent.
4. Run several baseline trials.
5. Make one targeted change.
6. Rerun the target case plus the whole regression suite.
7. Compare quality, reliability, latency and cost.
8. Keep the change only if the overall balance improves.

Common mistakes

• grading only the final prose
• using only happy-path prompts
• checking whether instructions were repeated rather than followed
• asking an LLM judge about facts that a test could verify
• running once and treating the result as deterministic
• changing prompt, model, tools and skill all at once
• accepting deleted tests or tampering with the grader as success
• using synthetic cases that look nothing like real user work
• overlooking prompt injection hidden in files or tool output

Routing metrics in concrete terms

Suppose a security-review skill exists and an evaluator decides, for each prompt, whether it should trigger. Four outcomes are possible:

• True positive (TP): it should trigger and did.
• False positive (FP): it shouldn't have triggered but did, a false trigger.
• False negative (FN): it should have triggered and didn't.
• True negative (TN): it shouldn't have triggered and didn't.

Precision asks how often a trigger was correct: TP divided by (TP + FP). If the skill fired ten times and eight were right, precision is 80%. High precision means few false triggers.

Recall asks how many of the prompts that needed the skill it actually caught: TP divided by (TP + FN). If ten prompts needed it and seven were caught, recall is 70%.

A false trigger is simply a false positive. A prompt like "Upgrade React dependencies" wrongly invoking security-review is one. Many false triggers mean low precision.

A confusion matrix tallies everything. With TP = 8, FN = 2, FP = 3 and TN = 7, prompts that should trigger split 8 and 2, and prompts that shouldn't split 3 and 7. Precision is 8 / 11, about 72.7%, and recall is 8 / 10, or 80%. High precision means the skill seldom activates by mistake. High recall means it seldom misses a relevant request. Broad trigger wording usually lifts recall and lowers precision.

Where to start

Per important skill, a compact kit goes a long way. Gather ten prompts that should trigger it and ten near misses that shouldn't, plus five realistic tasks to execute. Add deterministic outcome checks, three to five repeats of each task, a baseline run without the skill, and a regression suite that retains each failure you uncover.

A setup that small exposes most routing problems and shows whether the skill truly helps instead of merely sounding sophisticated.

A minimal framework you can copy

Enough theory. A small harness in plain Python with only the standard library can evaluate both a skill's routing and an agent's task outcome. At roughly 70 lines, it shows each idea above: case files, layered graders, repeated runs and reliability metrics.

The folder layout:

evals/
  cases/
    skills/security-review.trigger.json
    agents/fix-null-condition.json
  graders.py
  metrics.py
  runner.py
  run.py        # CLI entry1. Case files. A skill trigger case is just positives and negatives. An agent case is a prompt plus graders and a run count.

A trigger case file lists the skill name with prompts that should and shouldn't fire it:

// cases/skills/security-review.trigger.json
{
  "skill": "security-review",
  "positives": [
    "Audit this endpoint for SQL injection",
    "Check the login flow for auth bypasses",
    "Is this file-upload handler safe?"
  ],
  "negatives": [
    "Upgrade React dependencies",
    "Rename this variable across the repo",
    "Add a loading spinner to the form"
  ]
}

An agent case file carries an id, a prompt, a fixture and the checks to apply:

// cases/agents/fix-null-condition.json
{
  "id": "fix-null-condition",
  "prompt": "Fix saving rules with a null condition",
  "fixture": "fixtures/null-condition",
  "setup": ["npm ci"],
  "runs": 5,
  "graders": [
    { "type": "shell", "cmd": "npm test -- null-condition", "critical": true },
    { "type": "shell", "cmd": "git diff --check" },
    { "type": "forbidden", "pattern": "it\\.skip|xit\\(", "message": "must not disable tests" }
  ]
}

Graders are next. One runs shell commands such as the test suite, another scans the trajectory for edits that were off limits, and a judge model that can't see which variant it's scoring handles the subjective qualities.

# graders.py
import re, subprocess
def shell(step, ctx):
    r = subprocess.run(step["cmd"], cwd=ctx.workdir, shell=True,
                       capture_output=True, text=True)
    ok = r.returncode == 0
    return {"pass": ok, "score": 1 if ok else 0, "detail": r.stdout[-400:]}
def forbidden(step, ctx):
    bad = re.search(step["pattern"], ctx.diff) is not None
    return {"pass": not bad, "score": 0 if bad else 1,
            "detail": step["message"] if bad else ""}
def judge(step, ctx):
    v = ctx.llm.rate(step["rubric"], ctx.artifact)  # 0..1, blinded
    return {"pass": v >= step.get("threshold", 0.7), "score": v}
GRADERS = {"shell": shell, "forbidden": forbidden, "judge": judge}3. Metrics. Success rate, pass@k, pass^k, and the routing confusion matrix — exactly the numbers from earlier.

The metrics module turns lists of run results into numbers:

# metrics.py
def success_rate(rs):
    return sum(1 for r in rs if r["pass"]) / len(rs)
def pass_at_k(rs):
    return 1 if any(r["pass"] for r in rs) else 0
def pass_hat_k(rs):
    return 1 if all(r["pass"] for r in rs) else 0
def routing(positives, negatives, fired):
    tp = sum(1 for p in positives if fired(p))
    fp = sum(1 for n in negatives if fired(n))
    fn = len(positives) - tp
    tn = len(negatives) - fp
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": tp / (tp + fp or 1),
            "recall": tp / (tp + fn or 1)}4. The runner. One function per suite. The agent runner repeats each case so pass^k is meaningful; the skill runner just asks the router which skills fire.

A runner loads the cases, calls the graders, repeats each run and prints the metrics:

# runner.py
import json
from graders import GRADERS
from metrics import success_rate, pass_at_k, pass_hat_k, routing
def run_agent_case(agent, path):
    c = json.load(open(path))
    runs = []
    for _ in range(c.get("runs", 3)):
        ctx = agent.run(c["prompt"], fixture=c["fixture"], setup=c["setup"])
        ok = True
        for step in c["graders"]:
            g = GRADERS[step["type"]](step, ctx)
            if step.get("critical") and not g["pass"]:
                ok = False
        runs.append({"pass": ok})
    return {"id": c["id"], "success": success_rate(runs),
            "pass_at_k": pass_at_k(runs), "pass_hat_k": pass_hat_k(runs)}
def run_skill_trigger(router, path):
    c = json.load(open(path))
    fired = lambda prompt: c["skill"] in router.route(prompt)
    return {"skill": c["skill"], **routing(c["positives"], c["negatives"], fired)}5. Run it. The CLI just dispatches on the case type and prints the metrics.

Running it produces output of this kind:

$ python run.py cases/agents/fix-null-condition.json
fix-null-condition   success=0.80   pass@5=1.00   pass^5=0.20
$ python run.py cases/skills/security-review.trigger.json
security-review      precision=0.86  recall=1.00   (tp=6 fp=1 fn=0 tn=5)

Read those lines the way this whole approach recommends. The agent passes four runs in five, so pass@5 looks perfect, yet pass^5 of 0.20 says it isn't reliable yet. The skill catches every genuine request, with recall 1.0, but fired once when it shouldn't have, with precision 0.86, so there's one false trigger to chase.

That is the point of evals: converting "it seemed to work" into numbers you can improve. Start from a harness like this, add a case whenever something breaks, and let the regression set grow into your working definition of good.

Use an existing framework when you can

A homemade harness is good for learning the mechanics and for full control, but mature open projects already provide runs, graders, datasets and dashboards. Look at them before building your own.

General prompt and LLM evals

OpenAI Evals is the original registry-style framework, defining evals in YAML with Python graders against any model, and it suits output-level and classification checks. promptfoo is configuration-driven, shows prompts against models in a side-by-side matrix, ships assertions such as contains, regex, JSON schema and LLM rubric, and has a red-team mode, which makes it a strong start for skill routing and prompt regressions. DeepEval behaves like pytest for LLMs, with faithfulness, relevancy and hallucination metrics plus custom G-Eval rubrics, and it slots into CI naturally. Ragas targets RAG pipelines specifically, scoring context precision and recall, faithfulness and answer relevancy. MLflow's LLM evaluation adds tracking and comparison if MLflow is already in use.

Tracing, datasets and judge platforms

LangSmith offers datasets, trajectory tracing and evaluators, useful when you must inspect a whole trace instead of just the answer. Langfuse is open source, self-hostable and model-agnostic, with tracing, evals and datasets for production monitoring. Braintrust treats evals as CI, with scoring functions, versioned datasets and a diff view for experiments. Arize Phoenix builds on OpenTelemetry tracing and includes eval templates, handy for spotting regressions in live traffic. Inspect, from the UK AI Safety Institute, is a carefully designed solver and scorer framework gaining ground for serious agent and safety evaluations.

Agent benchmarks

SWE-bench and SWE-bench Verified are the standard for whether an agent can fix a real GitHub issue, using the repository's own tests as the grader. AgentBench, WebArena, GAIA and τ-bench provide external yardsticks for tool use, web tasks and multi-step agents. Frameworks such as AutoGen and LangGraph expose test hooks for asserting on tool calls and intermediate steps.

Choosing

• For prompts and skill routing alone, promptfoo is the fastest route to configuration plus assertions.
• For code-first, CI-native metrics, consider DeepEval or Braintrust.
• To grade the trajectory of multi-step agents, look at LangSmith, Langfuse or Inspect.
• For RAG, add Ragas.
• For a real coding-agent benchmark, use SWE-bench Verified.

The pattern holds everywhere: cases, graders, repeated runs and tracking over time. A framework saves the plumbing, so keep the discipline and pick the tool that fits your stack. The custom harness is best reserved for gaps those tools leave, or for projects that want zero dependencies in CI.