Agent Evals With a Travel Bot: Outcome vs Trajectory, Gates and pass^k

Agents don't just call APIs. They reason, pick tools on the fly, build multi-step plans and tackle open-ended jobs that used to eat hours of human time. The moment one reaches production, though, an uncomfortable question appears: how do you know it works?

Not on your favorite demo prompt today, but next week, after someone edits the system instructions, adds a tool, swaps the underlying model or tweaks the prompt. Did that change fix one thing and quietly break three others?

Ordinary software has deterministic unit tests, such as assert add(2, 2) == 4. Agents are probabilistic and can walk different paths to the same goal, so a plain string comparison on the final text falls short. Agent evals fill that gap.

The math exam analogy

Picture a teacher who marks an exam by glancing only at the final number. Two things go wrong:

1. The lucky guess. The working is nonsense, two arithmetic slips cancel out, and the answer lands right by accident.
2. The last-line slip. The method is flawless and the theorem understood, but a typo at the very end writes 41 where 42 belongs.

A good teacher marks both the method and the answer. Agents deserve the same treatment, because an agent produces a sequence of steps: tool calls, searches, formatted objects and decisions about state. Judging it fairly means measuring two things:

• Outcome: the business result it delivered.
• Trajectory: the path it took to get there.

Put simply, agent evaluation is outcome evaluation plus trajectory evaluation, measured on the same trial using a mix of hard code checks and flexible judgment checks.

Two axes for grading

Any agent can be examined along two independent dimensions.

What is being looked at?

• Outcome checks ask whether the right final result exists, for instance the correct flight, the correct hotel rate or the right booking record.
• Trajectory checks ask whether the agent got there reliably: tools called in a sensible order, valid parameters passed, no endless looping.

How is it graded?

• Deterministic checks cover facts with an objective answer. A price is $150 or it isn't, and a tool was called or it wasn't. Plain code assertions handle these.
• Judgment-based checks cover anything needing context, intent or nuance. Did "budget" get read as "economy"? Does the chosen hotel match what the user actually meant? An LLM acting as a neutral evaluator grades these.

A worked example: a travel booking agent

Take an agent that handles itineraries. The request: a cheap flight, three hotel nights in New York and a ride from the airport. To do that it may use tools named search_flights, search_hotels, book_flight, book_hotel and book_airport_transfer.

Trajectory metrics read the execution log, call order, tool arguments and reasoning traces. Typical questions: Did it form a plan before acting? Were the searches done before the bookings? Do the hotel dates match the flight dates? Did it avoid redundant calls?

Outcome metrics inspect the booking objects the agent actually created: the fare, the nightly rate, the transfer, the currency and the dates.

Gates before averages

A lesson that tends to be learned painfully: not every metric belongs in one averaged score. Imagine an agent that scores 95% overall thanks to elegant reasoning and a perfect tool order, yet books the flight at $1,500 instead of $150. In production that isn't an A-grade run. It's a billing disaster.

To stop strong reasoning scores from hiding business-critical failures, evaluation can run in two stages.

Stage 1, the gate. Deterministic checks run first on critical fields: prices, dates, currency, billing identifiers and fee triggers. If any gated check fails, the whole trial fails at once. No amount of good reasoning can overturn that.

Stage 2, the weighted rubric. Only trials that clear every gate get a quality score, computed across the remaining metrics such as tool efficiency, reasoning quality, plan adherence and semantic match. The rule is simple: gate passed and weighted score at or above the threshold, say 0.85, means pass.

Diagnosing failures with a 2x2 matrix

A single "82% pass rate" tells you nothing useful for debugging. You need to know why runs failed, so you can decide whether to fix the prompt, the tools or the dataset. Plot outcome against trajectory and four quadrants appear:

• Good path, good result: the target.
• Bad path, good result: a lucky guess, and a risk waiting to happen.
• Good path, bad result: a last-mile slip, often a tool or data problem.
• Bad path, bad result: a fundamental failure that needs rework.

An end-to-end trace

Here is how the pipeline would handle one trial of the booking agent. It starts with the user's request and the agent's trace, which opens with a plan (search flights, search hotels, book both, add the transfer) and then records every thought and tool call in order.

// Step 1: Upfront Plan Formulation
Thought: "I need to book budget travel to NYC for 3 days. Plan: 1) Search economy flights, 2) Search budget hotels in NYC, 3) Book flight, 4) Book hotel for matching dates, 5) Book airport transfer."

// Step 2: Tool Call - search_flights
→ call: search_flights({ origin: "SFO", destination: "JFK", cabin_class: "economy", date: "2026-10-10" })
← return:
[
  {
    flight_id: "fl_eco_9921",
    airline: "BudgetAir",
    price: 150.00
  }
]

// Step 3: Tool Call - search_hotels
→ call: search_hotels({ city: "New York", checkin: "2026-10-10", nights: 3, max_rate: 150 })
← return:
[
  {
    hotel_id: "ht_manhattan_pod",
    name: "Pod Midtown NYC",
    rate_per_night: 120.00
  }
]

// Step 4: Tool Call - book_flight
→ call: book_flight({ flight_id: "fl_eco_9921", passenger: "Chris Dias", price: 150.00 })
← return:
{
  booking_ref: "BK-FL-7701",
  status: "confirmed"
}

// Step 5: Tool Call - book_hotel
→ call: book_hotel({ hotel_id: "ht_manhattan_pod", nights: 3, rate_per_night: 120.00, guest: "Chris Dias" })
← return:
{
  reservation_id: "RSV-HT-8832",
  status: "confirmed"
}

// Step 6: Tool Call - book_airport_transfer
→ call: book_airport_transfer({ pickup: "JFK", dropoff: "Pod Midtown NYC", fee: 35.00 })
← return:
{
  ride_id: "TX-4402",
  status: "confirmed"
}

// Step 7: Final Synthesized Output Payload
Final Output:
{
  flight: {
    id: "fl_eco_9921",
    cabin: "economy",
    price: 150.00,
    status: "confirmed"
  },
  hotel: {
    id: "ht_manhattan_pod",
    nights: 3,
    rate_per_night: 120.00,
    status: "confirmed"
  },
  transfer: {
    service: "JFK Pickup Shuttle",
    fee: 35.00,
    status: "confirmed"
  },
  total_billed: 545.00
}

Next, the gate looks at the mission-critical fields: flight price, hotel rate per night and transfer cost. Suppose all three match the expected values of $150, $120 per night and $35. The gate passes, and the trial moves on to Stage 2.

In Stage 2, each remaining metric receives a score between 0 and 1 and is multiplied by its weight, with the weights summing to 1. As an illustration, suppose five metrics carry weights of 0.20, 0.20, 0.15, 0.15 and 0.30 and score 1.0, 1.0, 0.8, 1.0 and 0.95. The weighted total is 0.20 + 0.20 + 0.12 + 0.15 + 0.285, which equals 0.955. Against a threshold of 0.85 declared before the suite ran, that's a pass.

The verdict then records the three facts: the gate passed, the weighted score cleared the threshold, and the diagnostic matrix puts the run in the good-path, good-result corner.

Measuring variance with pass@k and pass^k

One trial doesn't tell the full story, because agents are non-deterministic. Run the same prompt with the same tools ten minutes later and the model may search hotels first, format an argument differently or reason along another chain. A test that passed today can fail tomorrow on identical input.

Anthropic's engineering write-up on evaluating agents describes two statistics for taming that variance, pass@k and pass^k.

pass@k: capability

Here the question is whether at least one of k separate tries succeeds. As k grows, the score rises, because more shots on goal make a hit more likely. The metric became popular through OpenAI's HumanEval work on code generation, where a system produces several candidate programs and checks whether any passes the unit tests.

It fits human-in-the-loop setups and systems with automated validators: three draft itineraries for a travel manager to choose from, or five candidate SQL queries where an engine runs the first valid one.

pass^k: consistency

This is the probability that the agent succeeds every single time across k attempts, which is the per-trial rate p raised to the power k. As k grows, the score falls fast. Suppose the agent passes 75% of individual trials:

• one run: 75.0%,
• three runs in a row: about 42.2%,
• five in a row: about 23.7%,
• ten in a row: about 5.6%.

This is the bar for autonomous, mission-critical production use. When the booking agent is wired to payment cards and provider APIs, users don't care that it could succeed once in five tries. They need it to work reliably each time.

Why the choice matters

At k equal to 1, the two metrics are identical and equal the base success rate. By k of 10 they tell opposite stories: pass@k heads toward 100%, since enough tries will eventually stumble on success, while pass^k heads toward zero, since ten flawless runs without a single hallucination or tool failure is hard. Picking the metric that matches your product's promise separates vanity benchmarks from real reliability.

Practical takeaways

1. Don't test only the final answer. An agent that reaches the right answer through broken reasoning is a production risk.
2. Mix code and LLM judges. Use deterministic code for prices, IDs and presence checks, and LLM judges for reasoning quality, plan adherence and intent matching.
3. Gate what must never be wrong. Elegant reasoning shouldn't be able to dilute an incorrect price, bad currency or broken date in an average.
4. Treat evals like CI. Run the suite whenever a prompt or model changes, so regressions surface before users find them.

Reliable agents don't come from hoping the model behaves today. They come from a harness that checks, every day, that it does.