How to Check an AI Agent Isn't Wrong: Evals and Human Review
Build a test set from your own documents, measure field accuracy, false alarms and abstentions, run evals on every change and route uncertain cases to people.
This article is published in English.
An AI agent that reads documents or answers questions will be wrong some of the time. The useful question is not whether it makes mistakes but how often, on which cases, and whether anyone notices before the mistake reaches a customer or a ledger. Answering that takes two things: an evaluation set you trust and a review step built into the workflow.
Start with a test set from your own data
Generic benchmarks tell you how a model does on someone else's tasks. They say little about your supplier certificates or your contract templates. The first deliverable of any serious AI automation should be a test set built from real cases:
- 50 to 200 real examples to start with, more for processes with many document types;
- the correct answer for each, written down by the person who does the work today;
- deliberately included hard cases: bad scans, handwritten notes, missing fields, documents in another language, two documents stapled into one PDF;
- a note on which errors are expensive and which are merely annoying.
Personal data in the test set should be minimised or masked, and the set itself stored with the same access rules as the source documents.
Measure the right thing
A single accuracy number hides the failures you care about. Break the result down:
- Field-level correctness for extraction: was the invoice number, the date, the amount, the VAT rate read correctly?
- Rule checks for review tasks: did the agent flag the expired certificate, the missing signature, the payment term that differs from the contract?
- False alarms: how many correct documents were flagged. Too many and people stop reading the flags.
- Abstention: how often the agent said "not sure" instead of guessing. A well-built agent abstains on cases it cannot handle.
Rules that can be checked with code should be checked with code. If the question is "is the certificate date in the future", compare dates in a program and use the model only to find the date in the document. That alone removes a whole class of errors.
Run the evaluation on every change
Prompts, models, OCR settings and parsing code all change over a project's life. Each change can fix five cases and break three others. The evaluation set should run automatically before any change goes to production, with the result compared against the previous version. If a number drops, the change waits.
The same applies when you switch models. A newer or cheaper model is not automatically better on your documents. Run the set, compare, then decide.
Put a person where the risk is
Human review is not a sign that automation failed. It is how you get the benefit of automation without betting the business on a model. A practical setup:
- results above an agreed confidence threshold and with all rule checks passed go through automatically, or to a quick spot check;
- uncertain results, failed checks and high-value cases go to a review queue with the source document shown next to the extracted data;
- every approval and correction is logged with who made it and when;
- corrections feed back into the test set, so the next evaluation includes the case that went wrong.
The threshold is a business decision. Start strict, measure how much the reviewers actually change, and relax it only when the numbers support it.
Watch it in production
Real input drifts. A supplier changes its template, a scanner gets replaced, a new document type appears. Monitoring for an AI agent should show at least the share of documents sent to review, the share of reviewer corrections, processing errors and timeouts, and the age of the oldest item in the queue. A rising correction rate is usually the first sign that something upstream changed.
What good looks like
You can answer, with numbers from your own data, how often the agent is right on each field and each rule, what happens to the cases it gets wrong, and how you will notice when that changes. If a vendor cannot show you that, you are looking at a demo, not a production system.
We build this layer as part of our demo to production service and in AI agents for document review. For the organisational side of who builds and owns it, see what a forward-deployed engineer is.