A Dependency-Free LLM Eval With a Judge You Can Actually Trust

Unit tests work because deterministic code gives the same answer every time: assert that a function returns four and it always does. Language model output changes wording on every call, so there is no expected string to compare against, and teams often respond by adopting an evaluation platform that uses another model to grade the results. The grading model is where most evaluations quietly stop measuring anything, because nobody has confirmed that it agrees with a human. This guide builds a complete evaluation in plain Python in a few hours, and spends its most important section on calibrating the judge so the resulting number can be defended.

The Three Parts of Any Eval

Strip away the tooling and an evaluation has three components, none of which needs a library:

• A dataset of inputs, preferably drawn from real usage rather than invented.
• A way to generate an output for each input, which is simply your system under test.
• A scoring function that converts each output into a verdict you can count.

Dashboards, tracing and hosted datasets are conveniences built around these parts. Some of them are genuinely helpful, but the core fits in roughly eighty lines of Python, and code that small will keep working long after any particular platform has changed or disappeared.

Step 1: Collect a Hundred Real Inputs

Take the inputs from your production logs. Avoid synthetic examples, avoid the cases you already know succeed, and avoid a cleaned-up sample. A hundred cases is large enough to reveal real problems and small enough to label by hand later, which turns out to be essential. Include difficult and awkward inputs on purpose: a dataset of easy cases produces a score that stays flat even when your system gets worse.

Loading them is trivial:

import json
with open("requests.jsonl") as f:
    cases = [json.loads(line) for line in f][:100]

Notice that slicing the first hundred lines takes whatever happens to be at the top of the file, which may all come from one day or one customer. Shuffling before you slice, or sampling across time periods, gives a more representative set.

If you have no logs yet, write the hundred cases yourself, but expect the results to look better than reality until real traffic replaces them.

Step 2: Separate Code Checks From Judgement Calls

This is the step teams most often skip, and it shapes everything after it. Many properties can be verified deterministically, and those should never be sent to a model:

• The output is valid JSON.
• A category belongs to a known set.
• A number falls within an allowed range.
• Required fields are present.
• The response stays under a length limit.

Each of these is an assertion. The following function (Python, despite how it may be labeled) runs a few of them and returns a dictionary of named results:

def code_checks(output):
    checks = {}
    try:
        parsed = json.loads(output)
        checks["valid_json"] = True
        checks["has_fields"] = all(k in parsed for k in ("answer", "confidence"))
    except json.JSONDecodeError:
        checks["valid_json"] = False
        checks["has_fields"] = False
    checks["under_limit"] = len(output) < 2000
    return checks

Returning named checks rather than a single boolean lets you see which property failed when a case goes wrong. Whatever remains after deterministic checks is the part that requires judgement: did the response answer the actual question, did it introduce facts not present in the source, did it decline when it should have? Those properties need a judge, and a judge needs validation.

Step 3: Write a Judge That Grades One Thing

Give each judge call exactly one criterion. A rubric that asks for five qualities at once produces a blended verdict, and when it fails you cannot tell which quality was missing. The judge below checks a single property, unsupported claims, and demands a fixed response format:

JUDGE = """You are grading one property of a response.

PROPERTY: Does the response contain any claim that is not supported by the
source text provided?

SOURCE:
{source}

RESPONSE:
{response}

Answer with exactly one word, PASS or FAIL, then a new line, then one
sentence explaining your verdict. Do not explain anything else."""

def judge(source, response, call_model):
    out = call_model(JUDGE.format(source=source, response=response))
    verdict = out.strip().split("\n")[0].strip().upper()
    return verdict == "PASS", out

The format constraint matters as much as the criterion. Requiring a single word, PASS or FAIL, on the first line makes the result easy to parse. Free-form verdicts vary in structure from call to call, and if a parsing failure is ever treated as a pass, every score you report afterwards is inflated without anyone noticing. This implementation errs on the safe side: anything other than an exact PASS, including PASS. or a verdict wrapped in formatting, counts as a failure. It is worth logging how often that happens, since a spike in unparseable answers is itself a signal.

Step 4: Run Everything and Keep the Raw Output

Store the full output and the judge's explanation for every case, not only the verdict. When a score changes, you will want to read what actually changed, and a saved boolean cannot tell you that.

results = []
for case in cases:
    output = call_model(case["prompt"])
    passed, reasoning = judge(case["source"], output, call_model)
    results.append({
        "id": case["id"],
        "output": output,
        "code_checks": code_checks(output),
        "judge_pass": passed,
        "judge_reasoning": reasoning,
    })
rate = sum(r["judge_pass"] for r in results) / len(results)
print(f"{rate:.1%} pass on {len(results)} cases")

The loop records the deterministic checks alongside the judge result, although the headline rate only counts the judge. In practice you would report both, since a response that fails JSON validation is a failure regardless of what the judge thinks. Also note that the same call_model function generates and grades here; using a different or stronger model for the judge is a common choice, but whichever you pick, the next step is what makes it trustworthy.

At this point you have a percentage. It does not mean anything yet.

Step 5: Calibrate the Judge Against Your Own Labels

This is the step most guides omit, and it takes about thirty minutes. Pick fifty of the hundred cases and label each one yourself as PASS or FAIL, without looking at the judge's verdict first. Then compare the two sets of labels:

agree = sum(1 for r, human in zip(results[:50], human_labels)
            if r["judge_pass"] == human)
print(f"judge agrees with me on {agree}/50 = {agree/50:.0%}")
both_fail = sum(1 for r, h in zip(results[:50], human_labels)
                if not r["judge_pass"] and not h)
judge_fails = sum(1 for r in results[:50] if not r["judge_pass"])
human_fails = sum(1 for h in human_labels if not h)
print(f"judge caught {both_fail}/{human_fails} of the failures I found")
print(f"judge flagged {judge_fails - both_fail} things I considered fine")

The script reports three figures: overall agreement, how many of your failures the judge also caught, and how many cases the judge failed that you considered acceptable. The code assumes human_labels is a list of booleans in the same order as the first fifty results, and it will divide by zero if you found no failures at all, which is itself a hint that your dataset is too easy.

Why raw agreement misleads

Overall agreement is the weaker figure, even though it is the one most often quoted. If ninety percent of your cases pass, a judge that answers PASS to everything reaches ninety percent agreement while detecting nothing at all.

The number that matters: failures caught

What deserves your attention is the share of your own FAIL labels that the judge also flagged. In classification terms, this is the judge's recall on the failure class. A judge that catches two of your eleven failures is not grading anything; it is approving everything with a convincing explanation attached. The third figure, false alarms, matters too, because a judge that flags good responses will push you to "fix" things that were never broken.

When You and the Judge Disagree

Read every case where your label differs from the judge's. There are only three explanations, and each has its own remedy.

The rubric is ambiguous

The judge is wrong because the property statement leaves room for interpretation. Narrow it. A good test is whether you can express the property in one sentence that a new colleague would apply exactly as you do.

Your own labels are wrong

This happens more often than people expect. Human labeling drifts across fifty cases, especially on borderline examples toward the end of a session. Relabel the disputed cases with fresh eyes before blaming the judge.

The property is inherently subjective

Some qualities cannot be judged consistently by anyone. If that is the situation, no judge will be reliable. Either split the property into smaller, checkable sub-properties or accept that it cannot be scored.

Iterate: adjust the rubric, relabel where needed, and rerun until the judge catches enough of your failures that you would defend the figure in a meeting. Then freeze the rubric and store its version with every future result. Runs graded with different rubrics cannot be compared. For how to keep a judge healthy once it runs continuously in production, see managing LLM-as-a-judge as a living production system.

What the Calibrated Eval Gives You

The result is a number you can place next to a change and believe. Before modifying a prompt, swapping a model or changing a retrieval step, you take a measurement; afterwards you take another. The difference carries information only because you first confirmed that the instrument agrees with a person.

Because the code depends on nothing except the function that calls your model, it keeps working when you switch providers or when the tooling ecosystem reshuffles, which is more than can be said for most of what gets recommended for this job.

Limits of a Hundred-Case Eval

Three constraints are worth understanding before you build on this approach.

Small regressions are invisible

A drop from 91 to 89 percent is two cases out of a hundred, which is indistinguishable from noise at this sample size. To detect small effects you need either much more data or a paired comparison: run both versions on the same inputs and count only the cases whose verdict flipped. Paired comparisons remove most of the variation that comes from case difficulty.

Hand labeling does not scale

Manual labels stop being practical beyond a few hundred rows, and quality declines as a session goes on. Your judgement on case forty-eight is not the same as on case two, which introduces real error into the calibration itself. Labeling in several short sessions rather than one long one reduces that drift.

Calibration expires

A judge validated on today's outputs may not hold for tomorrow's. When you change the model behind your feature, its failure modes change too, so the judge must be revalidated rather than reused. It is tedious work, and it is exactly what separates a measurement from a ritual.

Two Additions Once It Runs

• Version everything. Store the model version and the rubric version with each result. A score missing either cannot be compared with any other score, for the same reason a benchmark figure without its harness says very little.
• Keep the disagreement set. The cases where you and the judge disagreed are the most informative rows in the whole exercise. Rerunning them after every rubric change is the quickest way to learn whether the change helped.

The snippets here favor clarity over production hardening. Treat them as a template to read and adapt, adding retries, error handling and persistence as your setup requires.

Key Takeaways

• An eval is a dataset, a generator and a scorer; everything else is optional tooling.
• Use real, difficult inputs, and verify anything deterministic with code instead of a model.
• Give each judge one criterion and a strict, parseable output format.
• Report how many human-identified failures the judge catches, not just overall agreement.
• Freeze and version the rubric, revalidate after model changes, and use paired comparisons when effects are small.