Training Your Own LLM Judge: From PandaLM and JudgeLM to Prometheus

Once a language model can write essays, answer support tickets and summarize contracts, the question "is this output any good?" stops having a cheap answer. Human review is the gold standard but it is slow and costly, classic overlap metrics barely track quality anymore, and prompting a proprietary model to grade outputs works well only when your criteria resemble what that model already knows. This guide walks through the research on finetuned LLM judges, open models trained specifically to score and critique other models, and ends with a concrete recipe you can follow to build an evaluator for your own domain.

Why evaluating modern LLMs is so difficult

Two properties of current foundation models make evaluation hard. The first is scope: a single model is expected to code, translate, reason, follow formatting rules, stay safe and sound friendly, and every one of those behaviors deserves its own measurement. The second is open-endedness. For almost any prompt there is a large space of acceptable answers that differ in wording, structure, length and emphasis, so you cannot simply check whether the output equals an expected string. Measuring even one capability well is a research problem, and a production model has dozens of them.

The most trustworthy signal still comes from people reading outputs and rating them. The trouble is that human judgments are noisy, expensive and slow to collect, which clashes with the pace of model development. A team that retrains or re-prompts a model several times a week cannot wait for a fresh annotation campaign after each change. Some amount of human evaluation remains non-negotiable, but it has to be complemented by something that scales.

That need gave rise to LLM-as-a-Judge, introduced in the paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: you prompt a strong proprietary model to assess the output of another model. The approach is now everywhere in research, but it has a blind spot. Domain-specific products often care about fine-grained criteria, such as whether a medical summary preserves dosage information or whether a legal answer cites the right clause, and a general-purpose model may not judge those reliably. In that situation the natural next step is to train a specialized judge, and understanding how to do that well is the goal of the rest of this article.

The evaluation toolbox: humans, metrics and model-based judges

In practice, teams combine human evaluation with automatic metrics. Humans provide the definitive read on quality; automatic metrics provide speed. Because automatic scores are only an approximation of what people think, you keep checking them against periodic human reviews, but in between those reviews you can compare far more model variants than humans could ever read.

Automatic evaluation of modern LLMs mostly falls into two families. The first is benchmark-style evaluation based on likelihood or perplexity, which covers most traditional NLP benchmarks such as MMLU and BIG-bench. These behave roughly like multiple-choice exams that probe what the model knows. They are useful, but they say little about how well a model writes a long, stylistically appropriate answer to an open question.

For long, open-ended outputs there are really only two routes: have humans rate the responses, or have a capable model rate them. Given the costs of the first route, model-based evaluation has become the default. It needs no reference answer, it is easy to set up, and it copes with a broad variety of tasks. The sections below look at each piece of the toolbox in turn, starting with the human side, because everything else is measured against it.

Human evaluation is the ground truth, not a perfect one

People can notice what actually matters in a response: whether it is appropriately brief, creative, well pitched in tone, or culturally sensitive. That is why human review remains the reference point. Annotators can come from many places. You can crowdsource votes as Chatbot Arena does, hire a dedicated team, or, as the Alpaca project did, have the model developers rate outputs themselves. Treating humans as ground truth does not mean their labels are clean. In practice they are frequently inconsistent, tiring to produce and subject to bias.

Agreement and calibration

Even tasks that look straightforward often produce surprisingly low agreement between annotators. You can quantify it with a simple match rate or with chance-corrected statistics such as Cohen's kappa (two raters), Fleiss' kappa (many raters) or Krippendorff's alpha (flexible with missing labels and different scale types). Chance correction matters: if 80 percent of responses are "good", two raters who label everything "good" will agree most of the time without learning anything.

When agreement is low, the fix is calibration, meaning you teach annotators to apply the task the same way. A typical loop is a review session where raters look at the items they disagreed on, argue the cases, and reach a shared interpretation. The outcomes of those discussions then get written back into the task guidelines so the next batch starts from a clearer baseline.

Writing guidelines that actually get used

Guidelines are the written contract for an annotation task. At minimum they must state what is being evaluated, meaning the criteria, and how each criterion should be applied. Good guidelines mix prose instructions with worked examples of correct and incorrect labels, and annotators consult them throughout the job rather than reading them once. They make labels more consistent and dramatically shorten onboarding for new raters. A useful test: if an annotator needs knowledge that is not in the document to label an item correctly, the document is incomplete.

Guidelines should also be treated as living documents. The InstructGPT team, for example, described revising their labeler instructions repeatedly over the project as their feedback, metadata fields and understanding of the target evolved, and fixing passages that turned out to be confusing or contradictory. That experience is typical. Tasks that seem obvious usually hide a lot of subjectivity, and it only surfaces when a group of people tries to agree on hundreds of real examples.

Monitoring agreement over time

The heavy investment happens up front: drafting criteria, calibrating raters and iterating until agreement is acceptable. Only then do human scores become a trustworthy measure of model quality. The work does not end there, though. Agreement can decay for mundane reasons: new raters join, experienced ones gradually drift from the written rules, or the guidelines themselves change. Tracking inter-annotator agreement continuously, for instance by routing a fraction of items to several raters, is the only way to catch this before it quietly corrupts your evaluation data.

Reference-based metrics and why they stopped working

Earlier generations of language models were commonly scored with metrics like ROUGE for summarization and BLEU for translation. Both are reference-based: they compare the model output to a gold answer, largely by counting overlapping n-grams, and reward outputs that share more words and phrases with that reference.

As models improved, these metrics lost their grip. Studies of LLM outputs have found that such metrics correlate weakly with human preferences, to the point where none of the classic metrics examined tracked human judgment well. They survive mainly because they are nearly free to compute. If you already have reference answers, they still work as a quick sanity check, for example to catch a regression where a summarizer suddenly starts producing empty or off-topic text.

The core reason they fail is structural. A reference-based metric is essentially sophisticated fuzzy matching against a single target. Modern models are open-ended: one prompt admits many equally good answers that may share few n-grams with the reference. A score that rewards similarity to one particular answer cannot represent the full range of valid responses, so it penalizes perfectly good paraphrases and can reward fluent text that happens to reuse the reference's vocabulary while being wrong.

Prompted LLM judges

The weaknesses of overlap metrics pushed the field toward more flexible, model-based evaluation. The most popular form is LLM-as-a-Judge: a strong model is prompted to act as a grader for another model's outputs. The MT-Bench and Chatbot Arena work showed that this correlates well with human preferences, takes little more than a well-written prompt to implement, handles open-ended outputs naturally and can be pointed at many different criteria. That work framed it as a scalable, explainable approximation of human preference, which is otherwise expensive to gather.

In the rest of this article, "LLM-as-a-Judge" refers specifically to prompting an off-the-shelf proprietary model, such as GPT-4o or Gemini 1.5, and "finetuned judge" refers to a model you train yourself for evaluation. For broader background on prompted judges, useful further reading includes a practical guide to building judges that drive business decisions, an analysis of how effective LLM evaluators are and a survey of LLM-as-a-Judge research. If you are already running a prompted judge in production, the operational side is covered in our guide to managing LLM-as-a-judge as a living production system.

Pairwise and pointwise scoring

There are two main ways to ask a judge for a verdict:

• Pairwise: the judge sees a prompt plus two candidate responses and picks the better one (often with a tie option).
• Pointwise: the judge sees a prompt and a single response and assigns a grade, typically on a Likert scale from 1 to 5.

Pointwise scoring also goes by names like direct assessment or single-answer grading. Either setup can be made reference-guided by adding a known-good answer to the judge's prompt, so the judge compares against it instead of relying only on its own knowledge. A widely used public example of a prompted judge is AlpacaEval, a leaderboard that uses LLM-as-a-Judge to predict human preference between model outputs and publishes all of its evaluation prompts, including earlier versions.

Choosing a scoring setup

No setup wins everywhere; the right choice depends on the application. Pairwise judgments tend to be more stable, because comparing two things is easier than placing one thing on an absolute scale. The price is position bias, the tendency to favor whichever response appears first (or second), and weaker scalability: pairwise results only tell you how outputs rank relative to each other, and comparing many systems requires many pairs. Pointwise scoring is more versatile because each output gets its own score that can be stored, averaged and compared later. It is also harder for the judge, since it has to anchor the score purely on its internal sense of quality. Supplying a reference answer is a common way to make pointwise grading more reliable.

The nature of the criterion matters too. Objective properties such as factuality or faithfulness to a source document tend to be binary: the response is faithful or it is not. Asking which of two responses is "more faithful" is poorly defined. Subjective criteria, like helpfulness, tone or overall preference, are where relative judgments shine, because people and models find comparisons easier and more repeatable than absolute ratings. A reasonable default is pointwise, often binary, scoring for objective checks and pairwise comparison for subjective quality.

The biases every judge inherits

Whatever kind of judge you use, prompted or finetuned, a handful of systematic biases show up again and again, and most of the finetuning techniques later in this article exist to counter them. It helps to know them by name before reading further:

• Position bias: in pairwise mode, the judge favors the first or the second slot independent of content.
• Verbosity or length bias: longer, more elaborate answers get higher marks even when the extra text adds nothing.
• Self-enhancement bias: a judge prefers outputs written by itself or by models from the same family.
• Knowledge bias: the judge cannot evaluate correctness on topics it does not know well, and may reward confident but wrong answers.
• Format bias: the judge works only when the prompt looks exactly like the one it was built or tuned around.

None of these can be fully eliminated, but each can be measured. Swapping the order of candidates, padding a response with harmless filler, or grading the same content under two prompt templates are cheap experiments that reveal how much a given judge is swayed by factors that should not matter. Running them once when you adopt a judge, and again after any change to its prompt or model, is one of the highest-value habits in evaluation work.

Where prompted judges fall short

LLM-as-a-Judge is effective and widely adopted, but relying on a proprietary model comes with real drawbacks:

• The model is a black box, and sending evaluation data to a third party can raise security and privacy concerns.
• You do not control versioning. The provider can update the model underneath you, and your evaluation numbers can shift or break without any change on your side.
• Every judgment is an API call, so evaluating large volumes of outputs gets expensive.
• These models do best on evaluation tasks that resemble their training, such as predicting general human preference, and less well on niche criteria.
• General-purpose assistants are not specialized for critique, and they tend to hedge, avoiding strong scores or firm opinions.

Almost all of these stem from depending on a model you cannot inspect or change. That leads to the obvious question of whether you can train your own judge. The research summarized below says yes: a finetuned evaluator can deliver domain-specific, granular and more critical feedback, provided you invest in the data behind it.

Meta-evaluation: measuring the judge itself

Before training a judge, decide how you will know whether it is any good. The answer is a set of human-labeled evaluation data that you trust deeply, because it becomes the yardstick for every judge you build. Meta-evaluation, evaluating the evaluator, then means running the judge on that data and comparing its verdicts to the human ones.

If the judge produces a binary output, such as a pairwise choice or a pass/fail grade, you can use ordinary classification metrics like accuracy, precision, recall and the ROC curve. Because these are so easy to interpret, many practitioners argue that binary scoring is the most practical design for LLM judges in general.

For non-binary scores, such as a 1 to 5 Likert grade, you measure how well judge scores correlate with human scores. As a reference point, the G-Eval work reported that a GPT-4-based evaluator reached a Spearman correlation of 0.514 with humans on a summarization benchmark, which at the time beat earlier methods by a wide margin. Spearman's rank correlation is probably the most common choice, but you will also see Kendall's tau, Pearson correlation and Cohen's kappa used as agreement measures. These are harder to reason about than classification metrics, so learn the quirks of whichever one you report. Pearson measures linear agreement and is sensitive to outliers; Spearman and Kendall only care about ordering, so a judge that is consistently one point too generous can still score perfectly on them.

Practical meta-evaluation habits

A few habits make meta-evaluation far more informative. Report agreement per criterion rather than a single blended number, because a judge can be excellent at spotting factual errors and poor at rating tone, and an average hides that. Look at the confusion between adjacent scores separately from gross disagreements: a judge that says 4 where humans say 5 is a very different problem from one that says 1. Compare judge-human agreement with human-human agreement on the same items; if two trained annotators only agree 70 percent of the time, a judge that agrees with them 68 percent of the time is essentially at the ceiling, and chasing more is pointless. Finally, keep the meta-evaluation set strictly separate from anything used to write prompts or train the judge, or the numbers will flatter it.

The first generation of finetuned judges

Early on, nearly everyone used proprietary models for judge-style evaluation, and there was a good reason. Unless you hand the judge a reference answer, it has to be capable of solving the task itself in order to tell a good answer from a bad one. Open models trailed closed ones for a long time, so closed models were simply the better graders. That changed once stronger open base models such as LLaMA and LLaMA-2 appeared, and researchers began finetuning them into dedicated evaluators. The papers below trace that first wave.

A finetuned Vicuna judge in the MT-Bench paper

The very paper that introduced LLM-as-a-Judge also ran one of the first experiments with a finetuned judge. Most of that work is about prompting proprietary models with generic templates to predict human preferences, but a short appendix explores training an open model for the same job, and its conclusion was that a finetuned Vicuna-13B looked promising as an inexpensive open alternative to closed judges.

The motivations it lists are still the main reasons to consider a custom judge today:

• Evaluating through a paid API gets expensive at scale.
• A proprietary model offers neither control nor transparency.
• Open models keep getting more capable, so the gap is shrinking.

If a finetuned open judge could match proprietary quality, all three problems would go away.

The experiment used Vicuna-13B, a LLaMA derivative. Out of the box it made a poor judge: it had a high error rate, often failed to follow the evaluation template or instructions, and showed strong position bias. Finetuning changed that picture substantially. The model was trained on 20K single-turn human votes from Chatbot Arena, each comparing outputs from a wide range of models, and the task was simplified to three-way classification: the first response wins, loses or ties. After training, position bias dropped sharply and agreement with human votes improved markedly, although the finetuned model still did not reach GPT-4's level.

Framing the task as three-way classification is a sensible simplification. It removes the need to calibrate numeric scores, it maps directly onto how Arena votes are collected, and it makes accuracy trivial to compute against held-out human votes. The downside is that the judge produces no explanation, so when it disagrees with people you learn little about why.

The whole exploration occupies roughly a page, but the results were encouraging enough to kick off a line of work on open evaluators.

PandaLM: a judge for choosing between model checkpoints

A frequent use of a judge is model selection: given several candidate models, which one is best? This gets hard when the candidates are all strong and differ only subtly. Humans can make that call, but slowly, so a judge that reliably spots small quality differences is valuable.

PandaLM was designed for exactly this. Its goal goes beyond checking whether an answer is correct; it also weighs subjective qualities like relative conciseness, clarity, comprehensiveness, formality and adherence to the instructions. The headline application is hyperparameter tuning. You train several variants of a model with different settings, let PandaLM compare their outputs, and keep the winning configuration. In the paper's experiments, every model tuned this way performed better than with the default settings, which shows that an evaluator can feed directly back into training decisions rather than only producing reports.

The training set contains more than 300K examples. Each input is a tuple of (instruction, input, response1, response2) and each target is a tuple of (evaluation result, evaluation rationale). Every example has a reference answer, and the result either names the better response or declares a tie. The team finetuned LLaMA models of several sizes on this data using plain next-token prediction with cross-entropy loss, in a sequence-to-sequence style, with no separate classification head. In other words, the verdict and the explanation are just text the model learns to generate.

The instructions and inputs come from the Alpaca training data. Response pairs were produced by several roughly 7B-parameter models, namely LLaMA-7B, BLOOM-7B, Cerebras-GPT, OPT-7B and Pythia. The verdicts and rationales for those pairs were generated synthetically with GPT-4, following a Self-Instruct-style process, and then cleaned with heuristics such as hand-written rules and filters that discard invalid or unstable evaluations. Separately, human annotators produced a smaller "golden" test set used purely for meta-evaluation.

When PandaLM is compared with other evaluators, including GPT-3.5 and GPT-4, their preferences follow similar trends: they rank the evaluated models in a consistent order. Head to head with proprietary judges, the 7B PandaLM is competitive and the 70B version exceeds GPT-4 on the paper's evaluation, which suggests that judging ability, like most capabilities, improves with scale.

PandaLM also holds up in specialized domains. The paper applies it to responses on LSAT questions and to biomedical question-answering datasets, with the larger model doing especially well. Because it is trained specifically on evaluation data, it transfers to domain-specific judging tasks that are close to its training distribution, not only to generic chat evaluation.

JudgeLM: what actually makes a finetuned judge good

Most early finetuned judges said little about why they worked. JudgeLM set out to study the factors systematically, including:

• how much training data is used, and how good and diverse it is;
• the size and quality of the base model;
• how bias affects the judge's verdicts;
• whether the judge generalizes to other formats, such as multi-turn chat, single-answer grading, pairwise ranking and multimodal inputs.

A finetuned judge can fall behind a proprietary one for many reasons: a weak base model, noisy data, systematic bias and so on. The value of JudgeLM is in showing that these issues can be tackled directly.

The most important ingredient is a large dataset that is high quality, fine-grained and diverse, and the JudgeLM team explicitly positioned their dataset as the most diverse and high-quality one available at the time. It starts from 105K seed instructions drawn from public sources such as ShareGPT and a version of the Alpaca data with GPT-4 answers.

For each seed, answers are generated by several models, including Alpaca, Vicuna and LLaMA, and combined with the original answer from the source dataset, so every instruction ends up with multiple candidate answers plus a reference. Pairs of answers are then sampled at random and GPT-4 scores each pair and explains its reasoning. GPT-4 effectively acts as the teacher: JudgeLM learns to reproduce its judgments.

The GPT-4 grading templates come in two variants, with and without a reference answer. They look much like standard public judge prompts, with one important difference: they spell out fairly detailed evaluation criteria. That pushes GPT-4 toward granular assessments, and finetuning on those assessments passes the same granularity on to JudgeLM.

JudgeLM was trained at 7B, 13B and 33B parameters, on datasets ranging from 3.5K to 100K examples, using instruction tuning with supervised finetuning (SFT). The input contains one instruction, two candidate answers and a reference answer. The model outputs a separate score for each answer followed by a rationale. Turning that into a pairwise verdict is trivial: whichever answer scores higher wins, and equal scores are a tie. A nice side effect of this design is that one model serves both as a grader and a comparator.

Reducing bias was a central goal. JudgeLM targets three kinds:

1. Position bias: preferring a response because of where it sits in the prompt rather than what it says.
2. Knowledge bias: the base model lacks pretraining knowledge that is needed to judge the response correctly.
3. Format bias: the judge performs well only when the prompt matches the exact format used during finetuning.

Each has a fairly direct remedy. Format bias shrinks when the judge sees several prompt formats during training. Knowledge bias shrinks when you give the judge a reference answer, so it does not have to know the facts itself. Position bias is handled with position switching: evaluate each pair in both orders, both when building training data and at inference time. If the verdict flips when the order flips, the pair is usually treated as a tie. Swapping positions in training teaches the model to attend to content rather than placement, and swapping at inference caps whatever bias remains. Used together, these techniques cut JudgeLM's bias substantially.

The cost of position switching at inference is worth keeping in mind: it doubles the number of judge calls for pairwise evaluation. For large offline evaluations that is usually acceptable; for latency-sensitive online use it may not be.

On its own evaluation set, JudgeLM beats PandaLM and GPT-3.5, both with and without reference answers. That comparison favors JudgeLM, since it was trained on data from nearly the same distribution, but it also does well on PandaLM's evaluation set. Note that much of JudgeLM's reported performance is agreement with GPT-4, the same model that generated its training labels, not agreement with humans. Overall, the models perform solidly, get better as they get bigger, and are more robust to position switching than the alternatives.

This points to a general limitation of distilled judges. When a student model is trained on a teacher's verdicts, the teacher's accuracy acts as a soft ceiling, and the teacher's blind spots and biases are copied along with its strengths. High agreement with GPT-4 tells you the distillation worked; it does not tell you the judge agrees with your users. Only a human-labeled test set answers that second question.

Auto-J: grounded in real user queries

Auto-J, from the paper "Generative Judge for Evaluating Alignment", is another finetuned evaluator, with a focus on scenario-specific grading in realistic settings. Its distinctive traits are:

• It supports both pairwise comparison and single-response scoring.
• It emphasizes well-structured, high-quality natural-language critiques.
• Its training queries come from real people, so the data reflects the variety of questions a model actually receives.

The outcome is a judge that is flexible, interpretable and practical.

Auto-J's data starts by defining scenarios, each with its own definition and evaluation criteria. The paper settles on 58 scenarios grouped into eight broader categories. For each scenario, real user queries are collected, responses are generated, and GPT-4 produces the judgments. GPT-4 is guided by hand-written criteria specific to each scenario, and the outputs are post-processed with heuristic filters and format normalization to reduce noise. Each scenario contributes up to 100 examples, for roughly 3,500 in total.

The queries come mostly from public sources built on human interaction, such as Chatbot Arena and WebGPT. Because they mirror what people ask in the wild, Auto-J specializes in realistic evaluation. Both pairwise and single-response labels are produced by GPT-4, and filters remove predictions that are inconsistent or badly formatted.

Auto-J is finetuned from LLaMA-2-13B-Chat on this mixed-format dataset. An interesting design decision is that the evaluation criteria are not included in the input. The reasoning was that the model should absorb the criteria implicitly from the data, which would make it more general. Later work, notably Prometheus, came to the opposite conclusion and found that feeding rubrics and other reference material into the judge is important. To counter position bias, Auto-J uses a simple augmentation: every pairwise example is duplicated with the two responses swapped, and the labels "Response 1" and "Response 2" in the judgment are swapped accordingly.

The Auto-J test set is built like its training set, except that every label comes from human annotators. Against a range of other models, Auto-J outperforms almost everything except GPT-4. That confirms that a specialized judge trained on domain-appropriate data can approach, and sometimes beat, proprietary judges in its target setting. But its absolute agreement with humans on this dataset is still low, a reminder that judge quality in general had plenty of room to improve.

Critics, verifiers and synthetic data generators

In parallel with these judges, a closely related line of research trained models to verify, critique or give feedback on another model's output. Like judges, these models grade responses and explain their reasoning. The difference is that critics and verifiers also act on the verdict: they revise the response, or trigger a new attempt, when the original falls short.

SelFee is a good example: a LLaMA-based model trained on ChatGPT generations, ChatGPT feedback on them, and revised generations. Given a response, SelFee writes feedback and then rewrites the response based on that feedback, and it can keep iterating until the output clears a quality threshold. That makes it a verifier, since it checks its own output, and a critic, since it writes feedback and revises.

Many other critique methods have been proposed, including Self-Correction, PEER and self-critiquing models. The same machinery can generate and filter synthetic training data for SFT or preference tuning, as in RLAIF and Constitutional AI.

Seen from a distance, all of these systems, prompted judges, finetuned judges, synthetic data pipelines, critics and verifiers, do essentially the same thing: produce a score and an explanation. What differs is how the score is used, whether to report model quality or to decide which samples enter a training set. Recent work on generative verifiers and critique-based reward models argues that at least some of these roles can be unified in a single model that handles all the scoring tasks.

The Prometheus family: finetuned judges that rival GPT-4

Finetuned judges only went mainstream with Prometheus, a model built for domain-specific, fine-grained evaluation. Its methodology is not radically new; the big change is a much stronger starting point in LLaMA-2 and a carefully designed dataset. Because it matched many proprietary judges on evaluation quality, it quickly caught on and spawned several follow-up variants.

Prometheus: rubrics as first-class input

Prompting a proprietary judge is easy: describe the criteria and let the model work. The problems are the ones already covered, uncontrolled versioning and cost, plus the fact that these models are mostly exercised on generic criteria like overall human preference and can struggle with custom ones. Prometheus addresses this with a finetuned evaluator that, once trained for a domain, can match or beat proprietary judges.

The key idea is that Prometheus takes a custom scoring rubric as part of its input, which lets it adapt to new domains and criteria without retraining. It is trained on the Feedback Collection, a synthetic dataset created with GPT-4 that spans many evaluation tasks. Every example contains the same components:

• Instruction: the prompt given to the model being evaluated.
• Response: that model's answer, which is what gets graded.
• Rubric: custom criteria describing how the response should be judged.
• Reference answer: an example response that would earn the top score.
• Rationale: an explanation of why the response gets its score.
• Score: a 1 to 5 Likert grade.

A rubric has two parts: a general statement of the criterion, and a description of what each score from 1 to 5 means. Taking this as input is what lets the model grade at a fine level of detail and carry over to scoring schemes it has not seen. The Prometheus work was among the first to show how much a finetuned judge benefits from rich reference material in its input, and in particular from reference answers.

Generation starts from 50 rubrics written by humans and proceeds in three stages:

1. GPT-4, shown examples of the hand-written rubrics, expands them to 1K rubrics.
2. For each rubric, GPT-4 writes 20 instructions, giving 20K instructions overall.
3. For each instruction, GPT-4 generates a response, then feedback on it, then a score, in that order.

Two balancing tricks make the data more useful. First, GPT-4 produces five training examples per instruction, one targeting each score from 1 to 5, so the score distribution is uniform and the judge does not learn to default to high marks. Second, those examples are kept at similar lengths so the judge does not pick up verbosity bias, the habit of rewarding longer answers regardless of content.

Prometheus is finetuned from LLaMA-2-13B-Chat on the Feedback Collection together with the MT-Bench and Vicuna Bench evaluation sets. With all the reference material in the prompt, the model learns through SFT to write its feedback first and only then produce the score. The ordering is deliberate and mirrors chain-of-thought finetuning: the written critique becomes context the model conditions on when choosing a number, rather than a justification bolted on afterwards. The Prometheus team also stresses inserting a special separator token, [RESULT], between the feedback and the score, which keeps generations from degenerating at inference time and has the practical benefit of making the score trivial to parse.

Prometheus was compared with GPT-3.5, GPT-4 and several open base models on multiple datasets, measuring both agreement with human scores and the usefulness of its feedback. Its scores reached a Pearson correlation of 0.897 with human raters, on par with GPT-4 and far ahead of GPT-3.5. So a model finetuned on high-quality evaluation data for a domain can reach, or exceed, the human agreement of much larger proprietary models.

The feedback holds up too. Human reviewers preferred Prometheus's feedback over GPT-4's in 58.67% of comparisons, and over GPT-3.5's in close to 80%. When asked why, reviewers pointed to Prometheus being more critical and more specific. The paper sums it up as GPT-4 tending to stay neutral and abstract while Prometheus clearly commits to an opinion about whether the response is good. That is the payoff of specialization: Prometheus is not a general assistant trained to be agreeable, it is a model whose only job is critique.

Prometheus 2: one model for pointwise and pairwise scoring

Prometheus 2 continues the line with another open, fine-grained evaluator that takes custom criteria. It targets two weaknesses of the original. Agreement with human scores still had room to grow, and, more importantly, the first Prometheus could only do direct assessment. You could derive a ranking by comparing two pointwise scores, but neither Prometheus nor most earlier open judges could natively compare two responses. Prometheus 2 combines both modes in a single model that tracks humans and GPT-4 closely in each.

Pairwise examples in Prometheus 2 differ from direct-assessment ones in three ways:

• The reference material contains no reference answer.
• It consists only of a description of the evaluation criterion.
• The feedback is written differently.

In pairwise mode the feedback has to compare the two responses, pointing out where they differ and using those differences to justify picking one. Most earlier pairwise datasets and models relied on generic criteria and often provided a bare verdict with no comparative explanation at all.

To train the pairwise skill, Prometheus 2 introduces the Preference Collection, derived from the Feedback Collection. Since the latter has five responses per instruction, one at each score, you can form every pairing of them: ten pairs per instruction. The label for each pair follows directly from comparing the two Likert scores.

Because explanatory, chain-of-thought-style feedback is central to the Prometheus approach, labels alone were not enough. GPT-4 was prompted to write new feedback for each pair, focusing on what the two responses have in common and where they diverge.

There are several ways to get one judge that handles both scoring modes:

• Prompting: use an off-the-shelf model with a different prompt per mode.
• Single-format training: train one model per mode and use whichever fits.
• Joint training: train one model on a mix of both kinds of data.

Only joint training yields one specialized model that covers both modes; prompting relies on generic models, and single-format training leaves you with two. A fourth option is to train the two single-format models and then combine them by model merging, which is what Prometheus 2 does. Two evaluators are trained separately, one on the Feedback Collection and one on the Preference Collection, and their weights are combined with simple linear merging, producing one model that can score either way.

The notable result is that the merged model beats both the jointly trained model and the single-format models. Several merging methods were tried, including task arithmetic, TIES and DARE. The fancier methods sometimes help; with Mixtral 8x7B as the base, for example, DARE worked best. But plain linear merging with a coefficient of 0.5, which is simply averaging the two models' weights, performs well in most configurations and is trivial to implement. One intuition for why this works is that the two training runs start from the same base model and learn related skills, so their weights stay in a compatible region where averaging combines abilities instead of destroying them.

Prometheus 2 was built on Mistral-7B and Mixtral 8x7B: separate copies are finetuned on each dataset and then merged with various algorithms. The resulting models outperform every other open evaluator on a set of direct-assessment and pairwise benchmarks. Their Pearson correlation with proprietary evaluators improves by 0.2 across all datasets, and on pairwise ranking benchmarks they close roughly half of the gap to GPT-4.

The gains look smaller in pairwise mode, but the comparison there includes reward models alongside open evaluators. Reward models, the scorers used in reinforcement learning from human feedback (RLHF), are trained only for pairwise preference, produce no written feedback and are very good at pairwise accuracy. Prometheus 2 also shows better cross-mode consistency than other evaluators: the response it prefers in a pairwise comparison usually also receives the higher score under direct assessment. That property matters in practice, because a judge whose two modes disagree is hard to trust in either.

Prometheus-Vision: judging vision-language models

Everything so far concerns text-only models. Vision-language models (VLMs), which accept images as well as text, have become common in both proprietary form, such as GPT-4V and Gemini, and open form, such as LLaVA and the vision variants of Llama 3.2. Apart from the extra input modality, they are trained and structured much like text LLMs.

A VLM produces text from an image plus an instruction, and evaluating it adds a dimension. You have to check two things:

1. Whether the response follows the instruction, as with any LLM.
2. Whether the response is grounded in what the image actually shows.

One workaround is to convert the image into text, for example with a captioning or image-text model, and pass that description to a text judge such as GPT-4 or Prometheus. Multi-stage pipelines like this compound errors, though: anything the caption misses, the judge cannot evaluate. Using a VLM as the judge is the more direct design.

Prometheus-Vision is an open VLM evaluator. Proprietary VLMs like GPT-4V had already been used as judges for other VLMs, but Prometheus-Vision was the first open VLM able to grade against fine-grained, user-defined criteria. On top of the transparency and control of an open model, its scores correlate well with both humans and proprietary judges, and it can pick up subtle distinctions, such as telling a genuine artwork from a parody of it.

When the work was published, no feedback, critique or preference dataset existed for training a fine-grained VLM evaluator, so the team built one. The Perception Collection follows the Feedback Collection's design, and each instance has five input parts:

• Image: the picture the user gives the VLM.
• Instruction: the user's text instruction.
• Response: the VLM's text answer to the image and instruction.
• Rubric: detailed scoring guidance, including a description of the criterion and of each score on the 1 to 5 scale.
• Reference answer: a response that would score 5, generated with GPT-4V.

The outputs are a written rationale and a 1 to 5 score.

Construction mirrors the text dataset. The process samples 5K images from the MS-COCO and MMMU datasets, writes 50 seed rubrics by hand, uses GPT-4 to expand them to 15K rubrics, generates 30K instructions with matching reference answers, and finally adds five responses per instruction, one per score, each with feedback. Compared with the Feedback Collection, there are many more rubrics (15K versus 1K) but far fewer instructions per rubric (two versus 20).

A text-only base model cannot be used here; training has to start from a pretrained VLM. Prometheus-Vision uses the 7B and 13B versions of LLaVA-1.5. The recipe is the same chain-of-thought finetuning as before: SFT teaches the model to write feedback, then give a score. In every training example the two parts are separated by the fixed phrase So the overall score is, which plays the same role as the [RESULT] token in the text model. With rubrics and reference answers in the input, the result is a model tuned for customized, fine-grained evaluation.

Prometheus-Vision correlates better with human scores than GPT-3.5-Turbo and than the text-only Prometheus, but GPT-4 and GPT-4V agree with humans more closely overall. Its weakest showing is on VisIT-Bench, which is full of text-heavy images. There, GPT-4 does well, even better than GPT-4V, because it receives the text extracted from the image as input, while Prometheus-Vision struggles to read it. On the other benchmarks Prometheus-Vision is roughly on par with GPT-4 and GPT-4V and in several cases correlates with humans more strongly than either.

Human reviewers rated its feedback as about as good as that of GPT-4 and GPT-4V. The team also probed common evaluator biases, such as length bias, the preference for longer responses, and self-enhancement bias, the preference for one's own outputs. They found no obvious signs of either, with the caveat that isolating a single bias experimentally is genuinely hard.

Other directions in finetuned evaluation

Prometheus is not the only line of work. A number of other projects have trained LLMs for evaluation, each contributing a distinct idea. The short tour below covers the ones most relevant if you are designing your own judge.

Self-rewarding language models

Self-rewarding models aim to automate alignment with the help of an internal evaluator. Instead of training a separate reward model on human preferences, the model being aligned is itself trained to judge its own outputs using LLM-as-a-Judge-style prompts, and those judgments become the reward. Applied to LLaMA-2, the method produced useful feedback and improved the underlying model. Conceptually it resembles Prometheus in that the model learns to act as a judge, but the judge's output is fed straight back as a training signal for the same model.

LLM-as-a-Meta-Judge

A limitation of self-rewarding models is that nothing improves the judging skill itself; the model gets better at responding, but its evaluations are only as good as they started. Meta-rewarding models add a third role. Besides acting as actor and judge, the model also acts as a meta-judge that evaluates its own judgments, which produces training data for improving the judge alongside the policy. Both capabilities then improve together over the training iterations.

Self-taught evaluators

Self-taught evaluators remove human preference labels from the loop entirely. Human preference data is expensive to collect and goes stale as models improve, so this approach generates synthetic evaluation data iteratively. Starting from a set of instructions, an LLM produces pairs of contrasting outputs, one deliberately worse than the other, and a judge is trained to evaluate them by writing explanations and verdicts. Repeating this with the improved judge yields better training data each round. Without any human-labeled data, the resulting evaluators were reported to beat both GPT-4 used as a judge and reward models trained on human preferences on scoring accuracy.

The obvious caveat is circularity. A judge that bootstraps its own training data can drift toward whatever it already believes is good, and without an external check that drift is invisible. Even if you adopt a self-teaching loop, a small human-labeled set remains the only way to confirm that each round is actually improving rather than just becoming more confident.

FLAMe: foundational autoraters

Foundational Large Autorater Models (FLAMe) go the opposite way on data: they are trained on a very large pool of human judgments, more than 5M of them across over 100 quality-assessment tasks. Compared with other evaluators, FLAMe models generalize better to unseen evaluation tasks, show less bias, and serve as strong starting points for further finetuning into specialized judges. They were reported to consistently outperform prompted judges built on GPT-4 and Claude variants, as well as judges finetuned on synthetic GPT-4 data.

The generalization trade-off

An empirical study of finetuned judges, pointedly titled as finding that "fine-tuned judge models are task-specific classifiers", examined the trade-off with proprietary judges in depth. Its conclusion: finetuned judges are excellent in-domain, sometimes beating GPT-4 on test sets drawn from their training distribution, but they fall behind GPT-4 on generalization, fairness, aspect-specific evaluation and scalability. That is not surprising. A judge trained on one distribution learns that distribution; ask it about something else and it behaves like a narrow classifier. The practical implication is to keep a finetuned judge on the task it was trained for, and to re-validate it against human labels whenever the inputs it sees start to drift.

Direct judgement preference optimization

Direct judgement preference optimization applies preference optimization to evaluators rather than SFT alone. Preference pairs are gathered for three use cases: single rating, pairwise comparison and binary classification. The first two correspond to the usual pointwise and pairwise setups; the third frames evaluation as a yes-or-no question, such as whether a given property holds in the response. An auxiliary task is added as well: given the instruction and an evaluation result, the model has to infer which response was being evaluated. For every use case, a good and a bad judgment form a preference pair, and the evaluator is trained on them with direct preference optimization (DPO). The appeal is that the judge learns not just to imitate good judgments but to prefer them over plausible bad ones.

What drives a judge's scores?

A fair question about any LLM judge is whether it really applies the criteria you give it, or simply rewards text that looks like its training data. Analyses of LLM judges across several scoring setups found that detailed scoring instructions in the prompt improve their judgments only slightly, and that perplexity-based scoring, which rates a response by how likely the model finds it, sometimes aligns with human judgments better than prompting does, especially for textual quality.

This seems to cut against the Prometheus finding that reference material matters a great deal. The two results are easier to reconcile once you notice what was being measured. The perplexity-friendly analyses focus mainly on general response quality from a human-preference angle. Most LLMs are trained heavily on preference data, so it is natural that they assign high likelihood to answers people like. Prometheus, by contrast, targets a wide variety of custom criteria that go beyond generic preference, and for those, explicit rubrics carry information the model cannot infer from fluency alone. If your criteria are close to "which answer do people like more", the rubric may add little; if they are specific to your domain, it is essential.

A step-by-step recipe for building your own judge

With the research landscape in view, it helps to condense it into a process. The Prometheus work already sketches a framework, and the version below adapts it slightly to incorporate what the other papers teach.

Step 1: pin down the evaluation criteria

Decide precisely what you are evaluating before anything else:

• List the specific criteria that matter for your application.
• Write a detailed description of each one, including what each score level means.

The Feedback Collection is a good model of what such a rubric looks like, with a criterion description, per-score descriptions and reference answers. Expect to revise the criteria repeatedly. As you learn where humans and models disagree, you will refine and extend the rubric, and each revision makes both human and model-based evaluation more consistent.

Step 2: collect high-quality human labels

Before training or even prompting any judge, you need trustworthy human evaluation data. It is what lets you tell whether a judge is accurate, and it should reflect the inputs the judge will face in production. If the data is only for meta-evaluation, for example to validate a prompted LLM-as-a-Judge, a modest set of roughly 100 to 1K examples is usually enough. Finetuning a Prometheus-style judge takes considerably more, on the order of 1K to 100K examples.

Human data collection is hard, and it is exactly where teams most often under-invest. If people cannot agree with one another on the task, there is no way to tell whether a model is doing it well, and training a judge on inconsistent labels wastes the entire effort. Everything in the human-evaluation section above applies here: calibrate raters, maintain guidelines and keep measuring agreement.

Step 3: augment with synthetic data

The Prometheus models demonstrate that synthetic data can produce strong judges: almost all of their training data, including instructions, responses, scores and feedback, was generated by GPT-4. Purely synthetic data carries a risk, because it exposes the model to a narrow slice of possible inputs and inherits the teacher model's biases. Mixing human-labeled and synthetic data tends to work much better than either alone. Use the human set to anchor quality and the synthetic set to reach the volume and coverage finetuning requires, and apply the balancing ideas from the Feedback Collection, an even spread of scores and controlled response lengths, when you generate it.

Step 4: invest in the rationales

Accurate scores and rankings are obviously required, but the rationales deserve just as much care. Finetuning is known to be a poor way to inject new knowledge into a model; what it does well is teach style and format. For a judge, the feedback format is the main thing you control. Rationales that are specific, critical and tied to the rubric produce a judge whose explanations are actually useful, and, because the model writes the rationale before the score, better rationales tend to support better scores. Place a fixed separator between the feedback and the score so outputs are easy to parse.

Step 5: add reference answers where you can

This step is optional but usually worthwhile. Reference answers help both prompted and finetuned judges. For a finetuned judge they counter knowledge bias, filling gaps in what the base model learned during pretraining, and they make pointwise scoring noticeably more accurate and stable. For factual or domain-heavy tasks, a reference answer often matters more than any prompt wording.

Step 6: train and meta-evaluate

With data, and optionally reference answers, in place, train the judge with standard SFT. The Prometheus repository provides a complete implementation of this process. Hold out a test split of human-labeled data from the start and use it for meta-evaluation: classification metrics if the judge's output is binary, correlation metrics for everything else. For pairwise judges, also measure how often the verdict survives position switching; a judge that flips when the order flips is not ready.

Before you rely on the judge, run a few extra checks:

• Evaluate on out-of-distribution examples to see how far its competence extends.
• Compare its score distribution with the human one to catch systematic leniency or harshness.
• Look for length and position effects directly, by varying response length and order on the same content.
• Re-run meta-evaluation whenever the criteria, the inputs or the base model change.

When not to finetune

A finetuned judge is not always the right tool. If your criteria are generic, such as overall helpfulness, a well-prompted proprietary judge or an existing open evaluator may be good enough and far cheaper to set up. If you cannot collect a reliable human-labeled set, you cannot validate any judge, finetuned or not, so fix that first. And if your task distribution shifts constantly, the task-specific nature of finetuned judges means you will be retraining often. Finetuning pays off when the criteria are specialized, the volume of evaluation is high, and you need control over versioning, cost or data privacy.

A note on cost and operations

Owning a judge shifts costs rather than removing them. You trade per-call API fees for GPU time, model hosting and the engineering needed to keep the judge versioned and reproducible. In exchange you get a fixed model version that cannot change under you, evaluation data that never leaves your infrastructure, and the ability to run millions of judgments without watching a bill grow. Treat the judge like any other production model: pin its weights, record which version produced which scores, and keep the meta-evaluation set around so that a future retrain can be compared with the current one on equal terms.

Key takeaways

• Human judgments remain the reference point, but only after calibration, living guidelines and ongoing agreement tracking make them reliable enough to measure anything against.
• Overlap metrics like ROUGE and BLEU correlate poorly with human preference for open-ended outputs; treat them as cheap sanity checks, not quality measures.
• Prompted proprietary judges are easy to start with but come with versioning risk, cost, hedged opinions and weaker performance on niche criteria.
• Finetuned judges such as PandaLM, JudgeLM, Auto-J and the Prometheus family show that open models can match or exceed GPT-4 in-domain, largely through careful data design: rubrics and reference answers as input, feedback before score, balanced labels and position swapping.
• Model merging offers a simple way to combine pointwise and pairwise skills in one judge, with plain weight averaging working well in most cases.
• Finetuned judges are task-specific. Keep them close to their training distribution and re-validate them against held-out human labels whenever anything important changes.