LLM as a Judge: How to Grade AI Output Without Fooling Yourself
An LLM judge is a second model that grades your first model's output against a written rubric and returns a verdict with a reason. You need one because most valuable AI output is open-ended and cannot be checked with an assertion. You also need to distrust it: repeated identical comparisons flip about 13.6 percent of the time. Both things are true, and the second is why most judges quietly lie to their owners.
TL;DR
- Use a judge only for output code cannot check. Anthropic's engineering guidance places model-based graders between code-based (fast, cheap, brittle) and human (gold standard, slow, expensive), and notes they "require calibration with human graders for accuracy."
- Aggregate accuracy is real. The MT-Bench and Chatbot Arena paper found strong judges above 80 percent agreement with human preferences, the rate humans agree with each other.
- Per-case reliability is not. A repeated-trial study from April 2026 ran 8,700 judge calls over 29 tasks: a 13.6 percent average pairwise flip rate, 28 percent of questions above a 20 percent flip rate, one at 56 percent.
- The biases are measured and fixable: a 72 percent first-position majority for one judge, 15 to 30 points of verbosity inflation across GPT-4, Claude, and PaLM-2 judges, and a self-preference effect traced to text familiarity rather than quality.
- One trial is not a verdict. That study needed 11 trials on average, 15 on high-variance questions, before a majority vote matched the 50-trial reference answer 95 percent of the time.
What a judge is for
Judges exist because the interesting half of AI output has no correct string. A classifier's label can be compared to a gold label. A summary cannot, nor can a support reply or a research brief. Once you ship anything open-ended, your options are read every output by hand, ship blind, or have a model read them against a rubric.
So the first decision is not which judge model to use. It is whether you need one. If the output can be graded by a schema check, a regex, a compile, a test run, or a diff against a known answer, use code: free, instant, deterministic, easy to debug when wrong. Every case you move from judge to code stops costing money and stops being noisy. The surrounding eval set is covered in how to test an AI agent; this post is the grader itself.
Pointwise, pairwise, or reference-scored
Three shapes, answering different questions.
Pointwise gives one output an absolute score, usually 1 to 10 or per-dimension pass/fails. Use it to track a level across releases. Less sensitive to small differences, and models cluster in the 7 to 9 band unless the rubric forces spread.
Pairwise shows the judge two candidates and asks which is better. More sensitive, because side-by-side surfaces differences an absolute score smooths over. One severe failure mode: it forces a winner. The 2026 study found judges returning decisive verdicts while their own pointwise scores for the same two answers differed by 0.19 to 0.36 points on a ten-point scale, a gap not statistically significant in aggregate. The judge was inventing a preference to satisfy the question.
Reference-scored hands the judge a gold answer beside the candidate and asks whether it matches on what matters. The most reliable of the three, because it converts an open judgment into a comparison against something fixed. Use it wherever you can write a gold answer, which is more often than people assume.
Whichever you pick, the judge prompt has a fixed anatomy.
The rubric is the eval
Everything else is plumbing. A grader with a vague instruction returns a confident number with no information in it, which is worse than no number, because you will chart it. Rubrics fail in three predictable ways.
It is one blob. "Rate the quality of this answer from 1 to 10" asks the model to average factual accuracy, tone, structure, and completeness into one number, then throws away which one moved. Anthropic's guidance is explicit: write structured rubrics with named dimensions and "grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions." Five one-line judges beat one five-line judge at about the same cost.
It has no floor. A dimension needs a written failure condition, not an aspiration. "Accurate" is an aspiration. "Every factual claim about the customer's account appears in the provided context; if any does not, fail" is a floor a model can apply consistently.
It has no exit. If the judge cannot tell, it guesses, and the guess looks identical to a real verdict. Give it a third option and instruct it to use it: return UNKNOWN when the rubric does not cover the case. Those cases are your best source of new rubric lines. Same principle as the grounding rules in how to reduce AI hallucinations: an explicit way out beats a confident invention.
Then lock the output shape, because prose has to be parsed and parsing is where scores go missing. Enforce a schema with a verdict field, a per-dimension result, and a short reason string, using the constrained-decoding approach in how to get reliable structured output from an LLM. Keep the reason field: it is the only thing that tells you whether the judge graded your rubric or something it made up.
Four biases, with numbers attached
These are not hypotheses. They have been measured repeatedly.
Position bias. Judges favor the answer they read first. The MT-Bench work documented it; the 2026 study measured a significant first-position majority of 72 percent for a GPT-4-family judge (p = 0.024). Fix: randomize order per trial, or run each pair twice with the order swapped and discard pairs where the winner changes. The second option costs more and tells you something the first one hides.
Verbosity bias. Longer answers score higher whether or not they are better, measured at 15 to 30 points of inflated preference across GPT-4, Claude, and PaLM-2 judges. Fix: state in the rubric that length is not a quality signal, then check length distributions on your winners. If your new prompt "won" and its outputs are 40 percent longer, you may have measured word count.
Self-preference bias. A judge scores familiar text more highly. The research traces this to perplexity: higher scores go to text the judge finds predictable, not text that is better. Fix: never grade a model with itself. Use a different family, or an ensemble of two, and treat "our new model wins on our own judge" as a claim with a known confound.
Limited reasoning. A judge cannot reliably grade math or logic it could not solve itself. Fix: grade anything verifiable with code and keep the judge for what is genuinely subjective.
One trial is a coin flip
This is the part almost nobody implements, and it decides whether your eval numbers mean anything.
The April 2026 study ran the same comparison 50 times per question, across 29 tasks and two GPT-4-family judges, at default temperature. Pairwise preferences flipped 13.6 percent of the time on average. Twenty-eight percent of questions flipped more than 20 percent of the time. One hit 56 percent, a coin toss with extra steps. Semantically equivalent rewordings of the judge prompt changed the majority outcome in 25 percent of tested cases. Temperature zero reduced the inconsistency without eliminating it. Cross-judge agreement was 76 percent, a Cohen's kappa of 0.51, which psychometrics files under poor to moderate: two judges from the same provider disagreeing on a quarter of the same items.
The reliability curve gives the operating numbers. Eleven trials, on average, before a majority vote recovers the 50-trial reference verdict with 95 percent probability. Fifteen for high-variance questions. Past about 20 the curve flattens and you are burning money.
So the rule is: odd number of trials per case, take the majority, store the vote split. A 6-to-5 majority and an 11-to-0 majority are both "wins" in a spreadsheet and completely different facts. Log only the winner and you have thrown away the uncertainty and kept the confidence.
One honest caveat: that study used deliberately competitive pairs, both answers strong, mean pointwise scores of 9.3 and 9.4 out of 10. It is a stress test, and flip rates run lower when one candidate is obviously worse. But "obviously worse" is not the case you are usually deciding, and not the case that ships a regression.
Calibrate the judge, then trust the judge
An uncalibrated judge produces numbers that look like evidence. Calibration is one afternoon of work and it is not optional.
Take 50 real cases from your own task. Grade them yourself, or have the person whose standard actually matters grade them, and store those labels. Run the judge on the same 50. Compute plain agreement and then Cohen's kappa, which corrects for chance. Read every disagreement: each is either a rubric gap or a judge you should not be using.
Anthropic's guidance is that model graders "should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading." Reference points: over 80 percent human agreement for strong judges in the MT-Bench work, human inter-annotator kappa on subjective language tasks typically between 0.3 and 0.6, and the 2026 cross-judge kappa of 0.51 inside that band. Target the neighborhood of your own human raters, not perfection.
Re-run the calibration whenever the judge model, the rubric, or the prompt template changes. A 25 percent outcome change from an equivalent reword is a documented finding, not an edge case, which makes your judge prompt a versioned artifact. Pin the judge model by exact name, or a provider update will silently move every historical score you have, the same argument for config-pinned models in how to choose an AI model.
The judge harness, in seven lines
What a production grading setup contains once you stop hand-running prompts:
- A judge model pinned by exact name, different from the model being graded.
- A rubric file in version control, one named dimension per judge call.
- A strict output schema: verdict, per-dimension result, reason,
UNKNOWNbranch. - Randomized candidate order on every pairwise trial.
- An odd trial count per case, with the vote split stored, not just the winner.
- A human-labeled calibration set with a recorded agreement number and kappa, re-run on every judge or rubric change.
- Cost and token logging per run, so the eval budget stays visible next to the product budget.
Line seven matters more than it looks. Eleven trials across five rubric dimensions is 55 judge calls for one item, and 11,000 per run on a 200-case set. Grade with a cheap model where the rubric is mechanical and reserve the expensive judge for dimensions that need reasoning, same tier logic as cutting your AI API bill. To run all of it unattended on every change rather than when you remember, write loops not prompts covers the automation shape, and the canary method is how I catch a long-running session drifting before the eval does.
When to leave the judge out
Skip it when code already works: assertions, schema validation, a compile step, a passing test. Skip it as the sole gate on legal, medical, financial, or safety decisions, because a 13.6 percent flip rate is not an acceptable failure mode on a compliance call, and that is what approval gates exist for. Skip it on math and logic it could not solve unaided. And skip any judge you have not calibrated, because the alternative is not "no signal," it is a false signal you will act on.
The bottom line
An LLM judge is a measurement instrument, and every instrument has a noise floor. The failure is not using one, it is using one as if a single call were the truth. Name your rubric dimensions and write a failure condition for each. Randomize order. Force a schema with an UNKNOWN branch. Run an odd number of trials and keep the vote split. Calibrate against 50 human-labeled cases and re-run that whenever the judge changes. Do those five and your eval score becomes a number you can ship decisions on. Skip them and you have built an expensive way to agree with yourself.
Every Wednesday I send one build breakdown with the numbers attached, no theory. Join the newsletter, then grab write loops not prompts for the automation pattern that runs these evals without you.
What is LLM as a judge?
LLM as a judge is the practice of using one language model to grade another model's output against a written rubric, returning a score or a pass/fail verdict plus a reason. It exists because most useful AI output is open-ended: a summary, an email, a research brief, a support reply. There is no string to match against, so exact-match assertions cannot grade it. A judge model reads the task, the rubric, and the candidate output, then reports a structured verdict you can aggregate across hundreds of cases. It is the only method that grades subjective work at a speed and price that lets you run it on every change.
How accurate is an LLM judge compared to a human?
In aggregate it can be close to human-level. The 2023 MT-Bench and Chatbot Arena study found strong judges such as GPT-4 reached over 80 percent agreement with human preferences, which is about the same rate at which two humans agree with each other. The catch is that aggregate agreement hides per-case instability. A 2026 repeated-trial study of 8,700 judge calls found that identical pairwise comparisons flip 13.6 percent of the time on average, and that two judge models from the same provider agreed with each other only 76 percent of the time, a Cohen's kappa of 0.51. Treat a judge as a good aggregate instrument and a poor single-case oracle.
Should the judge score one output or compare two?
Compare two when you are choosing between versions, and score one when you are tracking a level over time. Pairwise comparison is more sensitive to small quality differences because the judge sees both candidates side by side, but it forces a winner even when there is no real gap: the same 2026 study found judges picking a decisive winner while their own pointwise scores differed by only 0.19 to 0.36 on a ten-point scale. Pointwise scoring is less sensitive but gives you an absolute number you can chart across releases. If you use pairwise, randomize which candidate goes first on every trial.
What biases affect LLM judges?
Four are well documented. Position bias: the judge favors whichever answer appears first, measured at a 72 percent first-position majority for one GPT-4-family judge. Verbosity bias: longer answers score higher regardless of quality, measured at 15 to 30 points of inflated preference across GPT-4, Claude, and PaLM-2 judges. Self-preference bias: a judge rates text it finds familiar more highly, which research traces to lower perplexity rather than genuine quality. And limited reasoning: judges are weak graders of math and logic they could not solve themselves. Randomized order, length-aware rubric language, a different judge model than the one being graded, and code-based checks for anything verifiable all reduce the damage.
How many times should you run an LLM judge on the same case?
More than once, which is what most teams get wrong. A 2026 reliability-curve analysis found that 11 repeated trials were needed on average for a majority vote to recover the 50-trial reference verdict with 95 percent probability, rising to 15 trials for high-variance questions, with gains flattening past about 20 trials. Setting temperature to zero reduces the variance but does not remove it. The practical rule: run an odd number of trials, take the majority, and log the flip rate alongside the verdict so a 6-to-5 result never gets reported as a clean win.
When should you not use an LLM judge?
Do not use one when a cheaper grader already works. If the output can be checked by an assertion, a regex, a schema validation, a compile step, or a passing test, use code: it is faster, free, deterministic, and easier to debug. Do not use a judge as the sole gate on anything with legal, medical, financial, or safety consequences, because a 13.6 percent flip rate on a compliance decision is not an acceptable failure mode. And do not use a judge you have never calibrated against human labels on your own task, since an uncalibrated judge produces a number that looks like evidence and is not.
OpusJake is Jake Schincariol's operating system for building with AI: agents, workflows, prompts, and the free resources behind them. Get the next move every week.