Judges Who Agree With Each Other

Last week our three judges agreed with each other at 0.875 and with one person at between 0.631 and 0.685. We went looking for whether anyone else had found the same thing. Nine published studies bear on it. None of them measured it, and that gap is part of the story. Here are nine published studies of AI judges, from 2023 to 2026: what each one found, where it reaches our own work, where it does not, and what we propose to take from it.

Oct 08, 2026 · Houston, Texas · Drafted by AI, edited and signed by a person

The claim everything started from

In June 2023 a team behind the Chatbot Arena reported that strong models could stand in for human raters. GPT-4, used as a judge, matched controlled and crowdsourced human preferences “achieving over 80% agreement, the same level of agreement between humans.” That sentence is widely cited, and the paper helped make grading by model normal, including in programmes like ours.

Two details of that result matter for everything that follows. The agreement was plain agreement, the share of decisions that matched, with no correction for the matches that would happen by chance. And the decisions were mostly pairwise: which of two answers is better. The same paper named the weaknesses it saw: a preference for whichever answer comes first, for longer answers, and for answers the judge itself wrote, and limited reasoning on hard questions. The authors proposed fixes for some of them. The three years since have mostly been spent measuring how far those weaknesses go.

Four things agreement does not show

Order

Eleven days earlier, Wang and colleagues had shown how much order alone could move a judge. With ChatGPT judging, a 13-billion-parameter model beat ChatGPT itself on 66 of 80 queries just by having its answer placed in the favoured position. Among their fixes: judge each pair in both orders and combine the results.

Self-preference

In April 2024 Panickssery, Bowman and Feng reported that judges such as GPT-4 and Llama 2 can tell their own writing from other models’ and people’s at better than chance, and that they rate it higher. When the authors fine-tuned models, the stronger the self-recognition, the stronger the self-preference: a linear correlation, in their words. Our own rule six, that a judge should not check itself, addresses a related but different problem: a reference that contains the grader being validated.

Leniency hidden inside agreement

In June 2024 Thakur and colleagues found that only the best and largest judges reached reasonable alignment with people, and that even they could differ from human scores by up to five points. Their warning is the one this blog keeps returning to: “judges with high percent agreement can still assign vastly different scores.” They also found judges tending towards leniency. On our labels the seated judges went the other way: they were stricter than one person. With one labeller we cannot say which side is off.

Hard cases

In October 2024 the JudgeBench team built pairs of answers where one is objectively correct and the difference is hard to see. On those, many strong models, GPT-4o among them, did only slightly better than random guessing. A judge that agrees with people on easy cases can still fail on the cases that decide a result.

Steady is not the same as right

The newest and largest study we found says this most plainly. In June 2026 Norman, Rivera and Hughes ran 21 judges from nine providers across three benchmarks, about 541,000 judgments in all. Its title is its finding: Reliability without Validity.

Three of their results bear directly on our work. First, plain agreement overstated chance-corrected agreement by 33 to 41 percentage points for every judge on MT-Bench. Second, a judge’s rank could move by up to 14 places between benchmarks. Third, in two judges deployed in production, test–retest reliability above 0.95 sat alongside a position bias above 0.10. A judge can give the same answer every time and still be pushed around by something that has nothing to do with the answer.

That is the point The Bench post What a New Judge Has to Beat made about steadiness. Laya returned identical outputs on all five gradings, so its count of verdict changes was zero and told us nothing. The 2026 study shows the same gap at scale.

A 2025 study points the other way, and the contrast is useful. Han and colleagues tested 54 models as judges of answer accuracy. Of the 27 that met their bar, 23 judged within the range of ordinary human disagreement. Four were what the authors call super-consistent: they agreed with the human consensus score more tightly than people agree with each other. The authors do not say whether that is good. It might mean the models filter out human noise, or that they miss real nuance. In their words, their benchmark “cannot definitively determine which interpretation is correct.”

Our result is the reverse shape. Our three judges agreed with each other at 0.875 and with one person at no better than 0.685: they were closer to one another than to the person. None of the nine studies here reports that pattern as a finding, because none of them sets judges’ agreement with each other beside their agreement with people. The 2026 study comes nearest in spirit, with its title. A panel of judges that agree with one another is reliable in that sense, and on our labels that reliability did not carry over to agreement with the person.

Which number to print

The 2026 study recommends one thing to practitioners above all: report Cohen’s kappa (or Krippendorff’s alpha) alongside any plain agreement figure, and treat the chance-corrected one as the headline.

The three studies here that, like us, judge single items point the same way. Thakur and colleagues found Scott’s pi “more informative” than percent agreement (in the paper’s current version), and Han and colleagues and Bavaresco and colleagues use Cohen’s kappa.

We already publish a chance-corrected coefficient, but a different one. Protocol STAMP uses Gwet’s AC1, chosen because it is not dragged down when one verdict dominates, as PASS does in our samples. Kappa is known to collapse in exactly that situation: it can be low even when nearly every decision matches. AC1 was meant to avoid that.

Calibration Vol 003 showed the cost of the choice. The property that protects AC1 from the collapse is the same one that let a judge answering PASS to everything score 0.871 against the person. Kappa gives that judge exactly zero, because a judge that never varies has no agreement beyond chance. Here are both coefficients on the same labels.

87 public draws from 35 items, one labeller, 10 FAILs on 6 items. AC1 is from Calibration Vol 003. Plain agreement and kappa are our arithmetic on the counts that issue published. Balanced accuracy is from The Bench post What a New Judge Has to Beat. No intervals are given; with ten failures, every column is a rough order, not a measured gap.

Read down each column and the coefficients disagree about which judge is best. By plain agreement and by AC1, the do-nothing judge beats every real judge, and of the real ones Jev comes closest to the person. By kappa and balanced accuracy, J-A does, and the do-nothing judge falls to the bottom with Laya. Neither family of numbers is wrong. They answer different questions. AC1 corrects for chance using a model that expects little chance agreement when one verdict dominates, so a judge that matches the dominant verdict scores well. Kappa and balanced accuracy ask whether a judge can tell the person’s failures from the passes.

Calibration Vol 002 already printed Cohen’s kappa once, for Jev against our two-judge reference (0.323), as an exploratory figure beside AC1, and it printed Fleiss’ kappa for the panel as a pre-registered figure. The same issue said AC1 is the right coefficient for this programme, and we still think it is the right one to keep. What we propose is a change of emphasis: Cohen’s kappa printed beside AC1 as standing practice, not as an exception, so that neither stands alone.

Where the research does not reach us

Most of it judges pairs; we judge one answer at a time. Order bias, the most studied weakness, is measured by swapping two answers. Our judges read a single response against the item’s own pass criterion, so there is no second answer to swap. Three of the nine, Thakur’s, Han’s and Bavaresco’s, judge single items, and they reach our setup most directly; Zheng and colleagues also tested single-answer grading. Single-answer grading has ordering choices of its own, such as whether the criterion comes before or after the response, and it is not immune to the other biases.

Our criteria were written first, by the person who labelled. Shankar and colleagues, in April 2024, described what they called criteria drift: people need criteria to grade outputs, but grading outputs is how they work out the criteria. Each bank version is fixed, and every probe carries a written pass criterion set down with it, which guards against drift within a run. Probes enter the bank because models fail them, which is a cousin of the same problem. But the person who labelled the 87 draws also wrote those criteria, which brings a different risk: an author reads a criterion the way it was meant, not necessarily the way it is written.

None of this transfers automatically. Bavaresco and colleagues tested 11 models against human labels on 20 datasets in 2024 and found substantial variation by the property being judged, by the expertise of the human raters, and by whether the text was written by a person or a model. Their conclusion is ours too: judges “should be carefully validated against human judgments before being used as evaluators.” A judge’s record elsewhere is a reason to test it here, not a substitute for the test.

What we propose to take from it

These are proposals. Like those in The Bench post What a New Judge Has to Beat, they bind us only once they are written into a pre-registration and signed before any run.

Print four numbers for every judge, not one. Every table that scores a judge against a person would show plain agreement, AC1, Cohen’s kappa and balanced accuracy, with the always-PASS judge in its own row. No single coefficient would be the headline. A reader could see in one line whether a judge agrees because it discriminates or because the sample is lopsided.

Record whether a judge shares a family with the model it grades. The self-preference research is about judges favouring their own writing; we would extend that, as a precaution, to models from the same maker. Each Calibration would list, for every judge and every graded model, whether they come from the same maker, so a reader can see any overlap at a glance.

Measure people against people. Zheng and colleagues compared their judge with the agreement between humans; Han and colleagues built their whole method on the same yardstick. With one labeller, we cannot compute it. A second, independent labeller would give us the baseline Zheng and Han use, and would tell us whether 0.631 to 0.685 is low for a judge or about what two people would manage.

None of these changes a published number. The judges seated today stay seated, and Calibration Vol 003’s findings stand as issued.