What a New Judge Has to Beat

Four days ago this blog set seven rules for any new kind of model that wanted to grade our work, and two decision models went through them. The rule on agreement with our judges was met, to three decimal places, by a judge that never says anything but PASS. What stopped the two models was the test that asked a person. So we propose that the first thing any new judge has to beat is the judge that always says yes. This post sets out a proposal, not yet binding, for how a new kind of model, Jev or whatever comes after it, should be tested here.

Oct 05, 2026 · Houston, Texas · Drafted by AI, edited and signed by a person

What the seven rules caught, and what they missed

The Bench post Five Cases, a Hundred Times Each promised that hosted Jev and the open model Laya would go first, under seven rules written before any run. They did. Calibration Vol 002 tested them against our judges, and Calibration Vol 003 tested every judge against one person’s blind labels. Here is what each rule did when it met a real candidate.

Figures from Calibration Vol 002 and Vol 003. The always-PASS scores are labelled exploratory in both issues: they were added after review, not pre-registered. Whether to seat either model was left to the investigator in Vol 002; neither was seated after Vol 003.

Two of the seven did not do their job on their own: rule two, which was too easy, and rule four, which could not measure a model that never varies. Rule two matters most. On a sample where most answers pass, a judge that rarely fails anything agrees with the majority almost automatically, and the coefficient we use cannot tell a careful judge from a lenient one. Calibration Vol 002 said so in its own words, and that problem is the main subject of this post.

The judge that always says yes

We propose that every candidate be scored beside a judge that does nothing. That judge answers PASS to every draw. If a candidate cannot beat it, this test gives no evidence that the candidate can tell failures from passes.

On Calibration Vol 003’s labels the do-nothing judge looks very good: its agreement with the person, 0.871, beats every judge we seat. That is not because it is right. The person passed 77 of the 87 draws, so a judge that passes everything is right 77 times by default. What it cannot do is catch a single one of the 10 failures.

The fix is to score the two piles separately and weight them equally. One share is how many of the person’s FAILs a judge caught; the other is how many of the person’s PASSes it left alone; the score is their average. This is called balanced accuracy. A judge that says PASS to everything scores exactly 0.500 on it, and so, on average, does one that guesses. Applied to Calibration Vol 003’s labels, it gives this.

One labeller, 87 public draws from 35 items. The person’s 10 FAILs fall on 6 items. Intervals are 95% percentile bootstraps that resample whole items, not single draws, so draws of the same item move together (20,000 resamples, seed 20261005; the few resamples with no FAIL in them are dropped). Re-run with four other seeds, every lower end moved by 0.005 or less. Read the order as rough, not as a measured gap. Laya’s 0.461 here is a coincidence of numbers: it is unrelated to the 0.461 majority-class baseline quoted for Laya in The Bench post Five Cases, a Hundred Times Each.

The balanced-accuracy and AC1 columns tell opposite stories. By agreement, the decision models look like the best judges we have. By balanced accuracy, every judge except Laya is above the do-nothing judge’s 0.500 on these counts. The margins are J-A 0.246, J-B 0.183, J-C 0.146 and Jev 0.124. Laya sits below 0.500: it failed six answers the person passed and caught none that the person failed.

The intervals matter as much as the points. Only J-A’s and J-B’s intervals sit wholly above 0.500. Jev’s lower end is 0.482. J-C’s, the tier-1 grader whose verdicts Calibration Vol 001 published, is 0.491: just under the line on every seed we tried, and close enough that a handful more labels could move it either way. With ten failures on six items, these counts cannot show that J-C or Jev beats the do-nothing judge. A test that a judge we already rely on only narrowly misses is not comfortable, but the same test has to apply to all of them.

None of this makes the seated judges right. Each of them failed between 16 and 18 answers the person passed. Calibration Vol 003 found that, on Luna’s public half, they are stricter than this person, and one person cannot say which standard is correct.

A proposed intake for any new kind of model

Decision models were the new thing in September. Something else will be new next quarter. We propose that any kind of model that wants to grade our work go through these steps, in this order. Each step could end the trial, and a trial that ends early would still be published.

Step one: a file before any run

Before a candidate grades anything, we would write down what it is and date it: who makes it, whether the weights are published, the license, the context length, the price, and what its makers and testers have published about it. Its data terms would be checked and quoted. If the terms are unknown, as Jev’s were on OpenRouter on Oct 02, 2026, the candidate would see only the public half of the bank. A model on our own hardware could see both.

Step two: does the work fit?

In Calibration Vol 002, 13 of Laya’s grading states lost text before it gave a verdict. Some were cut by our own script, which shortened 12 states to 1,800 characters, 3 of which would have fitted uncut. Others were cut by the model’s 512-token reading limit. Before a run we would count both kinds of cut, use the checkpoint and settings that cut the fewest, and report any draw that lost text separately rather than mixing it in.

Step three: what does it return, and what do we use?

A decision model returns a probability with every verdict, and that is much of its appeal. Laya’s checkpoint warned that its shipped temperatures were invalid, and its makers say its probabilities are over-confident as shipped. We would not use a candidate’s probabilities as confidence until they had been checked against a person’s labels. Until then, only the verdict would count.

Step four: shadow grading against a person, beside the do-nothing judge

The candidate would grade draws that a person has labelled blind, and its verdicts would count for nothing while it did. To pass, it would have to clear three tests on the same draws.

Catch at least half of the person’s FAILs. This is the failure half of the test Calibration Vol 003 pre-registered for rule three.

Beat the always-PASS judge on balanced accuracy. The lower end of its 95% interval, resampled by item with 20,000 resamples and a seed fixed in the pre-registration, would have to be above 0.500. This is new.

Agree with the person at least as well as the weakest judge we already seat. This is the agreement half of Calibration Vol 003’s test. It is now the weakest of the three, because the do-nothing judge passes it, and it cannot test an incumbent, since the weakest seated judge always clears its own score. For incumbents, the floor would be fixed in advance in the pre-registration: Vol 003’s lowest seated score, 0.631, unless the pre-registration says otherwise and why.

The first two tests overlap. A judge that catches half the failures will usually clear 0.500 on its point estimate. The interval is what separates them: it asks whether the counts are large enough to show the judge is better than doing nothing. On Vol 003’s counts, Jev fails both of the first two tests. J-C passes the first, catching exactly half, and falls just short on the second, by a margin small enough that the next labels could reverse it. Agreement with the panel, the old rule two, would still be computed and printed, but it would no longer decide anything on its own.

Step five: steadiness, only where it can be measured

The candidate would grade a sample five times at its normal settings, and we would count how often its verdict changed, as before. If it gave identical outputs every time, as Laya did, we would call it deterministic and report its steadiness as not measured, not as zero. A model that cannot vary has not shown that it is stable.

Step six: one seat, never the whole job

A candidate that passed every step could take one panel seat. It would never become the only grader, and it would never sit in a reference that checks its own verdicts. The seat would be reviewed at the end of the quarter against fresh human labels.

Step seven: published either way

Unchanged. A candidate that fails at step two would get the same kind of page as one that is seated.

And the judges we already seat

Rule three of The Bench post Five Cases, a Hundred Times Each already promised that every judge, including the ones we use, would be scored against a person’s labels. If this proposal is adopted, J-A, J-B and J-C would be scored under step four in the same Calibration as the next candidate, on the same labels. If one of them fails, that Calibration would say so and say what happens to its seat.

Two things the sample itself would have to change

It needs more failures. Calibration Vol 003’s person failed 10 draws, on 6 items, with up to 4 on one item. That is why every interval above is so wide. The next human labelling round should deliberately include more of the draws that any judge failed, so that the person sees more of the hard cases. That would change the share of failures in the sample, which is exactly why each pile would be scored separately, with the old agreement figure printed beside it.

It needs a second person. One labeller is a reference, not ground truth, and that labeller built the bank. A second, independent labeller, scored against the first, is the only way to say whether the seated judges are too strict or the first person too lenient. We are looking for one, and the Calibration that uses them will say who they are and what they were paid.

And when the new kind of model is the one being measured

Everything above is about a new model as a judge. Protocol STAMP’s main measurement points the other way: it asks a model questions and checks whether its prose answer shows its receipts. A decision model does not write prose. It picks among options you give it, scores on levels you define, or returns a probability. So we propose not to score Jev or anything like it on the prose bank, and not to rank it beside models that write. It would fail for the wrong reason.

The honest version would be a typed form of the bank, where each probe offers the options a careful answer would choose between, including “I can’t verify that.” That is a different instrument, with its own pre-registration. We have not started it, and if we do, the plan will be published here before any run.

What happens next

None of this changes a published number. The seven steps are a proposal until they are written into the pre-registration of the next Calibration that tests a judge and signed before any run. If that happens, the seven rules of Five Cases, a Hundred Times Each stay on the page where they were published, and the corrections register records the change.

Jev and Laya are not seated, and nothing in this post reopens that. A newer version of either, Open-Jev’s 9B or 27B, or the next model nobody has heard of yet, would start at step one like everything else. The first hurdle would be the cheapest judge there is: one that never fails anything.