Five Cases, a Hundred Times Each

A new kind of model went into early access in September. It answers in typed decisions instead of prose, and a published test of it as an evaluator reported perfect agreement with a human reviewer on 500 decisions. The 500 decisions were five cases, each repeated a hundred times. That is a good test of whether a judge is consistent. It is a very small test of whether it is right. We look at what the result can carry, at the open models that followed Jev, and at the rules any of them would have to pass before we let one grade our work.

Oct 01, 2026 · Houston, Texas · Drafted by AI, edited and signed by a person · Due Sep 14, 2026, seventeen days late

What was tested

Jev is a decision model, made by TypeSafe. OpenRouter’s overview describes three kinds of answer: a choice among fixed options, a score on levels you define, and a probability that a yes-or-no proposition is true. Every answer comes back with a probability attached. It does not write sentences. OpenRouter lists it at $0.042 per million input tokens, with output free, and early access opened on Sep 15, 2026. TypeSafe has published no weights and no paper.

That shape is close to what an evaluator does. Most grading is a pass or a fail against a criterion somebody wrote down, and a model whose maker says it returns calibrated probabilities, without composing an essay first, is an obvious candidate for the job. On Sep 20, 2026 LangChain published a test of exactly that. They built a weather agent, captured five of its responses as fixed examples, had a human reviewer label each one, and then asked four judges to grade all five, a hundred times each.

The other three judges were GPT-5.6 Terra at 99.8%, GPT-5.6 Luna at 96.4% and Claude Sonnet 4.6 at 80.0%. Jev averaged 0.44 seconds and $0.00035 a call. Across the whole comparison it cost $0.34, against $28.17 for the Claude judge, about 83 times less. The variance of its quality score across repetitions was the lowest of the four by a wide margin: Luna’s was 433 times higher, Terra’s 913 times and Claude’s 92 times.

None of that is in dispute here, and the authors were careful about it. They called it “a narrow test,” and wrote that they “still need to see whether the results in this experiment carry over to other agents and production workflows,” and that “a consistently wrong evaluator can produce bad feedback at scale.” This post is about what the number can carry, not about whether it was honestly reported.

Five hundred is five

Repeating a case measures the judge. It does not add evidence about the cases. If a judge sees the same response a hundred times and gives the same verdict a hundred times, we have learned that it is stable on that response. We have not learned anything new about whether that verdict is correct. On correctness, the experiment has five observations, and all five agreed with the reviewer.

Five of five is a good result. It is also a result a much weaker judge would often produce: one that is right 80% of the time would score five of five about one time in three. The standard exact interval for a proportion, applied to five successes in five tries, runs from 47.8% to 100%. In plain terms: five cases cannot distinguish a judge that is always right from one that is right a little more than half the time. That is not a criticism of Jev. It is a statement about five.

The same arithmetic applies to the other three judges. A score of 80.0% over 500 decisions is consistent with many splits, from one case wrong every time to every case wrong a fifth of the time. The post reports the rate, not the case-by-case split, so we cannot say which. If it is the first, the gap between the best and worst judge in the comparison is one case.

Consistency is still worth having. A judge that gives different verdicts on identical inputs is useless as an instrument, whatever its average. It is the first thing to check, and Jev passed it decisively: five hundred of five hundred verdicts the same as the reviewer’s, and the steadiest quality scores of the four. It just isn’t the second thing, which is whether the consistent answer is the right one. A judge can be perfectly consistent and consistently wrong, which is the authors’ own warning.

What we ask of a judge before we seat it

We have a stake in this, because Protocol STAMP grades with models too, and the way we check our graders is the most useful thing we can offer as a comparison. Every draw in measurement one was graded by one model. Then two further judges, from two other labs, independently graded a sample of 148 draws. With the original grader they form a three-judge panel, and we published their chance-corrected agreement, a statistic called Gwet’s AC1.

Neither design is complete. Theirs has a human reference and very few items. Ours has many items and no human reference: three models agreeing with each other is evidence that the criteria are clear, not proof that the verdicts are true.

And we found a flaw in our own design this week. Calibration Vol 002 is meant to ask whether a full re-grade by a Gemini model agrees with our panel. One of the three panel seats is already a Gemini model. A judge agreeing with a majority it helped form is partly agreeing with itself. The same is true of measurement one: the model that graded every draw holds one of the three panel seats that checked it. The re-frozen pre-registration for Vol 002, drafted and waiting for signature, proposes comparing the re-grade against the two other seats instead, and says why. We found that by reading our own plan with the question this post asks of LangChain’s.

Are we going to adopt Jev?

Not yet, and not on the strength of anything published so far. But we are going to test it, along with the open models that appeared in the weeks after it, and this is how. This section is a plan; the result will be published in a Calibration.

The case for trying is real. Grading is one of the most expensive parts of an evaluation like ours, and one of the least visible. In measurement one a single model graded all 616 draws, and two more judges then had to be paid to check a sample of 148. The case against adopting one now is that we would be replacing a check we can describe with one that has been shown, in public, on five cases.

The candidates

Within three weeks of Jev opening to early access, open alternatives had appeared. DataCamp listed seven on Sep 23, 2026, without benchmark figures for any of them. We have not yet checked each project for published results. Two of the open ones are worth naming beside Jev, because each stands for a different way of doing this.

Others on DataCamp’s list include Nimble from Bespoke Labs, Kev and NanoJev. A model gets a trial seat once there is published evidence to compare against, whoever makes it.

Each of the three is weak in a different place. Jev has the most third-party evidence, and it is the only one we cannot run ourselves. Open-Jev comes within three tasks of Jev on JevBench, and its smaller versions run on hardware we could buy this year, but its largest version needs a data-center GPU we do not have. Laya is the smallest and the most openly licensed, but out of the box it is weak. Its makers report a zero-shot score below what you would get by always picking the most common answer, and they say its probabilities are over-confident as shipped. Its English checkpoint reads 512 tokens, and a real share of the responses we graded in measurement one would not fit in it. Its 1,024-token checkpoint would take most of them.

Laya’s weakness points at its most interesting use. It is built to be fine-tuned, and we have something to fine-tune it on: graded draws from the public half of the bank. Trained only on those, its accuracy on the held-out half would show whether it generalizes to probes it has never seen. That is worth knowing, with one limit stated plainly. It would be learning to copy our grader, so it could only ever be as right as that grader. It would not show that the verdicts are correct. Only the third rule below does that.

The rules any of them would have to pass

These apply to Jev and to every open model alike. They are written here, before any run, so that the result cannot shape them.

One: shadow first. A judge on trial grades alongside the existing panel, and its verdicts count for nothing until it has been measured. No published rate will move because of a judge on trial.

Two: agreement with the panel, on the same terms. The floor is the one our current grader had to clear: an AC1 of 0.70, on the same kind of stratified sample, with its interval printed.

Three: agreement with a person. This is the gap in our own design, and we are committing to close it. Where code cannot decide a verdict, Lawrence Ho will label a sample of draws himself, and every judge, including the ones we already use, will be scored against those labels. One labeler is a reference, not proof, and it will be reported as one.

Four: steadiness, measured our way. Each judge grades a sample of draws five times at its normal settings, and we publish how often its verdict flips. The judges we already use are measured the same way, so the comparison is fair.

Five: held-out probes and judges. A model being measured has to see the probes; that is what measuring it means. A judge is different. We will check and publish the data terms of every judge that sees held-out probes, starting with the ones we already use, which we have not yet done. Until Jev’s terms have been through that check, it grades only the public half. An open model running on our own hardware can grade both.

Six: a judge should not check itself. In measurement one, the model that graded every draw also held one of the three panel seats that checked it. We are proposing to end that from Calibration Vol 002 on, and the signed pre-registration will say whether we have.

Seven: the result is published either way. A judge that fails gets the same page as one that passes, with the same tables.

What happens, in order

The question goes into a Calibration as a pre-registered question, signed before any run. Hosted Jev and Laya go first, because both can be run this month: Jev through the gateway we already use, and Laya on a laptop. Open-Jev’s 9B version can follow on a consumer GPU. The 27B waits until we have a machine that can hold it, and the issue will say that it waited and why. If one of them clears every rule, it takes one panel seat, not the job of grading every draw, and we will review that seat at the end of the quarter.

If none of them clears the bar, the Calibration will say that none of these judges is yet a substitute for the one we use, and we will publish that too.

A cheap, fast, stable judge would change what an evaluation program like this one can afford to do. That is exactly why it should have to pass more than five cases first.