Protocol STAMP is teamed with AI and can make mistakes. Verify before you implement anything. What we got wrong · Free to read, paid for by readers: Contribute
Below is the issue exactly as it was issued; nothing in it has been edited. This frame, the menu and the footer are the site’s, not the issue’s. To check it, download the Source and hash it: the first twelve characters of its SHA-256 are the build above. The Exact copy in the public repository is the same bytes.
Drawn from figures.json (keys cal003.*); hover any mark for its value and key.
The issued document below is unchanged — source fingerprint af3906b9d9ad.
Higher is better: a judge that misses the person’s failures passes what should fail.
Lower is closer to the person: these are answers the person passed and the judge failed.
| Judge | AC1 | Interval | Caught | Failed of PASS | Draws |
|---|---|---|---|---|---|
| J-A · Gemini | 0.685 | 0.515 – 0.819 | 7 of 10 | 16 of 77 | 87 |
| J-B · Grok | 0.631 | 0.453 – 0.780 | 6 of 10 | 18 of 77 | 87 |
| J-C · tier-1 | 0.659 | 0.479 – 0.798 | 5 of 10 | 16 of 77 | 87 |
| Jev | 0.846 | 0.741 – 0.935 | 3 of 10 | 4 of 77 | 87 |
| Laya | 0.779 | 0.657 – 0.885 | 0 of 10 | 6 of 77 | 87 |
| J-B/J-C reference | 0.684 | 0.520 – 0.826 | 5 of 9 | 14 of 71 | 80 |
One labeller, 87 public draws from 35 items; intervals treat draws as independent and are optimistic. Read the issue for the caveats.
We asked a person.
On the same draws, our three judges agree with each other at 0.875 [0.789, 0.950] (the same labelled draws). With the person, at 0.685 [0.515, 0.819] (labelled draws) at best.
A judge that never fails anything would have scored higher. Answering PASS every time: 0.871 (labelled draws). That is what agreement means when most answers pass.
The decision models agreed most and caught least. Jev caught 3 of the person’s 10 failures; Laya 0.
Shows the 87 public-half draws of the panel_luna_v1 sample, one at a time, in a fixed random order
(seed 20261003): the probe, the item's own pass criterion and the response. It never sends a judge's
verdict, an item's history or any rate to the page - the labels are blind by construction, because
the data needed to unblind them is not in what the page receives.
scripts/label_server.py, lines 7–10 · unedited
An agent must never be told about a precondition it could verify.
the house ruleA Calibration is one dated issue of a measurement programme. It is not a leaderboard, and the thing it is mostly about is not the models — it is the instruments people use to rank them.
Vol 001 and Vol 002 asked whether the judges agree with each other. They do. This issue asks the question both of them left open: whether the judges agree with a person.
figures.json under cal003, with the SHA-256 of the files it came from.
Figures not asked by the pre-registration are labelled exploratory.Issued the same day as Vol 002, off-cycle: this Saturday belongs to The Log, which follows this issue tonight. The next Calibration slot is Saturday 10 October.
On these draws, our judges agree with each other more than any of them agrees with a person. The three seated judges agreed with one another at 0.875 [0.789, 0.950] (the same labelled draws), and with one person’s blind labels at 0.685 [0.515, 0.819] (labelled draws) (Gemini), 0.659 [0.479, 0.798] (labelled draws) (the tier-1 grader behind Vol 001) and 0.631 [0.453, 0.780] (labelled draws) (Grok). The intervals are wide and overlap, so this is a direction, not a measured gap.
Read every agreement figure here against one number. The person passed 77 of 87 draws. On a sample like that, a judge that answered PASS to everything would score 0.871 (labelled draws) — higher than every judge we use.
The judges fail more than the person does. The person failed 10 draws; each seated judge failed more than twice that many, most of them answers the person passed. They also missed some of the person’s failures. On Luna’s public half, then, the judges’ standard is stricter than this person’s. One labeller cannot say which standard is right.
The decision models agree with the person more because they fail almost nothing. Jev caught 3 of the person’s 10 failures and Laya 0. Neither meets rule three, so neither is eligible for a panel seat.
The labels. Lawrence Ho labelled all 87 public-half draws of the Luna panel sample from Vol 002, by each item’s own pass criterion, blind on the page to every judge: 77 PASS, 10 FAIL and 0 unsure, between 12:01 and 12:43 on 3 October, before the 18:00 freeze. A label could not be changed once saved.
The judges. Every judge’s verdict on these draws was already on disk from Vol 002: J-A google/gemini-3.7-flash, J-B x-ai/grok-4.6, J-C bytedance-seed/seed-2-1-turbo (the tier-1 grader), Jev typesafe/jev-1.13 and Laya convaiinnovations/laya. No model was called for this issue and nothing was spent.
The questions. Q1: each judge’s agreement with the person, as Gwet’s AC1 with a bootstrap interval. Q2: of the person’s FAILs, how many each judge caught; of the person’s PASSes, how many each judge failed. Q3: a decision model meets rule three if its agreement is no lower than the lowest seated judge’s and it catches at least half of the person’s FAILs.
Agreement with the person (AC1), 87 draws
J-A Gemini 0.685 [0.515, 0.819] (labelled draws)
J-B Grok 0.631 [0.453, 0.780] (labelled draws)
J-C tier-1 0.659 [0.479, 0.798] (labelled draws)
Jev 0.846 [0.741, 0.935] (labelled draws)
Laya 0.779 [0.657, 0.885] (labelled draws)
J-B/J-C reference 0.684 [0.520, 0.826] (labelled draws) on 80 draws; 7 without a reference
Exploratory, on the same draws. The seated judges with one another: all three 0.875 [0.789, 0.950] (the same labelled draws); Gemini and Grok 0.905 [0.815, 0.981] (the same labelled draws); Gemini and the tier-1 grader 0.852 [0.737, 0.949] (the same labelled draws); Grok and the tier-1 grader 0.869 [0.763, 0.963] (the same labelled draws). A judge answering PASS to every draw would score 0.871 (labelled draws) against the person.
So two things are true at once. The judges agree with each other better than with the person, though the intervals overlap. And on a sample where the person passed most answers, a high agreement figure is easy to reach by passing everything; it is the failures that separate the judges.
Of the person’s 10 FAILs, caught · of the person’s 77 PASSes, failed
J-A 7 · 16
J-B 6 · 18
J-C 5 · 16
Jev 3 · 4
Laya 0 · 6
Reference, of 9 and 71: 5 · 14
Each seated judge failed more of the person’s PASSes than the person failed draws in total, and each also missed some of the person’s FAILs. In counts, the disagreement runs mostly one way: the judges fail answers the person passes. As shares of each pile, the misses on the person’s few FAILs are proportionally larger, so the judges are not simply a harsher copy of the person; they are stricter overall and also disagree about which answers fail.
The pre-registered test had two parts. On agreement, both pass: the lowest seated judge scored 0.631 (labelled draws), and Jev and Laya each scored higher — as would a judge that passes everything, at 0.871 (labelled draws). On failures, both fail: Jev caught 3 of 10 and Laya 0 of 10, where at least half was required.
The person’s FAILs fall on only 6 items, up to 4 on one, so the failure half of this test rests on a handful of items.
Consequence. Vol 002 left the seating of a decision model to the investigator. Neither model is eligible under the rule as signed, and neither is seated.
scripts/cal003_figures.py was written by the drafting agent in one pass once the labels were complete, and before any figure was computed. It implements the signed definitions; the exploratory figures were added after the independent review and are labelled as such.Lawrence Ho, Chief Human Investigator, Evaluation Administrator + Instrument Maker, signed the pre-registration, labelled every draw, and signs this issue, and is its sole authority. The prose was drafted by an AI agent, reviewed adversarially by an independent agent, and edited and signed by him. Budday Budderson Studio LLC publishes the series and grades nothing.