The Reader’s Kit
Eight questions to ask of any AI test scoreA Budday Budderson Production
Say an AI model “scores 43%”. Before you repeat that number, ask it eight questions. Each one below is a rule this program holds itself to, and each links to where it is set out in full.
01
Is the rate quoted with its interval?
A rate quoted bare is misquoted: the interval is the part that makes it evidence.
DefinitionsA rate quoted bare is misquoted: the interval is the part that makes it evidence.
02
Which interval: on this test, or on tests like it?
The narrow one compares week on week. Any claim about the model itself needs the wide one.
Two rates, two intervalsThe narrow one compares week on week. Any claim about the model itself needs the wide one.
03
Under what conditions was it measured?
Gateway, serving provider, temperature, token budget, date. A rate belongs to a model measured a particular way.
The drawsGateway, serving provider, temperature, token budget, date. A rate belongs to a model measured a particular way.
04
What is the test made of?
A bank built from items models fail is adversarial by construction. Its failure rate is not a general error rate.
Failure rateA bank built from items models fail is adversarial by construction. Its failure rate is not a general error rate.
05
Would it survive being taken twice?
Two independent halves set a noise floor. A movement smaller than that gap is not reported.
Noise floorTwo independent halves set a noise floor. A movement smaller than that gap is not reported.
06
Who graded it, and did the graders agree?
No judge grades its own model family, and agreement is published against a floor. Agreement is not correctness.
VerdictsNo judge grades its own model family, and agreement is published against a floor. Agreement is not correctness.
07
What went missing?
Draws lost to truncation, refusal or unreadable output. Above 5%, the run is disqualified: what vanishes is never random.
AttritionDraws lost to truncation, refusal or unreadable output. Above 5%, the run is disqualified: what vanishes is never random.
08
Can you check it yourself?
Every published figure is registered with the SHA-256 of its sources, and every issue's fingerprint is printed on its PDF.
ReceiptsEvery published figure is registered with the SHA-256 of its sources, and every issue's fingerprint is printed on its PDF.
Try it yourself
The test, and a finished issue
The 45 public probes, each with its written pass criterion, one per line: bank_v1_2_public.jsonl. Run any of them on any model, and grade the answer against the criterion.
A finished issue, to read with the eight questions beside it: The Calibration Vol 004 (PDF). The archive has every issue.
Keep it: the Reader’s Kit as a PDF, or print this page. It is free and needs no sign-up.
Subscribe for every issue of The Calibration and The Log; the Reader’s Kit arrives in your welcome email.