Methodology

The instrument

A fixed bank, under recorded conditions

The bank. 63 probes across thirteen categories — arithmetic, multi-part instructions, honesty, false premises, prompt injection, sycophancy, live data, safety, ambiguity, overclaim, impossible specifics, provenance and attribution. Items enter because models fail them; probes every model answers are retired as solved. The bank is therefore adversarial by construction and deliberately unrepresentative of ordinary use. 35 probes are public and downloadable below; 28 are held out so that no vendor can train against them, at the stated cost that per-item claims on the held-out half cannot be audited by a reader.

The draws. Each candidate is run in two independent halves of five draws per item, through a named gateway, from a pinned serving provider, at a stated temperature and token budget, on a stated date. Every one of those conditions is recorded with the result. A rate is a property of a model measured a particular way, never of a model alone.

Two rates, two intervals. The narrow interval answers “what is this model’s rate on this bank?” — the bank is fixed, only the draws are random. The wide one answers “what is this model’s rate on probes like these?” — where the choice of items is itself uncertain. Quoting a rate without its interval strips out the part that makes it evidence.

Verdicts

Three tiers, best to worst

• Tier 0 — no judge at all. Where ground truth is mechanical, code decides: does the DOI resolve, is the total correct, did the answer assert the thing that is provably false. 15 of 63 items qualify today, and the standing goal is to move items out of judgment: every item promoted to Tier 0 improves the instrument permanently.
• Tier 1 — one grader, every draw. A single grading model checks every response against a written pass criterion.
• Tier 2 — the panel, on a sample. Three judges from three different model families grade a 20% stratified sample plus every borderline item. Agreement is computed (Gwet’s AC1) and published, with a pre-registered floor below which results are not published. This is audit sampling: measure the error rate on a sample and extrapolate with a stated confidence.

The panel’s job is not to average out taste. Criterion checking is closer to classification, and when judges disagree on a well-written criterion the usual cause is an ambiguous criterion, not a difference of opinion. The panel exists to detect bad criteria and biased graders — and in Vol 001 it found the grader 4.1 points lenient, which is published as Finding 02.

Who may grade. A judge never grades its own model family; the runner refuses before the first call and seats a reserve. A candidate never grades. A grader is qualified by probe — one real call against the cases that have broken graders before — never by reputation. Graders whose safety filters refuse to grade a probe are recorded, because a filtered grader cannot reliably grade safety behavior.

Drift

The anchor, and how the bank grows

The anchor is a reference model measured every round, chosen by a live probe of liveness, provider, latency and reasoning burn rather than by catalog: a reference standard that sometimes returns nothing is not a reference standard. It separates “the harness drifted” from “the model drifted.” There is one anchor today; a second, from an independent family, is required before any movement is reported, and its absence is stated in every issue until then.

Growing the bank without breaking the series. A rate on 63 items and a rate on 103 items are not the same measurement. So the 63 items of bank v1 are the permanent core — never removed, never reworded — and every candidate reports a core rate (comparable to Vol 001 and to every issue that follows) alongside a full-bank rate on the current version. When items are added, every prior candidate is re-run on them so all full-bank rates sit on one scale. The two rates are never mixed in one sentence, and movement is judged on the core. Standardized testing calls this equating; it is how a mortality table moves without severing its own history.

Receipts

Nothing is published that cannot be computed

Every publishable number is computed from a file on disk and registered in figures.json with the SHA-256 of its sources. Reports cite figures by identifier; a number that cannot be computed from data has no route into a document. Every issue’s HTML is byte-frozen and its fingerprint is printed on every page of its PDF. Nine gates — preflight, config, circuit breaker, judge conflict, recusal, agreement, holdout, naming, publishing — each exist because of a failure that already happened, and each is a script that exits non-zero rather than an instruction that can be forgotten.

Downloads

bank_v1_2_public.jsonl — the 45 public probes with their pass criteria, one JSON object per line (bank v1.2; the earlier files are kept unchanged: bank_v1_public.jsonl, the 35 published with v1, and bank_v1_1_public.jsonl, the 37 of v1.1) · figures.json — the figure registry · every issue’s byte-exact source is linked from the archive.