Protocol STAMP is teamed with AI and can make mistakes. Verify before you implement anything. What we got wrong · Free to read, paid for by readers: Contribute
A reader should never have to guess what a word on these pages means. These are the terms as the program uses them; where a term has a wider meaning elsewhere, the narrower one here governs.
Measures
| Term | Meaning here |
|---|---|
| Failure rate | The share of draws that failed their item’s written pass criterion. On this bank, which is adversarial by construction, it is not a general error rate and cannot be converted into one. |
| Draw | One response from one model to one probe under recorded conditions. Each candidate gets two halves of five draws per item. |
| On-bank interval | The narrow 95% interval: this model, on these 63 items, draws random. Use it for week-on-week comparison. |
| Beyond-bank interval | The wide 95% interval: this model, on probes like these, item choice itself uncertain. Use it for any claim about the model. |
| Noise floor | The gap between a candidate’s two halves — what the instrument produces when nothing changed. No movement smaller than it is reported. |
| Reliability | Spearman-Brown full-test reliability from the two halves. Vol 001: 0.945. |
| Design effect | How many draws it takes to carry the information of one independent one, because most items sit at 0% or 100%. Vol 001: 616 draws ≈ 87 independent. |
| Agreement (AC1) | Gwet’s AC1 across the three judges on the panel sample. Pre-registered floor 0.50; below it, nothing is published. Agreement is not correctness. |
| Leniency | How far a grader sits from the panel mean on identical draws. Reported, and corrected for, never assumed. |
| Attrition | Draws lost to truncation, refusal or unparseable output. Above 5%, a run is disqualified, because what vanishes is never random. |
Grades
Publications
Three publications, and they are not the same kind of thing. What separates them is where the material came from and who is answerable for it.
The Calibration is what we ran ourselves — our own measurements, written up. We design the study, fire the bank under recorded conditions, and report what came back, the runs that failed included. Written and signed by a person, who answers for every number in it. Numbered, alternate Saturdays.
The Log is what the machine reports. The AI reporter assembles it from the figure registry and nothing else: what ran, what it cost, what moved. It does not interpret, does not argue, and prints no figure it cannot compute. No person writes it. Numbered, alternate Saturdays.
The Bench is what we read, watch and argue with — everything we did not run ourselves. Other people’s data, other labs’ models, and the questions we are still working out in public. It is a blog: a position rather than a measurement, written and signed by a person, posted Mondays and Thursdays at 15:37 America/Chicago. From Vol 002 it carries a volume number so a post can be cited by name, but a Bench volume is not an issue number, because an issue is a run and a Bench post is not one.
The line between them is a rule, not a mood. Nothing on The Bench is a finding. When something we read there turns out to be worth measuring, it goes into a Calibration, where it arrives with its conditions and its interval attached — and where being wrong costs us a published correction. Issues of The Calibration and The Log ship at 15:37 America/Chicago; an issued number is never edited.
STAMP shares its name with several unrelated protocols, papers, and products — see Other STAMPs for the full list and how to cite this one unambiguously.
By volume and date, which every issue carries in its name, with the URL, which is permanent. Write the full name, Protocol STAMP — two words, in that order, never run together and never camelCased. Not “the STAMP protocol,” which in networking means RFC 8762, and not “STAMP” alone; Other STAMPs spells out the distinction. Work issued before Sep 07, 2026 was published as The STAMP Protocol and is cited under that name.
Protocol STAMP. The Calibration Vol 001 · Sun20260830. Budday Budderson Studio LLC, 2026. https://protocolstamp.com/calibration/001-issued-sun20260830/ build be2bc9d7ca35
Cite a finding by its number inside the issue (“Vol 001, Finding 02”). Quote a rate with its interval and its conditions; a rate quoted bare is misquoted.