Protocol STAMP is teamed with AI and can make mistakes. Verify before you implement anything. What we got wrong · Free to read, paid for by readers: Contribute
Protocol STAMP measures whether a score given to an AI model survives being taken twice. Every rate it publishes carries its interval and its conditions, every figure is registered with the fingerprints of the files it was computed from, and it publishes its own errors as findings.
What this is
It is published by Budday Budderson Studio LLC, a New Mexico company, on alternate Saturdays at 15:37 America/Chicago. The latest Calibration is one click away at the top of every page.
How to read us
Everything on this site belongs to one of three publications. They look alike on the page, but they make different promises, and the difference decides how much weight a sentence can carry. The short version: The Calibration is what we measured, The Log is what the machine recorded, and The Bench is what we think.
What we measured. Pre-registered and signed before any run; the only one of the three that can state a finding.
Archive → The Saturdays betweenWhat the machine recorded: what ran, what it cost and what moved, assembled from the registry and written by nobody.
Archive → Mon and Thu · 15:37What we think: what we read, watch and argue with. Nothing on The Bench is a finding.
The Bench →| The Calibration | The Log | The Bench | |
|---|---|---|---|
| What it is | Our own measurement, written up | The harness’s record of what ran, what it cost and what moved | The blog: what we read, watch and argue with |
| Where the material comes from | Runs we designed and fired ourselves | The figure registry, and nothing else | Other people’s data, papers and models, our own published issues, and plans |
| Who writes it | Drafted by AI, reviewed adversarially by an independent agent, edited and signed by Lawrence Ho | The AI reporter assembles it; no person writes it. A person releases it | Drafted by AI, reviewed adversarially by an independent agent, edited and signed by Lawrence Ho |
| Signed before it starts | Yes: the questions are pre-registered and signed before any run | No: it reports what already happened | No: it is a position, not a study |
| Can it state a finding? | Yes. Every finding carries its interval and conditions | No. It prints figures; it does not interpret them | No. Nothing on The Bench is a finding |
| When | Alternate Saturdays | The Saturdays between | Mondays and Thursdays, 15:37 America/Chicago |
| If it is wrong | Never edited. A correction is issued under a new date | Never edited. A correction is issued under a new date | Corrected in the open, with a row in the register |
| Comes with | A PDF, the byte-exact source and the run files | A PDF and the byte-exact source | Its sources, listed at the foot |
| Read it for | Our numbers, and what they rest on | What changed since last time, and what it cost | What we make of other people’s claims, and what we plan to test next |
Vol 003 asked whether a new kind of model, Jev, could grade our work, and set rules before any run.
Numbering. Calibrations and Logs are issues, numbered and dated in their names, and listed in the archive. Bench posts carry volume numbers from Vol 002 so they can be cited by name, but a Bench volume is not an issue and is not in the archive; the posts are listed on The Bench. The exact rules are in Definitions.
Between the issues
Alongside the three publications the site keeps standing pages that are updated as the world moves. They are not findings; every line on them carries its source and the date it was last checked.
Rules and instruments that bear on how AI is evaluated, sourced and labelled.
/docket/ AI policyDated events in AI governance, each with its own citable entry.
/chronicle/ AI policyThe US rename of AI, traced: the order, who followed it, and why this site still writes “AI.”
/si/ AI policyHow armed forces govern AI, country by country.
/military/ MethodHow a rate is produced, versioned; a change is a disclosure, not an edit.
/methodology/ MethodWhat each term means here, including how to cite an issue.
/definitions/What it is for
Protocol STAMP works on AI governance and evaluation: whether a claim made by a model can be checked, and whether a score given to a model can be trusted. Of the three things an AI is usually asked to be — helpful, harmless, honest — we take honesty to be the one the other two rest on. A helpful answer that is false is not help, and a harmless answer that is false is not safe. That is a judgment, humbly held, and it is not a finding. Nor is it a claim to measure honesty: the paragraph above says we do not, and we do not. What we measure is narrower and duller — whether an answer arrives with the derivation that would let somebody else check it.
Not simply fewer wrong answers. The protocol asks a model to declare, in one line, the claims it cannot stand behind — the weak claim, where it came from, and one step that would settle it — and to declare them only when a stated condition holds, so that the line means something both when it appears and when it does not. Full compliance is the target, and full compliance is two-sided: every turn that meets the condition carries its line, and every turn that does not stays clean. What we are driving toward zero is not the model’s willingness to answer. It is the confident specific offered with nothing behind it.
A model narrating its reasoning is still generating text; a trace reads like an account of actions taken and is not one. The receipt is deliberately a fixed shape instead — a named weak claim, a basis that is exactly one of four words, and a check a reader can perform — because a shape can be scored, counted, and caught when it is wrong. A receipt is not a warranty. It is what makes being wrong auditable.
A witness in the box is not asked only what they saw. They are asked how certain they are, on the record, in front of someone who can come back to it. The protocol is being extended to ask a model the same thing, for an arithmetic reason. Confidence compounds. Ninety per cent, thirty times over, is 0.930 — about four per cent, roughly one in twenty-four — whenever each step has to hold for the answer to hold, which is what a derivation is. A chain of individually reasonable answers can end almost nowhere, and no single link in it ever looked wrong.
A stated confidence is not an instrument reading, and the protocol will not pretend otherwise: a model’s “90%” is generated text like the rest of the answer. One of the thirteen models in the cross-model test put it exactly right about its own — “that was a generated phrase, not a readout of an internal probability. It may correlate with reliability. It isn’t an instrument.” It is worth asking for anyway, because a number is a claim, and a claim can be scored: collect enough of them, compare each with what actually happened, and you learn whether that model’s ninety means ninety. That is calibration, and it is what turns a confident sentence into evidence. It is also what this publication is named after.
A juror does not only ask whether a witness is lying. They ask what the testimony is worth, and then what the verdict is worth once every necessary piece is stacked on top of it. Thirty pieces, each ninety per cent sound and each load-bearing, leave the verdict resting on about one chance in twenty-four. Correlated mistakes make that worse rather than better, because a step that goes wrong tends to carry the steps after it.
The lesson is not that a model should say a hundred. It is the opposite. A model that reports ninety and is right ninety times in a hundred is working exactly as intended, and an agent built on it can be designed accordingly — verify at the fifth step, branch at the tenth, hand it to a person at the twentieth. A model that reports a hundred and delivers ninety leaves its designer nothing to build on, and the chain rots without any single step ever looking wrong. What must approach a hundred per cent is compliance with the protocol — the line fires whenever the condition holds — and the truthfulness of the number. Not the number itself.
Putting honesty first is not a ranking of three separate virtues. The other two are derived from it, and neither survives without it.
Harmless. If you do not know, say you do not know. If you are eighty per cent sure, say eighty — not as a hedge that reads like modesty, but as the number. An answer delivered at full confidence when the confidence is not there is not a harmless answer: it is the harm itself, arriving in a form the reader cannot inspect. Harmlessness is not caution and it is not politeness. It is refusing to hand somebody a load-bearing claim without telling them what it will bear.
Helpful. Below the threshold, hedging and stopping is not help. Say what would close the gap: which document would settle it, which page of it, what to look for, what to go and check, who else to ask. The receipt already carries a check field for exactly this reason — a stated uncertainty with no next step is an apology, and an apology is not help. And this is why honesty sits underneath both: a model that will not say what it does not know cannot ask for the thing that would fix it.
The threshold is a choice rather than a measurement, and is named as one, the way the release hour is named in the colophon: below ninety-five per cent, ask. It is a cousin of the test the receipt already applies in cruder form — a specific the model would not stake $100 on. And it is bounded by the same rule that governs the receipt: the request has to be specific enough to act on. A model that asks for help on every uncertain turn has built a second signal that always fires, and a signal that always fires carries no information.
The confidence field is not in the receipt as published. STAMP v1.70 emits [weak | basis | check]; asking a model to state how sure it is is queued, not shipped, and this paragraph says so until it is.
Who makes this
The program is made by a crew of four: a person, a company, an AI agent and deterministic code. Findings are produced with AI assistance throughout, so what each of us may do unattended is published here rather than left to be inferred, including the things the machines are forbidden to do. When the crew grows, it grows on this page first, on the day it grows.
Designs the study, runs the bank under recorded conditions, builds the gates, reads every result, writes the prose of a Calibration and signs it. The person who answers for every number here.
RuleOnly a human named on this page issues a Calibration; no agent may, and none ever will.Publishes the program and every Budday Budderson Production; Protocol STAMP, Jester Mind, and bud.day are under it. Postal address: 2105 Vista Oeste NW, Suite #E3993, Albuquerque, NM 87120. Contact: s@bud.day.
May notThe studio’s accounts distribute what the publication issues; they never speak for it.Writes The Log, the derived record of what ran, what it cost and what moved, from the registry and nothing else.
May notReport anything typed rather than computed, editorialize about what a number means, write The Bench, or post anywhere.Deterministic Python with no weights. Seats and rotates the judging panel, refuses a judge that would grade its own family, and records every seating.
May notGrade anything.Our own receipts
Every issue in the archive links its byte-exact HTML source.
Compute its SHA-256. The first twelve characters are printed in the foot of every page of the issue’s PDF.
figures.json holds every citable figure with the fingerprints of the files it came from.
An issued Calibration is never edited; corrections are findings, dated in Corrections + changes.
Or get one email when an issue is released.