Protocol STAMP is teamed with AI and can make mistakes. Verify before you implement anything. What we got wrong · Free to read, paid for by readers: Contribute
These terms are written to pre-empt the specific ways this work could be misread, not to discharge a legal obligation. Where a statement exists because we could be wrong, it says so.
What a rate is not
A 43% failure rate on this bank does not mean a model is wrong 43% of the time. It means it failed 43% of a set of questions selected because they are hard, in categories chosen because models are known to struggle with them. The rates are useful for comparison between models on the same probes and for change over time on the same probes. They are not a statement about typical performance and cannot be converted into one. Anyone quoting a STAMP rate as a general error rate is misusing it.
Every number is conditional on the instrument, and describes an endpoint on a date rather than a model. Model weights, serving stacks, quantization and routing change without notice behind an unchanged name; 56 of 63 responses — 89% — differed between two runs of the same prompts at temperature 0 on the same pinned provider.
Not
STAMP is independent measurement published openly. It is not an accredited conformity assessment, not a regulatory audit, and not a certification of any kind. Nothing here satisfies an obligation under any AI statute or standard, and no result should be presented to a regulator as though it did. Nothing here is procurement, legal, financial or investment advice. Agreement between judges is not correctness. Model, product and company names are the property of their owners and are used for identification only.
Fallibility
It can be wrong, and has been. Every part of it is capable of error, and the parts are worth separating because they fail differently.
A model running the protocol can be wrong. The receipt line says the model could not stand behind a particular claim. Its absence does not certify the rest. A model can state something false in a sentence that carries no line at all, and the protocol makes that outcome less frequent rather than impossible. Nothing here is a substitute for checking a fact you intend to rely on.
A published figure can be wrong. The numbers are computed from files rather than typed, the graders are audited by a panel rather than trusted, and the gates refuse rather than warn — and none of that makes a figure correct. It makes an error findable. Five of the six findings in the first issue were about instruments rather than models, which is what a young program's errors look like.
The prose can be wrong. Issues are drafted with AI assistance and edited and signed by a person, who answers for them. That is accountability, not infallibility.
So: verify before you implement anything. Every figure names the file it was computed from and the conditions it was measured under; every issue publishes the SHA-256 of its own bytes so you can confirm you are reading what was issued. Use those. If you find an error, write to s@bud.day — every report is answered, and a confirmed error is issued in the register under its own date, because a program that hides its mistakes has no standing to measure anyone else's.
Use
Issues, the figure registry and the public half of the bank may be quoted, cited and redistributed with attribution — the issue’s name and URL, as shown under how to cite — provided a rate is never separated from its interval and its conditions, and provided no issued document is altered. Protocol STAMP text itself is released under the MIT License at github.com/buddaybudderson/stamp. Nothing on this site grants any right to a model, product or company name.
The site is provided as is. The publisher is Budday Budderson Studio LLC, a New Mexico company; Protocol STAMP, Jester Mind, and bud.day are under it. Its postal address is 2105 Vista Oeste NW, Suite #E3993, Albuquerque, NM 87120. The work is done wherever the investigator is; each issue carries its own dateline. Contact: s@bud.day.