Protocol STAMP is teamed with AI and can make mistakes. Verify before you implement anything. What we got wrong · Free to read, paid for by readers: Contribute
On the questions it was written against, mostly yes: in all ten configurations of our first round and on most rows of the study that followed, by an amount that depends on who does the grading. On questions it was not written against, we cannot tell yet. And its receipt, the part meant to make it checkable, was wrong on about a quarter of the replies in our five-model run.
On Aug 08, 2026 we ran STAMP v1.25 on five models through a free gateway: gpt-oss-120b, GLM-4.7, DeepSeek V4 Flash, Step 3.7 Flash and Nemotron 3 Super. Each probe was asked once without STAMP and once with it, and an AI grader judged each reply against the probe’s own pass criterion.
These models return their reasoning along with their answer, and the evaluation tool we use puts that reasoning in front of the reply unless it is told not to. It was not told. So every one of the 930 replies began with “Thinking: …”, and the grader read each model’s working as if it were part of the answer. Our report said these replies carried a draft receipt. Until Oct 15, 2026 it did not say that the verdicts had been given with the reasoning in view.
On Oct 14, 2026 and Oct 15, 2026 we separated each answer from its reasoning, without changing the published file, and graded the answers again with the same rubric. We used two graders from developers outside the five, Gemini 3.7 Flash and Claude Haiku 5.5, and had each grade every reply twice: once as the first grader saw it, and once as the answer alone. Holding the grader fixed that way isolates what the reasoning did. The re-grade cost $3.05.
STAMP minus native fail rate, in percentage points, with 95% intervals over 82 probes. A negative number means STAMP lowered the fail rate.
| Model | As first published | Gemini 3.7 Flash, answer alone | Claude Haiku 5.5, answer alone |
|---|---|---|---|
| gpt-oss-120b | -22.0 (-31.7 to -12.2) | -13.4 (-23.2 to -3.7) | -19.5 (-30.5 to -9.8) |
| GLM-4.7 | -17.1 (-28.0 to -6.1) | -18.3 (-29.3 to -8.5) | -23.2 (-34.1 to -12.2) |
| DeepSeek V4 Flash | -12.2 (-22.0 to -2.4) | -6.1 (-15.9 to 3.7) | -15.9 (-28.0 to -3.7) |
| Step 3.7 Flash | -11.0 (-20.7 to -1.2) | -8.5 (-18.3 to 0.0) | -20.7 (-30.5 to -11.0) |
| Nemotron 3 Super | -9.8 (-20.7 to 1.2) | -7.3 (-17.1 to 2.4) | -13.6 (-24.7 to -2.5) |
The report’s table now carries the Gemini column, because the first grader was also a Gemini model. Under it, the interval for three of the five models includes or reaches zero, against one of five as first published. Under Haiku, none does. Both readings are true of the same answers, which is the most useful thing this exercise taught us: the grader is part of the measurement, and a single grader’s number is one view of the effect, not the effect.
With the grader held fixed, taking the reasoning away shrank STAMP’s effect, pooled over the five models, by 3.4 points with Gemini (95% CI -0.2 to 6.8) and by 2.0 points with Haiku (-2.2 to 6.4). Both intervals include zero, so we read this as a lean, not a finding. With Gemini, 28 STAMP passes became fails against 14 native ones. One possible reading, which we have not tested: STAMP asks a model to check its work, the reasoning shows the checking, and a grader that reads it gives credit for the checking rather than the answer.
One failure was not subtle. In 62 scored replies the model spent its whole 1,024-token budget reasoning and returned no answer at all. The original grader passed 18 of them on the strength of the reasoning. Shown the empty answer, the new graders passed 2 and 1.
Every result above comes from probes the protocol was written against. The project’s rule is that every real failure becomes a new probe and the protocol is revised until it passes; that is good engineering and bad measurement, because it fits the test to the protocol. On a bank STAMP was not tuned on, the single controlled estimate was -0.9 points, and the grader that produced it was measured as 4.1 points lenient on its own.
The study we pre-registered and ran on Oct 14, 2026 switched reasoning off and on within the same models. STAMP lowered the fail rate clearly on five of its ten rows on tier-1 verdicts, including gpt-5.6-luna with reasoning off (-9.8 points, -16.5 to -3.7). With reasoning off, Nemotron 3.5 Lightning did no better with STAMP than without it (+0.6 points, -8.5 to 9.8), and 65.2% of its STAMP receipts failed. Reasoning did not shrink the effect, against our prediction. None of its rows meets our own registered rule for reporting an effect, because the judging panel that rule requires has not sat.
STAMP’s distinctive part is the receipt: a last line in which the model reports what it checked, in a form a program can test without a judge. In the five-model run, checked on the answer alone, 22.0% to 31.7% of STAMP receipts failed at least one rule (95% intervals within 14.4% and 42.4%). Before the correction, the report said 3.6% to 13.1%. About half of the failures, 55 of 107, were replies that hit the token limit before writing a receipt at all. STAMP asks for more work per reply, and a model with a tight budget can run out before it finishes. That is a cost of the protocol, and a deployment would have to allow for it.
On the questions it was built around, STAMP made answers right more often in all ten configurations of the first round, under every grader we used, and on most rows of the study that followed; with reasoning off, one model gained nothing. That is worth something: the project turns each failure it meets into a probe, such as a wrong total repeated into a board summary or a claim agreed with on demand. It is not yet evidence that STAMP makes a model more accurate in general, and the size of the gain moves by up to 12 points depending on which AI does the grading.
What would settle it: probes the protocol was not written against, graded by a panel from several model families with a human anchor, and the receipt-stripped regrade that measures whether the receipt itself talks the grader into a pass. Those are the next measurements.
The published run file is unchanged; the scripts that strip the reasoning and re-grade it, every verdict with the grader’s stated reason and cost, and the commands that rebuild the table are in the re-grade files. The full account, with its limits, is Section 6 of the technical report, and the correction is dated in Corrections + changes. If you find a number here that does not match its source, write to s@bud.day and the correction will be dated.