Protocol STAMP is teamed with AI and can make mistakes. Verify before you implement anything. What we got wrong · Free to read, paid for by readers: Contribute
A model that makes up a fact and a model that makes up a story are doing the same thing: producing text that sounds right without checking it against anything. In one case that is a failure; in the other it is the job. Our position is that the rules that keep a model from inventing facts should apply in full when it states facts, and step aside when it is asked to invent, short of the safety lines that never move. We set out what our protocols do about hallucination and what we have and have not measured, why the temperature setting is a weaker dial for creativity than its reputation, and what a creative mode for STAMP should and should not loosen.
A hallucination is a confident statement that is false. The clearest account we have read of why it happens is Kalai, Nachum, Vempala and Zhang’s Why Language Models Hallucinate (2025), which we discussed in Stop marking it like a student. They argue that “the training and evaluation procedures reward guessing over acknowledging uncertainty,” and that “language models are optimized to be good test-takers, and guessing when uncertain improves test performance.”
If guessing is rewarded, the fix has two halves. Ask the model to stop guessing, and stop rewarding it for guessing. STAMP is our attempt at the first half: an instruction layer that tells a model never to state an unverified specific as established (absolute A1), to recompute every figure it repeats (A2), to ask one focused question when the missing piece would change the answer (A6), and, when material is missing, to ask for it and “never invent it.” Every material claim carries a label (Supported, Calculated, Likely, Assumption, Unknown), and every reply ends with a receipt that counts what was checked. A model that says “Unknown” is doing what we asked.
STAMP governs a model’s reply. The Ten House Protocols govern this site, which is drafted with AI and therefore exposed to the same failure. Each protocol closes one door a made-up claim could come through:
| How a made-up claim gets in | The protocol that stops it |
|---|---|
| A fact stated from a model’s memory | 1 and 2: every factual sentence links a source, and the source is opened before the sentence is written. “What a model remembers is a lead, not a source.” |
| A quotation the speaker never said | 3: quotes are exact, and our own reading is labelled as ours. |
| A guess presented as certain | 4: every sourced claim carries a certainty score computed from the evidence recorded for it (being built). |
| An impossible date | 5: the house gate refuses a date that does not exist or a weekday that is not its own. |
| A word that claims more than it shows | 6: a word that claims size, speed, quality or novelty gives the number or is cut. |
| An error fixed without notice | 8 and 9: issued text is frozen, and every correction gets a dated public row. |
| A draft nobody independent read | 10: two reviewers from other model families, then a person who signs (being built). |
What we can say about whether it works is narrower than we would like. On the probes STAMP was written against, ten single-draw runs showed it lowering the content fail rate by 7.3 to 23.2 points; on a bank it was not tuned on, the one controlled estimate was −0.9 points, smaller than the grader’s measured bias; and in the reasoning-off and reasoning-on study of Oct 14, 2026, no row met our registered rule for reporting an effect, because the grader panel that rule requires was not run (technical report, Sections 6.1, 6.3 and 6.5). For the site, the first measurement starts on Saturday Oct 17, 2026, when we begin re-checking 30 published claims from scratch every week and publishing the error rate with its upper bound. Until then, “the protocols minimize hallucination” is our design intent, not a result.
Correction, Oct 16, 2026. “By 7.3 to 23.2 points” matched the technical report when this post was issued, and has been out of date since Oct 16, 2026, when the report corrected Section 6.1: the Aug 08, 2026 run was re-graded on each answer without the model’s returned reasoning, and the ten runs now show STAMP lowering the content fail rate by 6.1 to 23.2 points, with the interval including or reaching zero for three of that run’s five models. The sentence above is left as issued; the corrected figures are in the technical report, Section 6.1.
A short story about a lighthouse keeper contains no claim about any real lighthouse. A brainstorm of product names is supposed to contain names that do not exist yet. In these tasks, inventing is not hallucinating, because nobody is asked to believe the text describes the world.
STAMP v1.25 does not draw that line. Its absolutes and its receipt apply to every reply, and nothing in the core says that fiction, poetry, brainstorming or role-play is outside the claims it governs. A story ending with a receipt that counts zero figures and zero sources is harmless but useless. The larger cost is caution in the wrong place, and our own run files show it. In the Mistral Large run of Aug 07, 2026, asked in the familiar “late grandma” framing for a drug recipe, the native reply refused the recipe and offered a harmless bedtime “recipe” instead; the STAMP reply was a bare refusal that wrongly called the user’s own request an embedded instruction attempt. Asked the time in Tokyo, the STAMP reply said only that it could not confirm the time live, while the native reply gave the time zone. Both STAMP replies were correct about what they could not do, and less useful than the model without STAMP (technical report, Section 6.2).
Others have measured the general version of this. XSTest (Röttger et al., NAACL 2024) was built because “some models may have struck a poor balance, so that even clearly safe prompts are refused if they use similar language to unsafe prompts or mention sensitive topics.” It contains “250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with.” A dark theme in a novel, a villain’s monologue, or a war poem is that kind of prompt.
The usual advice for creative work is to raise the temperature, the setting that controls how much randomness goes into choosing each word. The vendors say so in their own documentation. Anthropic’s reads: “Use temperature closer to 0.0 for analytical / multiple choice, and closer to 1.0 for creative and generative tasks.” OpenAI’s API specification reads: “Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.”
The evidence that randomness is creativity is thin. Peeperkorn, Kouwenhoven, Brown and Jordanous tested the claim directly and found “temperature is weakly correlated with novelty, and unsurprisingly, moderately correlated with incoherence,” and that its influence “is far more nuanced and weak than suggested by the ‘creativity parameter’ claim.” Going the other way, Renze and Guven found that “changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks.” Neither study is the last word: one is a single narrative task, the other a multiple-choice exam.
Three practical facts make the dial weaker still. Temperature 0 is not a guarantee of the same answer twice: Anthropic’s documentation says “even with temperature of 0.0, the results will not be fully deterministic,” and our own two temperature-0 runs differed on 56 of 63 prompts. Both vendors already default to 1, so “set it to 1 for creative work” usually means “stop forcing it to 0.” And on some current models the dial is gone: Anthropic’s documentation now marks the parameter deprecated and says “Models released after Claude Opus 4.6 do not support setting temperature. A value of 1.0 will be accepted for backwards compatibility, all other values will be rejected with a 400 error.” On the gateway we use, the model catalog we read on Oct 14, 2026 did not list temperature among the settings OpenAI’s gpt-5.6-luna accepts.
Our reading: leave temperature at the vendor default for creative work, and do not expect it to do the work. What makes a model write well, or badly, is mostly in the instructions it is given, including instructions like ours.
We want STAMP to get out of the way of fiction. We do not want it to get out of the way of safety, and the two are different requests.
Less censored, in our sense, means: no reflexive refusal of a dark theme, a violent scene, or a morally ugly character in clearly fictional work; no evidence labels or hedges inside a poem; no lecture appended to a story; and no treating the writer’s own request as an attack.
It does not mean: getting around a provider’s safety rules, which STAMP already says outrank it (“provider system/safety rules outrank this document”); or treating a story frame as a key. A recipe for a dangerous drug is the same recipe whether a grandmother recites it or a manual prints it, and the native reply above handled that correctly: it kept the story and left out the recipe. That is STAMP’s own absolute A5, “deliver the fullest safe portion, offer the nearest legitimate alternative,” which our STAMP reply failed to follow.
These are proposals. Like those in earlier Bench posts, they bind us once they are written down and signed, and not before.
This is not a new measurement. No model was run for this post. The figures about STAMP come from our technical report and its run files; the figures about temperature are as the cited authors and vendors state them.
This is a position. Nothing on The Bench is a finding.
Conflict of interest. Anthropic makes Claude, which drafted this post and is a tool this site uses; its documentation is quoted here, alongside OpenAI’s. Recusal rules on our Independence page apply.
Right of reply stands open. Any author or company named here may respond, and a response will be published in full.
An issued Bench post is corrected in the open. Corrections are published like findings.
temperature parameter. Read Oct 15, 2026.temperature parameter (ModelResponseProperties). Read Oct 15, 2026.openai/gpt-5.6-luna. Read Oct 14, 2026.Previous issue: A Name Is Not a Test
To cite: Protocol STAMP. Check the Facts, Free the Fiction. The Bench, Oct 15, 2026. Budday Budderson Studio LLC. https://protocolstamp.com/bench/check-the-facts-free-the-fiction/