The Bench

Asked to summarize a book none of us had read, we searched only what is public, checked reviewers’ quotations against a search of the first edition, and labelled every claim as in the book or about it. We should have asked for the book first; now the rule says so.

What every measurement so far says, after we re-graded our five-model run without the models’ reasoning: fail rates fell on the questions STAMP was written against, by an amount that depends on the grader; on other questions we cannot tell yet; and the receipt failed on about a quarter of replies.

Our house protocols are built to keep a model from inventing facts; creative work asks it to invent. What the protocols do about hallucination and what we have and have not measured, why temperature is a weak dial for creativity, and what a creative mode for STAMP should loosen and what it never should.

Every grant program we plan to apply to, the amount we will ask each for and why, and the ones we will not apply to, published before any application is filed.

The US renamed AI ‘Super Intelligence’ without changing what the word covers, and the UN talks on autonomous weapons ended without a treaty. Both turn on one question: who checks what an AI is said to be? Why we still write “AI,” where military AI rules reach testing, and what we propose, starting with a public-test line on every Docket entry.

Our three judges agreed with each other at 0.875 and with one person at 0.631 to 0.685. Nine published studies of AI judges, 2023 to 2026, bear on that and none measured it: what each found, where it reaches our work, and what we propose to take from it, starting with printing four numbers for every judge instead of one.

Two decision models went through the seven rules of Five Cases, a Hundred Times Each. The agreement bar was met, to three decimal places, by a judge that only ever says PASS, and the rule that asked a person is what stopped them. A proposal for how a new kind of model, Jev or whatever comes next, should be tested here: seven steps, starting with the judge that always says yes.

LangChain reported that Jev, a new decision model, matched a human reviewer on all 500 pass/fail decisions it judged. The 500 were five cases repeated a hundred times. What that can carry, the open models that followed, and the seven rules any judge must pass before it grades our work. Due Sep 14, 2026; published seventeen days late, and the post says why.

Four Claude models were handed a record with two errors planted in it, twice — once with web search off, once with it on. All twelve cells caught both. Retrieval changed nothing about that, and everything about what could honestly be said afterwards. Run record · data.

Is hallucination really what people hate most about AI? The review data says yes. Then thirteen AI models were asked the same question, word for word, and the answer that came back was that our own table was wrong. It was, nine times over. Run record · PDF · source.

A model guesses when your question looks like it wants help, and hedges when it looks like it might cause harm. Neither behavior is about what the model knows. Both are about how it is marked.

Most prompt hygiene guards against invention and stops there. The case for treating abstention as a symmetric failure — and what it costs to guard only one side.