When to Trust a Decision Model

Part two of a series. Part one: A Model That Refuses to Talk.

I asked a model a question you can check on the back of an envelope. It told me it was 100% sure of the wrong answer.

Here is the question. A yearly service credit of 1,098 is prorated by active days in a 366-day leap year. The service ran February 28 to March 2 inclusive. A suspension removes February 29. How many credits are earned? Try it before you scroll.

Four calendar days, minus the suspended leap day, leaves three. The yearly credit works out to 3 credits a day. So the answer is 9. The trap is 12, which is what you get if you forget the suspension.

That item is real. Its answer key was written by an AI model, GPT-5.6 Sol, and so was every other key in Part one.

On 15 of the 111 hard questions, the model that wrote out its own probabilities was wrong at 90% or higher on exactly the planted trap. On this one, both systems that got it right were unsure, and the one with no hesitation was wrong.

So this post runs one test on nine systems, first on 111 hard questions with AI-written keys, then on 1,599 questions labelled by 100 people each: hosted Jev, a Qwen 27B I run myself read two ways, and five open-weight models built in Jev’s style, on one GPU. Post one said the plain 27B was calibrated at least as well as Jev. That held on two easy datasets and does not hold here.

The short version

Read the last two columns against the first. The full nine-system table on the human key is in Part one; what the checks changed is Part two.

Part one: the models

The one concept: a confidence gate

A gate is the plainest thing you can do with a probability: accept the answers above some bar automatically, send the rest to a person or a slower model. So the test of a probability is this: keep only the answers where the system said it was at least 90% sure. How many did you keep, and how many of those were right? A system whose wrong answers also sit at 90% has a gate that passes the bad through with the good. So the rule this post scores everything by: a system earns trust when the answers it keeps are much more accurate than its answers overall.

Where you set the bar depends on what a wrong answer costs you; auto-tagging a ticket and auto-issuing a refund do not deserve the same bar. The 90% throughout this post is a reporting convention, not a recommendation.

Drag the bar and watch who keeps what.

The familiar calibration score, ECE, asks whether a system that says 80% is right about 80% of the time. It is easy to flatter: Eve scored a tidy 0.056 on my second test set while getting 47.9% right, below the 52% from always guessing the most common answer for each question type. The gate is harder to flatter, as long as you also report how many answers it kept.

Same model, two ways to read its confidence

This is the cleanest result I have. One Qwen 27B, same weights, same server, same 111 hard questions, two ways of asking how sure it was. Asked to write its probabilities out as JSON, it held most of its wrong answers at high confidence. Read from its own token probabilities, which I will call reading from its probabilities, it held almost none there. One readout gave me a gate I could use. The other did not. The prompt and output format differ between the two as well, so this compares two pipelines, not one switch.

One caveat first. The probability I read is conditional on the option letters I captured, and on 21 of the 111 questions less than 90% of the model’s probability sat on any letter at all.

Option order matters to Qwen in a way accuracy hides. Listing the options in reverse changed its top answer on 36 of the 105 reorderable questions when it wrote its probabilities out, and on 26 of 105 read through them, while accuracy moved only 2 and 7 points, because flips went both ways. Jev is not deterministic, and reversing its options changed its answer (8 of 105) about as often as asking twice did (7 of 105), so I found no order effect in Jev beyond its own noise. One run per layout, and I do not know the reason.

This is not a discovery. Kadavath and colleagues in 2022 and Xiong and colleagues in 2023 point the same way, and Tian and colleagues in 2023, whom I cited in post one, found the opposite on TriviaQA, SciQ and TruthfulQA.

So the claim is local: on these items, with this model, on my server. It held again when 100 people wrote the key, by a rule I set before running it; see Scored by 100 people.

Five open models: what each earned

Post one promised this run: the open-weight models built in this style, on a single GH200 I run myself. Each of them is good at something I measured, and each has a limit I measured in the same session. Both belong in the same paragraph.

Decider-2b (Mark Marosi) is the fastest single answer here: 5 ms for one question, 166 ms for fifty. On the human-labelled set its gate is respectable, 593 answers kept at 90% and 75% of them right, though it trained on that set’s source. On the hard AI-keyed questions it is 45% right and holds 20 of its 61 wrong answers at 90%, so there it is a fast answer, not a gate.

Eve RLCD (Anthony Maio) is the model whose confidence you can act on. It is trained with a reward that pays for honest probabilities, a proper scoring rule, and it behaves that way: on the hard questions it holds 7 of 73 wrong answers at 90%, on human labels 12 of 673, and it sits closest of the nine to the human label distribution. The cost is coverage. It is right on only 34% of the hard questions and keeps 115 of 1,599 human-labelled answers at 90%. When it is sure, believe it; it is rarely sure.

Laya (Nandakishor, Convai Innovations) has the flattest latency line I measured, 119 ms for one question and 113 for fifty, and its author’s published figures reproduced on my machine. Its gate does little: on the hard questions it is 34% right and keeps only 2 answers at 90%; on human labels it keeps 890 at 66% right, five points above just answering everything. It reads the first 512 tokens by design.

Kev (Jared Palmer) speaks the hosted service’s API, so my harness ran against it unchanged, and its accuracy climbs with size: 32%, 48%, 57% on the hard questions at 0.8B, 4B and 9B, and 56%, 62%, 66% on human labels, where the 9B is the most accurate system in this post (it trained on that set’s source). What does not climb is its honesty about being wrong. The share of its wrong hard answers held at 90% is 24 of 75, 31 of 58, 22 of 48; at 9B nearly half of what it got wrong it was sure of, and on human labels its gate adds about 3 points over just answering, at every size. Bigger got righter.

JevBench (Florian Standhartinger) is the benchmark all of this runs on.

So the two failure modes are different, and a small model can have either. Kev and Decider on hard questions are often wrong and often sure. Eve and Laya are often wrong and almost never sure. “Small models are overconfident” is false for two of the five.

Some of these numbers may differ from what the authors would get: Kev ran on a newer torch than its author pins, and Decider’s accuracy run had its CUDA graphs off to fit beside my own servers.

Scored by 100 people, the gap held

Everything so far is agreement with an answer key that an AI model argued its way to. So before publishing I ran the whole table against people: ChaosNLI (Nie, Zhou and Bansal, 2020), 1,599 premise-and-hypothesis pairs, each labelled by 100 people, asking whether the hypothesis follows from the premise, contradicts it, or neither. Its authors chose items people disagreed on, so only 57 of the 1,599 have a 90% human majority. That makes it a hard set for any gate, which is the point. “Right” means matching the human majority.

I committed the rule before any system saw an item (commit 3dfff8c, 19:15 Central, September 21): the finding holds only if the whole 95% interval for the gate gap between the two readouts sits above 5 points. Then I ran all nine systems on all 1,599. Read the third column against the first: a gate that beats its own overall accuracy is doing something.

Kev and Decider list MNLI among their training sets, and these items come from MNLI’s development split, so their rows may reflect memory. Laya’s page does not say; Eve’s author says it never trained on MNLI.

Asked to write down its confidence, Qwen kept 89% of its answers and gained nothing by it: the kept set was exactly as accurate as answering every question. The probability reading kept 40% and was 13 points better on what it kept. The paired interval for that gap is +10.2 to +15.8 points, so by the rule I set, the finding stands on human labels.

Then the check I had promised myself: options in reverse. The gap shrank to +3.5 to +8.8 points, inconclusive under my rule, mostly because the written-out version held 310 fewer answers at 90% and got better at what it kept. So the honest sentence is: reading the probabilities gave the better gate under both layouts, and the size of the advantage depends on how the options are listed.

Jev held a third of its wrong answers at 90% or more here, against 2 of 31 on the AI-keyed questions. On items where 100 people split, though, the median human share on those confident wrong answers was 38%, so “wrong” is a harder judgment than the table makes it look.

Against the full human distribution rather than the majority, Eve, Decider, the probability reading and Jev were closest, within 0.02 of each other, and the written-out version and the Kevs were furthest. Whether Qwen, Jev or Laya have seen MNLI, I cannot say.

If you gate on a written-out confidence today, this is the experiment to run this week: same model, both readouts, two hundred of your own items.

Latency: how many questions per call

Jev and Laya answer fifty questions about as fast as they answer one. Nothing else here does. My own inference servers were down for about 23 minutes so each local model could be measured alone on the GPU (Jev is hosted; the one-structured-request Qwen row was measured separately with my servers running). Median response time is in the chart, in milliseconds.

For a builder the useful reading is the slope. Everything but Jev and Laya grows from one question to fifty, from Eve’s 40% to Decider’s thirty-fold, and the general LLM most in absolute terms. If one input carries twenty questions, a flat system is a different component from one that grows with every question. Tails matter too: at fifty questions Decider’s 90th percentile opens to 749 ms. Medians are what people quote; tails are what page you at night.

These are timings of these deployments on a synthetic workload. Loopback against a hosted API across the internet is not like for like, and I cannot subtract my network time to recover anyone’s compute time. My general LLM ran at its production setting of four concurrent sequences. One oddity I have not explained: Kev-9b answered a single question faster than Kev-4b. Rerun that before leaning on it.

Part two: the checks

Everything above is about when a number can be trusted. Here is when mine could not.

Three checks that changed the numbers

Prompts. My wrapper for Eve dropped the fine print on its yes/no questions. Fixed and rerun, Eve’s accuracy landed within two questions on each benchmark tier and a point lower on the second set, but its confidence changed: it was never 90% sure on a hard question, against 11 times before, 7 of them wrong. I report the first run throughout. The option-order control moved Qwen’s probability reading by 7 points, as large as its apparent lead over Jev, so that lead is not reported as one. And on my second test set, asking yes or no questions as lettered options instead of the words yes and no cost 16 points on the 600 yes or no decisions, 61% against 77%.

Instruments. The harness I started with opens a new connection for every request, which charged the hosted API a fresh handshake each time and made it look about 140 ms slower than it is; every timing here uses a connection that stays open. My first latency loop sent the same input on every call, which flatters anything with a cache: one model read 63 ms that way and 100 ms once I varied the input. Both errors produced the number I was expecting to see, which is why they lasted.

Statistics. Jev and the Qwen probability reading differ on 25 of 111 hard questions, 9 one way and 16 the other, p = 0.23. That is inconclusive, not a tie and not a win. And the hard answer keys were written by two models, 57 by GPT-5.6 Sol and 54 by Claude Opus 5, which I only knew after reading the whole provenance field rather than one row of it.

Two other AI systems checked the work before publication: Codex, which recomputed every number from the raw files, and Grok, with live web search. They found all of the above. They were wrong too: two of three checkable claims from one of them failed when I checked, a temperature flag that does not exist at that commit and a floor figure from a different definition than mine. A hostile review is a list of leads, not a verdict.

Yes. An AI reviewed an AI’s evaluation of AI models answering questions written by AI. I am not going to defend that. It is why the nine-system table in Part one is scored by people, and why I dropped my plan to label the 111 hard questions myself: a hundred people per item beat one tired author.

The three kinds have one thing in common: every one of these errors made the story simpler than the data did, and not one made it more complicated. I do not think that is chance. It is what motivated reasoning looks like from the inside, where it does not feel like reasoning at all. It feels like being finished.

Of 127 numbers in my methods write-up recomputed independently, 2 were wrong, and both are fixed. The arithmetic held. The interpretations were where the errors lived.

The two review prompts, a numbers audit and a methods review run separately, are generalized and available on request.

Check me. The test items are public (JevBench, ChaosNLI). My scripts, per-item results, raw timings, the pre-registration with its commit timestamp, the three reviews and their prompts are available on request.

Credits and licenses

JevBench, by Florian Standhartinger, MIT. The LocalLLaMA/typed-decisions set, Apache 2.0, whose labels are a teacher model’s rather than a person’s. Kev, by Jared Palmer, Apache 2.0. Decider, by Mark Marosi, Apache 2.0. Eve RLCD, by Anthony Maio, MIT code and Apache 2.0 weights. Laya, by Nandakishor at Convai Innovations, Apache 2.0. Qwen3.8-27B, Apache 2.0. ChaosNLI, by Yixin Nie, Xiang Zhou and Mohit Bansal, CC BY-NC 4.0, the only human labels here. Jev, by TypeSafe, hosted, early access. Thank you.

Found a mistake? Tell me on LinkedIn or X. I will fix it here, dated, with credit to whoever caught it.

If you have human-labelled decisions from real work, not a benchmark, I would like to run this on them.