---
title: "When to Trust a Decision Model"
description: "A confidence number is only worth what it lets you trust. I put nine decision models, open and hosted, through one test: keep only the answers each was 90% sure of, and score them, first on 111 hard AI-written questions and then on 1,599 questions labelled by 100 people each. Some earned the trust. Here is which, and under what conditions."
author: "Colin McNamara"
tags: ["Decision Models","Calibration","Evaluation","AI Architecture","Decision Models, Measured"]
image: "https://colinmcnamara.com/images/blog/when-to-trust-a-decision-model/fig0-one-question-five-answers.png"
date: 2026-09-23
url: https://colinmcnamara.com/blog/when-to-trust-a-decision-model
---

# When to Trust a Decision Model

Part two of a series. Part one: [A Model That Refuses to Talk](https://colinmcnamara.com/blog/jev-system-one-decision-models).

Bottom line up front

A decision model hands you an answer and a confidence number. The question that matters: can you auto-accept the answers it calls 90% sure and send the rest to a person? I tested nine models that way. What to do with each:

- **DO NOT USE** Qwen 27B, asked to write its confidence into its JSON answer. Do not auto-accept on that number. Asked to write its probabilities out, the model was just as sure of its mistakes as of its right answers. On 1,599 human-labelled questions, auto-accepting its 90% answers was no better than accepting all of them. [23 of 30 wrong answers claimed 90% or more.](#same-model-two-ways-to-read-its-confidence)

- **USE** Qwen 27B, its confidence read from its own token probabilities. Same model, same weights, same questions. If you host the model, read the probability it puts on each option letter instead of asking it to write a number. The answers it was sure of were worth auto-accepting. [4 of 24 wrong answers claimed 90%; 13 points more accurate on human labels.](#scored-by-100-people-the-gap-held)

- **USE** Jev, TypeSafe's hosted decision API. Safe to auto-accept at 90% on both answer keys, and it needs no GPU. Know two things: the same request can return a different answer, and on human-labelled data a third of its wrong answers still claimed 90%. [96% of its 90% answers were right on the hard questions.](#scored-by-100-people-the-gap-held)

- **USE, NARROWLY** Eve RLCD, a 0.6B open-weight model. When it says 90%, believe it. It almost never says 90%. Use it to auto-accept a thin slice and send the rest on. [12 of 673 wrong answers claimed 90%, but it reached 90% on only 115 of 1,599 questions.](#five-open-models-what-each-earned)

- **REVIEW ALL** Kev-9b and Decider-2b, open-weight models. Use them for the answer, not the confidence. On hard questions Kev-9b claimed 90% on nearly half of its wrong answers and Decider on a third, so send everything to review. Both trained on the human set's source. [Kev-9b was the most accurate system on human labels; Decider answered in 5 ms.](#five-open-models-what-each-earned)

- **TEST FIRST** Your own data, before you trust any number above. Before you trust a gap between two models, send the identical request twice and then reverse the option order. If the gap survives, believe it. [Reversing the options moved one model 7 points and changed a quarter of its answers.](#three-checks-that-changed-the-numbers)

Every number was recomputed from the per-item files by a second AI, and every model was rescored against 100 people per question. Each number links to its evidence. The checklist for running this on your own data is at the end.

I asked a model a question you can check on the back of an envelope. It told me it was 100% sure of the wrong answer.

Here is the question. A yearly service credit of 1,098 is prorated by active days in a 366-day leap year. The service ran February 28 to March 2 inclusive. A suspension removes February 29. How many credits are earned? Try it before you scroll.

*Figure: Card showing the leap day credit question and five systems answering it. Qwen 27B writing its own probabilities picked the trap, 12, at 1.00. Read through its token probabilities it picked 12 at 0.82. Kev-9b picked 12 at 0.59. Jev picked the right answer, 9, at 0.55. Decider-2b picked 9 at 0.41.*

Four calendar days, minus the suspended leap day, leaves three. The yearly credit works out to 3 credits a day. So the answer is 9. The trap is 12, which is what you get if you forget the suspension.

That item is real. Its answer key was written by an AI model, GPT-5.6 Sol, and so was every other key in Part one.

On 15 of the 111 hard questions, the model that wrote out its own probabilities was wrong at 90% or higher on exactly the planted trap. On this one, both systems that got it right were unsure, and the one with no hesitation was wrong.

So this post runs one test on nine systems, first on 111 hard questions with AI-written keys, then on 1,599 questions labelled by 100 people each: hosted Jev, a Qwen 27B I run myself read two ways, and five open-weight models built in Jev's style, on one GPU. Post one said the plain 27B was calibrated at least as well as Jev. That held on two easy datasets and does not hold here.

## The short version

| | Right on 111 hard items, AI key | Wrong answers it was 90%+ sure of | Trusting only answers at 90%+ | Same gate on 1,599 items, human key |
|---|---|---|---|---|
| Jev (hosted) | 72% | 2 of 31 | keeps 45, 96% right | keeps 730, 72% right |
| Qwen 27B, read through its probabilities | 71 to 78% by prompt layout | 3 to 5 | keeps 49 to 57, 90 to 94% right | keeps 628, 76% right |
| Same Qwen, writing its own probabilities | 73% | 23 of 30 | keeps 85, 73% right | keeps 1,426, 63% right |
| Kev-9b (small open model, trained on the human set's source) | 57% | 22 of 48 | keeps 72, 69% right | keeps 1,355, 69% right |

Read the last two columns against the first. The full nine-system table on the human key is in [Part one](#scored-by-100-people-the-gap-held); what the checks changed is [Part two](#part-two-the-checks).

## Part one: the models

### The one concept: a confidence gate

A gate is the plainest thing you can do with a probability: accept the answers above some bar automatically, send the rest to a person or a slower model. So the test of a probability is this: keep only the answers where the system said it was at least 90% sure. How many did you keep, and how many of those were right? A system whose wrong answers also sit at 90% has a gate that passes the bad through with the good. So the rule this post scores everything by: a system earns trust when the answers it keeps are much more accurate than its answers overall.

Where you set the bar depends on what a wrong answer costs you; auto-tagging a ticket and auto-issuing a refund do not deserve the same bar. The 90% throughout this post is a reporting convention, not a recommendation.

Drag the bar and watch who keeps what.

*Figure: Only trust it when it says it is at least this sure. A slider sets the confidence bar and two bars update per system: how many of 111 hard questions are kept, and how accurate the kept set is. At 90 percent Jev keeps 45 at 96 percent right, the probability reading keeps 57 at 93 percent, and written-out Qwen keeps 85 at 73 percent.*

The familiar calibration score, ECE, asks whether a system that says 80% is right about 80% of the time. It is easy to flatter: Eve scored a tidy 0.056 on my second test set while getting 47.9% right, below the 52% from always guessing the most common answer for each question type. The gate is harder to flatter, as long as you also report how many answers it kept.

### Same model, two ways to read its confidence

This is the cleanest result I have. One Qwen 27B, same weights, same server, same 111 hard questions, two ways of asking how sure it was. Asked to write its probabilities out as JSON, it held most of its wrong answers at high confidence. Read from its own token probabilities, which I will call reading from its probabilities, it held almost none there. One readout gave me a gate I could use. The other did not. The prompt and output format differ between the two as well, so this compares two pipelines, not one switch.

*Figure: Strip plot of top probability on wrong answers only, one row per system. Qwen writing its own probabilities piles up at the right edge, 23 of 30 wrong answers at 90 percent or higher. Jev shows 2 of 31 and the probability reading 4 of 24. The three Kev rows climb in accuracy, 32, 48 and 57 percent.*

One caveat first. The probability I read is conditional on the option letters I captured, and on 21 of the 111 questions less than 90% of the model's probability sat on any letter at all.

Option order matters to Qwen in a way accuracy hides. Listing the options in reverse changed its top answer on 36 of the 105 reorderable questions when it wrote its probabilities out, and on 26 of 105 read through them, while accuracy moved only 2 and 7 points, because flips went both ways. Jev is not deterministic, and reversing its options changed its answer (8 of 105) about as often as asking twice did (7 of 105), so I found no order effect in Jev beyond its own noise. One run per layout, and I do not know the reason.

This is not a discovery. [Kadavath and colleagues](https://arxiv.org/abs/2207.05221) in 2022 and [Xiong and colleagues](https://arxiv.org/abs/2306.13063) in 2023 point the same way, and [Tian and colleagues](https://arxiv.org/abs/2305.14975) in 2023, whom I cited in post one, found the opposite on TriviaQA, SciQ and TruthfulQA.

So the claim is local: on these items, with this model, on my server. It held again when 100 people wrote the key, by a rule I set before running it; see [Scored by 100 people](#scored-by-100-people-the-gap-held).

### Five open models: what each earned

Post one promised this run: the open-weight models built in this style, on a single GH200 I run myself. Each of them is good at something I measured, and each has a limit I measured in the same session. Both belong in the same paragraph.

**[Decider-2b](https://github.com/Mapika/decider)** (Mark Marosi) is the fastest single answer here: 5 ms for one question, 166 ms for fifty. On the human-labelled set its gate is respectable, 593 answers kept at 90% and 75% of them right, though it trained on that set's source. On the hard AI-keyed questions it is 45% right and holds 20 of its 61 wrong answers at 90%, so there it is a fast answer, not a gate.

**[Eve RLCD](https://github.com/anthony-maio/eve-rlcd)** (Anthony Maio) is the model whose confidence you can act on. It is trained with a reward that pays for honest probabilities, a proper scoring rule, and it behaves that way: on the hard questions it holds 7 of 73 wrong answers at 90%, on human labels 12 of 673, and it sits closest of the nine to the human label distribution. The cost is coverage. It is right on only 34% of the hard questions and keeps 115 of 1,599 human-labelled answers at 90%. When it is sure, believe it; it is rarely sure.

**[Laya](https://github.com/NandhaKishorM/laya)** (Nandakishor, Convai Innovations) has the flattest latency line I measured, 119 ms for one question and 113 for fifty, and its author's published figures reproduced on my machine. Its gate does little: on the hard questions it is 34% right and keeps only 2 answers at 90%; on human labels it keeps 890 at 66% right, five points above just answering everything. It reads the first 512 tokens by design.

**[Kev](https://github.com/jaredpalmer/kev)** (Jared Palmer) speaks the hosted service's API, so my harness ran against it unchanged, and its accuracy climbs with size: 32%, 48%, 57% on the hard questions at 0.8B, 4B and 9B, and 56%, 62%, 66% on human labels, where the 9B is the most accurate system in this post (it trained on that set's source). What does not climb is its honesty about being wrong. The share of its wrong hard answers held at 90% is 24 of 75, 31 of 58, 22 of 48; at 9B nearly half of what it got wrong it was sure of, and on human labels its gate adds about 3 points over just answering, at every size. Bigger got righter.

**[JevBench](https://github.com/fstandhartinger/jevbench)** (Florian Standhartinger) is the benchmark all of this runs on.

So the two failure modes are different, and a small model can have either. Kev and Decider on hard questions are often wrong and often sure. Eve and Laya are often wrong and almost never sure. "Small models are overconfident" is false for two of the five.

**The specialist trap.** Laya ships a second checkpoint, fine-tuned on the second test set. On that set it scores 76.9%, above Jev's 73.5%. On the hard questions it scores 27.9%, below its own untuned sibling's 34.2%. Both of its author's published figures reproduced on my machine, 36.2% and 76.6% published against 36.1% and 76.9% measured, which is to the author's credit. But the benchmark's page reports Jev's score on that set and Laya's page reports this one, so the two are easy to read side by side as if comparable. The rule I took from it: never put a specialist and a generalist in one unlabelled column.

Some of these numbers may differ from what the authors would get: Kev ran on a newer torch than its author pins, and Decider's accuracy run had its CUDA graphs off to fit beside my own servers.

### Scored by 100 people, the gap held

Everything so far is agreement with an answer key that an AI model argued its way to. So before publishing I ran the whole table against people: [ChaosNLI](https://github.com/easonnie/ChaosNLI) (Nie, Zhou and Bansal, 2020), 1,599 premise-and-hypothesis pairs, each labelled by 100 people, asking whether the hypothesis follows from the premise, contradicts it, or neither. Its authors chose items people disagreed on, so only 57 of the 1,599 have a 90% human majority. That makes it a hard set for any gate, which is the point. "Right" means matching the human majority.

I committed the rule before any system saw an item (commit `3dfff8c`, 19:15 Central, September 21): the finding holds only if the whole 95% interval for the gate gap between the two readouts sits above 5 points. Then I ran all nine systems on all 1,599. Read the third column against the first: a gate that beats its own overall accuracy is doing something.

| | Right | Kept at 90%+ sure | Right among kept | Wrong answers held at 90%+ |
|---|---|---|---|---|
| Qwen 27B, writing its own probabilities | 63% | 1,426 of 1,599 | 63% | 524 of 596 |
| Same Qwen, read through its probabilities | 59% | 628 | 76% | 149 of 651 |
| Jev (hosted) | 62% | 730 | 72% | 206 of 614 |
| Kev-9b, trained on MNLI | 66% | 1,355 | 69% | 417 of 545 |
| Kev-4b, trained on MNLI | 62% | 1,317 | 65% | 467 of 608 |
| Kev-0.8b, trained on MNLI | 56% | 1,169 | 60% | 473 of 701 |
| Decider-2b, trained on MNLI | 62% | 593 | 75% | 146 of 605 |
| Eve RLCD | 58% | 115 | 90% | 12 of 673 |
| Laya, generalist | 61% | 890 | 66% | 304 of 624 |

Kev and Decider list MNLI among their training sets, and these items come from MNLI's development split, so their rows may reflect memory. Laya's page does not say; Eve's author says it never trained on MNLI.

*Figure: Two bars per system on the human-labelled set: the share of 1,599 answers held at 90 percent sure, and how many of those were right. Qwen writing its probabilities keeps 1,426 and is 63 percent right on them. Read through its probabilities it keeps 628 at 76 percent. Jev keeps 730 at 72 percent. Eve keeps 115 at 90 percent. The three Kevs keep 1,169 to 1,355 at 60 to 69 percent, Decider 593 at 75, Laya 890 at 66.*

Asked to write down its confidence, Qwen kept 89% of its answers and gained nothing by it: the kept set was exactly as accurate as answering every question. The probability reading kept 40% and was 13 points better on what it kept. The paired interval for that gap is +10.2 to +15.8 points, so by the rule I set, the finding stands on human labels.

Then the check I had promised myself: options in reverse. The gap shrank to +3.5 to +8.8 points, inconclusive under my rule, mostly because the written-out version held 310 fewer answers at 90% and got better at what it kept. So the honest sentence is: reading the probabilities gave the better gate under both layouts, and the size of the advantage depends on how the options are listed.

Jev held a third of its wrong answers at 90% or more here, against 2 of 31 on the AI-keyed questions. On items where 100 people split, though, the median human share on those confident wrong answers was 38%, so "wrong" is a harder judgment than the table makes it look.

Against the full human distribution rather than the majority, Eve, Decider, the probability reading and Jev were closest, within 0.02 of each other, and the written-out version and the Kevs were furthest. Whether Qwen, Jev or Laya have seen MNLI, I cannot say.

If you gate on a written-out confidence today, this is the experiment to run this week: same model, both readouts, two hundred of your own items.

### Latency: how many questions per call

Jev and Laya answer fifty questions about as fast as they answer one. Nothing else here does. My own inference servers were down for about 23 minutes so each local model could be measured alone on the GPU (Jev is hosted; the one-structured-request Qwen row was measured separately with my servers running). Median response time is in the chart, in milliseconds.

*Figure: Log scale chart of median response time against 1, 5, 10 and 50 questions per call. Decider-2b starts at 5 ms and reaches 166. Jev runs 156 to 190. Laya ends at 113. Qwen as fifty separate requests reaches 2,766 ms, and as one structured request 1,838 ms. A toggle switches to the 90th percentile.*

For a builder the useful reading is the slope. Everything but Jev and Laya grows from one question to fifty, from Eve's 40% to Decider's thirty-fold, and the general LLM most in absolute terms. If one input carries twenty questions, a flat system is a different component from one that grows with every question. Tails matter too: at fifty questions Decider's 90th percentile opens to 749 ms. Medians are what people quote; tails are what page you at night.

These are timings of these deployments on a synthetic workload. Loopback against a hosted API across the internet is not like for like, and I cannot subtract my network time to recover anyone's compute time. My general LLM ran at its production setting of four concurrent sequences. One oddity I have not explained: Kev-9b answered a single question faster than Kev-4b. Rerun that before leaning on it.

## Part two: the checks

Everything above is about when a number can be trusted. Here is when mine could not.

### Three checks that changed the numbers

**Prompts.** My wrapper for Eve dropped the fine print on its yes/no questions. Fixed and rerun, Eve's accuracy landed within two questions on each benchmark tier and a point lower on the second set, but its confidence changed: it was never 90% sure on a hard question, against 11 times before, 7 of them wrong. I report the first run throughout. The option-order control moved Qwen's probability reading by 7 points, as large as its apparent lead over Jev, so that lead is not reported as one. And on my second test set, asking yes or no questions as lettered options instead of the words yes and no cost 16 points on the 600 yes or no decisions, 61% against 77%.

*Figure: Dumbbell chart of the Qwen probability reading under three prompt layouts across two test sets. On the hard questions, reversing the option order moves accuracy from 78 to 71 percent. On the second set the same reversal moves it from 65 to 71 percent, and words for yes and no reach 70 percent. A last panel counts changed top answers out of 105 reorderable hard questions: Jev 7 on an identical repeat and 8 with options reversed, written-out Qwen 0 and 36, the probability reading 0 and 26.*

**Instruments.** The harness I started with opens a new connection for every request, which charged the hosted API a fresh handshake each time and made it look about 140 ms slower than it is; every timing here uses a connection that stays open. My first latency loop sent the same input on every call, which flatters anything with a cache: one model read 63 ms that way and 100 ms once I varied the input. Both errors produced the number I was expecting to see, which is why they lasted.

**Statistics.** Jev and the Qwen probability reading differ on 25 of 111 hard questions, 9 one way and 16 the other, p = 0.23. That is inconclusive, not a tie and not a win. And the hard answer keys were written by two models, 57 by GPT-5.6 Sol and 54 by Claude Opus 5, which I only knew after reading the whole provenance field rather than one row of it.

*Figure: Six first measurements and what the check showed for each. They cover option order moving the Qwen probability reading by 7.2 points, a yes or no format that moved its second-set result, a wrapper fix that did not raise a score, a timing artifact of about 140 milliseconds, a difference at p equals 0.23 that is inconclusive, and a claim about small models that was false for two of five.*

Two other AI systems checked the work before publication: Codex, which recomputed every number from the raw files, and Grok, with live web search. They found all of the above. They were wrong too: two of three checkable claims from one of them failed when I checked, a temperature flag that does not exist at that commit and a floor figure from a different definition than mine. A hostile review is a list of leads, not a verdict.

Yes. An AI reviewed an AI's evaluation of AI models answering questions written by AI. I am not going to defend that. It is why the nine-system table in Part one is scored by people, and why I dropped my plan to label the 111 hard questions myself: a hundred people per item beat one tired author.

The three kinds have one thing in common: every one of these errors made the story simpler than the data did, and not one made it more complicated. I do not think that is chance. It is what motivated reasoning looks like from the inside, where it does not feel like reasoning at all. It feels like being finished.

Of 127 numbers in my methods write-up recomputed independently, 2 were wrong, and both are fixed. The arithmetic held. The interpretations were where the errors lived.

**Before you run this on your own data.** For anything that matters, none of these models until you have labelled a few hundred of your own items and run this same test on them.

1. Keep only the answers where the system said 90% or more, score them, and compare with its overall accuracy. That gap is the whole test.
2. If you can read token probabilities, read them, and test that against asking the model to write its confidence down.
3. Send the identical request twice, then reorder the options and reword the yes or no questions, before you believe a gap under 10 points.
4. Label specialists. A model tuned on the benchmark does not belong in the same column as one that was not.
5. Time it with a persistent connection and a different input on every call.
6. Have the write-up attacked before you publish, then check the attacker too.

The two review prompts, a numbers audit and a methods review run separately, are generalized and available on request.

**Check me.** The test items are public ([JevBench](https://github.com/fstandhartinger/jevbench), [ChaosNLI](https://github.com/easonnie/ChaosNLI)). My scripts, per-item results, raw timings, the pre-registration with its commit timestamp, the three reviews and their prompts are available on request. {/* TODO before publishing: point at the companion repo if the data goes public */}

### Credits and licenses

[JevBench](https://github.com/fstandhartinger/jevbench), by Florian Standhartinger, MIT. The [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) set, Apache 2.0, whose labels are a teacher model's rather than a person's. [Kev](https://github.com/jaredpalmer/kev), by Jared Palmer, Apache 2.0. [Decider](https://github.com/Mapika/decider), by Mark Marosi, Apache 2.0. [Eve RLCD](https://github.com/anthony-maio/eve-rlcd), by Anthony Maio, MIT code and Apache 2.0 weights. [Laya](https://github.com/NandhaKishorM/laya), by Nandakishor at Convai Innovations, Apache 2.0. [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B-FP8), Apache 2.0. [ChaosNLI](https://github.com/easonnie/ChaosNLI), by Yixin Nie, Xiang Zhou and Mohit Bansal, CC BY-NC 4.0, the only human labels here. [Jev](https://typesafe.ai), by TypeSafe, hosted, early access. Thank you.

**Found a mistake?** Tell me on [LinkedIn](https://www.linkedin.com/in/colinmcnamara/) or [X](https://twitter.com/colinmcnamara). I will fix it here, dated, with credit to whoever caught it.

If you have human-labelled decisions from real work, not a benchmark, I would like to run this on them.