---
title: "A Model That Refuses to Talk"
description: "We keep asking chat models for decisions, then parsing the prose back into something code can branch on. TypeSafe's Jev skips the prose. I spent a few days reading the public material, then got access and measured it myself. Fast and accurate held up. On my two prompts, calibration error came out near 0.09, and a plain open model on my own GPU was calibrated at least as well."
author: "Colin McNamara"
tags: ["Decision Models","Calibration","Semantic Routing","AI Architecture"]
image: "https://colinmcnamara.com/images/blog/jev-system-one/fig8-same-items-three-ways.png"
date: 2026-09-20
url: https://colinmcnamara.com/blog/jev-system-one-decision-models
---

# A Model That Refuses to Talk

A support ticket lands. Your code needs one thing from it: which queue.

So you call a language model. You ask for JSON and get back tokens shaped like `{"queue": "billing", "confidence": 0.9}`. You parse it. You branch on it.

Look at what just happened. You needed a decision. You asked a writer. It wrote you a sentence. You turned the sentence back into a decision.

Structured outputs fix the shape of that sentence. They do not fix the mismatch. And that `confidence: 0.9` is a generated estimate, not evidence of calibration. Whether it predicts being right is something you would have to measure.

I have been circling this from the other side. [The Router Was Right](https://colinmcnamara.com/blog/the-router-was-right) and [It Was Never the Model](https://colinmcnamara.com/blog/it-was-never-the-model) were both about a semantic router, a small component whose whole job is deciding where a request should go. A router is a decision model. It just happens to be one you have to build and tune yourself.

[TypeSafe AI](https://typesafe.ai) shipped a model that takes the other road. It never writes text at all. It is called Jev. The docs describe it as "TypeSafe's flagship model and the first System One model." [Early access opened on September 15, 2026](https://typesafe.ai/blog/introducing-system-one-models-and-jev) from a waitlist, alongside a [$40M seed led by DCVC](https://www.dcvc.com/news-insights/typesafe-emerges-from-stealth-with-a-new-way-of-doing-ai/).

Most of this post is a reading of the public docs, papers, and press. Partway through writing it I got early access, so near the end I stop reading and run it. I will tell you exactly which numbers are mine.

The short version: the idea is good, the speed is real, and the docs are unusually candid about where the model breaks. On my two test prompts, expected calibration error was about 0.09, underconfident on one and overconfident on the other. A plain open model on my own GPU was calibrated at least as well, and a few times slower. Measure before you trust.

| What I measured | Result |
|---|---|
| Accuracy with no examples and no tuning | 94.0% on sentiment, 88.2% on news topics |
| Calibration error (ECE, lower is better) | about 0.09 on both, in opposite directions |
| Latency from my desk | 143 ms for one question, 157 ms for ten |
| Against an open 27B model on my own GPU | accuracy close, the open model better calibrated, Jev 1.8 to 4.5 times faster |

If you only want the numbers, [jump to the measurements](#so-i-ran-it).

*Figure: Animated race between two ways of getting a decision out of a model. The top lane shows a chat model emitting a JSON answer one token at a time, then a parsing step, then the branch in your code. The bottom lane shows the same state read once, three question lanes lighting up in parallel, and three typed answers with probability distributions landing together. A Replay button restarts both lanes. The takeaway is that one path produces text you must convert back into a decision while the other produces the decision directly. Timing in the animation is schematic and not measured.*

## What Jev does instead

The contract inverts. You send a **state**, the material a panel of experts would read before judging anything. Then you send **typed questions** about it. You get typed answers with probability distributions back. No prose, ever.

There are only [three question types](https://docs.typesafe.ai/primitives).

[**Choice**](https://docs.typesafe.ai/primitives/choice) asks which of these options. It returns the winning `choice`, a probability per option, and a `confidence` value.

[**Score**](https://docs.typesafe.ai/primitives/score) asks where this lands on an ordered rubric. It returns a `score`, a probability per level, and `confidence`. The score is "a probability-weighted mean of the level numbers," so it can land at 1.3, between two levels you defined. Write the levels as situations, not degrees. As I write this, the docs show levels written as "0", "1", "2" returning a score of 0.55 at confidence 0.33 where descriptive levels return 0.0 at confidence 1.0. The model reads your words. It does not see the numbers.

[**Noul**](https://docs.typesafe.ai/primitives/noul) asks whether something is true. It returns one value from 0 to 1, the probability that the answer is yes, with no separate confidence field.

The structural part I like most: every question is evaluated in parallel and in isolation against the same state. A tenth question does not pollute the other nine. It barely moves latency either. TypeSafe says end-to-end response time is "70ms-500ms" and calls the model "40x-200x faster" than frontier LLMs.

**I tested this.** From my desk, one question took a median of 143 ms and ten questions took 157 ms. [The measurements](#so-i-ran-it).

[The hard limits](https://docs.typesafe.ai/models) are short. Model ID `jev-1.13.0`. Input costs $0.042 per million tokens and output is free. Context is 64k per request. Text only, English first. Every account gets the same weights, with no customer fine-tuning.

One practical note. The Python package is `typesafe-sdk`. Do not run `pip install typesafe`. That installs [an unrelated library from April 2010](https://pypi.org/project/typesafe/).

*Figure: Interactive panel showing the three Jev primitives answering questions about a support ticket you pick from a short list. A Choice appears as a horizontal stacked bar of probabilities across the candidate queues. A Score appears as a distribution over three rubric levels with a marker at the probability weighted mean, which can land between levels. A Noul appears as a single dial from zero to one for the probability that the answer is yes. Switching tickets animates the bars, and one ticket produces a nearly flat distribution with low confidence. The numbers are simulated to teach the shape of the output and are not Jev results.*

That panel is a simulation I built to make the output shape visible. It is not a Jev call. Watch the ticket where the bars go flat and confidence drops. That is how this kind of model says no option is a clear winner.

## The training claim: calibrated by design

[TypeSafe's primer](https://docs.typesafe.ai/introduction/machine-learning-primer) tells a three-step story. RLHF optimizes for responses people prefer, and it produced chatbots. RLVR optimizes for verifiable rewards, and it produced reasoning models that are slower and cost more. TypeSafe says its own method, RLCD, optimizes for calibrated decisions. Its argument is that preference training narrows the output distribution, which damages the honest spread of probability a machine consumer needs. The System One name comes from Kahneman: the fast judgment you make before you have reasoned about anything.

So what does calibration actually promise? Across many predictions the model gave 0.8, about 80% should turn out right. That is the whole claim. It is a property of groups of predictions. It does not guarantee that the single 0.94 in front of you is correct, or that calibration holds for the slice of traffic it came from.

The usual score for this is expected calibration error, or ECE. Sort the predictions into bins by what the model said, compare that with what actually happened in each bin, and average the gaps. Zero is perfect. Lower is better. It shows up a lot below.

*Figure: Interactive reliability diagram for teaching calibration. Predicted probability runs along the horizontal axis and observed frequency along the vertical, with a diagonal marking perfect calibration and ten binned dots sized by how many predictions fall in each bin. An overconfidence slider bends the curve away from the diagonal and updates a live expected calibration error readout. A toggle holds accuracy fixed while switching between good and bad calibration, showing that the two are different properties. All points come from a seeded simulation running in the browser and are not Jev results.*

Simulated again, generated in your browser to teach the shape. Flip the toggle. Accuracy stays the same while calibration falls apart. That is the idea worth carrying out of this post: a model can be right just as often and still lie to you about how sure it is.

**I tested this.** On two prompts of my own, scored against human labels, Jev's calibration error came out near 0.09. It was underconfident on one and overconfident on the other. Figure 7 has the real curves. [The measurements](#so-i-ran-it).

## Confidence is the lever, if it is honest

[The confidence page](https://docs.typesafe.ai/confidence) is where the product gets interesting to me as a builder.

Confidence is a statistic computed from the shape of the probability distribution. Peaked means confident. Flat means unsure. It is not the probability that the answer is correct, and conflating those two will hurt you. You always get the full distribution back, so you can compute your own measure if you prefer one.

**I tested this.** TypeSafe's public adapter code rescales the top probability into `confidence`. That formula matched the hosted values to within 0.0133 on all 500 of my Choice answers. On a Choice, `confidence` is mostly the top probability in different units. [The measurements](#so-i-ran-it).

The pattern is three bands. High confidence, act. Middle, confirm. Low, escalate to a human or to a reasoning model. What makes it an architecture and not a config value is that the thresholds scale with the stakes. Auto-tagging a ticket and auto-issuing a refund do not deserve the same bar.

**I tested this.** Three inputs that fit none of the options still got an answer, with middling confidence of 0.40 to 0.43. Given a `none_of_these` option, Jev picked it at 1.00 every time. [The measurements](#so-i-ran-it).

*Figure: Interactive confidence gate simulator. Two hundred simulated decisions are drawn as a strip of dots sorted by confidence and colored by whether they were right or wrong. Two draggable thresholds split the strip into escalate, confirm, and act bands, and live counters show how many were auto acted on, how many of those were wrong, and how many went to a human. A stakes toggle between a read only action and a money moving action shifts the default thresholds. The takeaway is that the right threshold depends on what a wrong answer costs, not on a number someone put in a blog post. The data is simulated and is not Jev output.*

Simulated decisions again, not Jev output. Move the handles and watch the counters. Widening what you automate can let more wrong answers ship unsupervised. Measure that trade on your own data, then set each threshold by what a mistake costs.

## The page I trust most is their own failure list

TypeSafe publishes a [model jaggedness page](https://docs.typesafe.ai/model-jaggedness/jev-1.13) that lists where Jev 1.13 breaks. Vendors do not usually do this. It is the page I would hand to a skeptical engineer first.

The greatest hits. Jev reads literally, answering what you wrote rather than what you meant. "It struggles with tasks that require numeric precision," so counting, arithmetic, and date comparison belong in your code. Give it a large state full of irrelevant detail and it "suffers from context rot." Adversarial text inside the state can move the answer, so prompt injection did not go away just because the output is typed.

One thing the launch coverage blurred. More **questions** do not cause context rot, because questions run isolated. A bigger, noisier **state** does. Both are true and they are about different inputs.

Then there is the part I keep coming back to.

*Figure: Two bars taken from TypeSafe's own jaggedness page. Asked as separate yes or no questions about the same support ticket, the probability of asking for a refund comes back 0.72 and the probability of asking for something other than a refund comes back 0.47. Stacked, they reach 1.19 and pass the 1.0 line, with the excess highlighted. TypeSafe presents the pair as a question and its negation, although they are not strict complements because one ticket can ask for both. Beside them a second pair shows the same question answered as a Noul at 0.22 and as a yes or no Choice at yes equals 0.01 with confidence 0.97. The takeaway is that answers to separate questions do not behave like one coherent probability distribution.*

Start with the cleaner example. "Is the customer asking for a refund?" returns 0.22 as a Noul. Asked as a yes or no Choice, the same question returns yes at 0.01 with confidence 0.97. Same ticket, same words, different number.

The second is the one TypeSafe calls a question and its negation. "Asking for a refund?" comes back 0.72. "Asking for something other than a refund?" comes back 0.47. Together that is 1.19. A careful reader will object that these are not strict complements, since one ticket can ask for both. Fair.

**I tested this.** Both examples reproduced closely on my key, three runs each. Then I asked the same ticket a strict negation of the refund question. It summed to 1.19 again. [The measurements](#so-i-ran-it).

These are examples the vendor wrote down, not bugs they hid. The lesson is that the probabilities are not one coherent belief state you can do algebra on. Do not carry a threshold tuned on a Noul over to a Choice, and do not expect answers to separate questions to add up like a single distribution.

This is also where "it cannot hallucinate" dies. That is true about output shape only. A valid option from your list can still be confidently wrong, and the CEO acknowledged exactly that [on Hacker News](https://news.ycombinator.com/item?id=49717558).

One design lesson from outside the vendor. Archer Hume ran an [independent probe](https://archerhume.com/posts/jevs-architecture-unmasked/) on two synthetic templates and found a position effect.

*Figure: Bar chart of Archer Hume's position experiment. When the fact needed to answer sat inside one of the Choice options, Jev was correct 12 of 16 times with that option listed first, 11 of 16 in the middle, and 16 of 16 last. When the same fact was moved into the shared state instead, it was correct 48 of 48 times. This was a narrow probe on two synthetic templates. The takeaway is to keep evidence that every option depends on in the shared state.*

His setup was deliberately odd: the deciding fact sat inside a reference option the model was told never to select. Accuracy moved with where that option sat: 12 of 16 first, 11 of 16 in the middle, 16 of 16 last. Moved into the shared state, it went 48 of 48. A narrow probe, so a narrow lesson: evidence every option depends on belongs in the state. Options still deserve good descriptions.

## This is not a new interface

Typed answers with probabilities is old ground, and saying so is not a knock. If you only came for the measurements, [skip to So I ran it](#so-i-ran-it).

Constrained decoding already guarantees output shape. Logprob classification, which reads the model's raw token probabilities instead of its text, already gives you a distribution over labels: map each option to its own single token, read the next-token logprobs after one forward pass, and renormalize. Nothing about that makes the distribution calibrated. Fine-tuned encoder heads have done fast fixed-task classification for years. [GLiNER2](https://arxiv.org/abs/2507.18546) supports multiple classification tasks in a single call, and its paper reports CPU latency of 130 to 208 ms as labels grow from 5 to 50.

The reproductions arrived fast. Niels Rogge at Hugging Face published a [visual explainer](https://x.com/NielsRogge/status/2100239244501430438) of the likely mechanism, based on a community reproduction. Run the context and schema through once and cache it. Then, for each field, read the probabilities of only the tokens that field allows. Eric Zhang shipped a [Jev-compatible API on an open model](https://github.com/ekzhang/openjev-sglang) and reported `64 tasks in <1s`. Those reproduce the interface and the caching trick. Nobody has shown that a reproduction matches Jev's accuracy or calibration.

The CEO points at where the value would sit. Diogo Almeida, a primary author on the [InstructGPT paper](https://arxiv.org/abs/2203.02155), [replied on X](https://x.com/CompleteSkeptic/status/2100617775823966680) that TypeSafe considers itself "a data research lab!" and that "100% of our data is synthetic." Han Xiao, VP of AI at Elastic, [expects](https://x.com/hxiao/status/2101001002816327867) "tool calling and routing move back to discriminative models."

So the new thing would be general zero-shot calibration, meaning with no examples and no tuning, at this price. That is the unproven part.

## What had anyone measured?

Not much, and mostly not against human labels. As far as I can find, TypeSafe has not published an ECE, a Brier score, or a reliability diagram. Its workflow evals grade against answers averaged from two frontier models, not independent labels. On its own invoice eval Jev scored 61.8% against 79.1% for a frontier model, [as Anthony Maio reported](https://anthonymaio.substack.com/p/jev-the-language-model-that-wont). Dan Shipper [wrote](https://x.com/danshipper/status/2099947471518474522) that in Every's testing Jev was "25x faster and 600x lower priced." [The article](https://every.to/also-true-for-humans/mini-vibe-check-typesafe-s-jev-judged-everything-i-ve-written-in-0-7-seconds) adds the useful part: across 12 passages Jev caught six of seven planted defects and Claude Fable 5.1 at high effort caught all seven. [Emil Lindfors published](https://lindfors.no/blog/a-first-look-at-typesafes-jev) an early third-party calibration check, small and scored against another model's labels.

So I added one of my own.

## So I ran it

Five small tests on September 20, 2026, from Austin, against `jev-1.13.0` pinned by version, using only public data and my own wording. A first look, not a verdict.

**The two documented failure examples reproduced closely.** Three runs each. The refund question as a Noul came back 0.21, 0.22, 0.21 against their 0.22. As a Choice, yes came back 0.00, 0.01, 0.00 against their 0.01. The refund pair came back 0.72 to 0.73 and 0.41 to 0.47 against their 0.72 and 0.47. Note the wobble. Identical calls did not return identical numbers.

**Ten questions cost about a tenth more than one.** After a warm-up, I interleaved 60 calls with one Noul and 60 with ten, on one short ticket. One question: median 143 ms, 90th percentile 191 ms. Ten: median 157 ms, 90th percentile 206 ms. That is SDK round trip from my desk, inside TypeSafe's "70ms-500ms." I cannot see what the server caches or how it schedules work. I can say nine extra questions cost me about 14 ms.

**Three out-of-scope inputs produced middling confidence.** A password reset ticket, a recipe, and keyboard mash, sent to a Choice of billing, shipping, returns. It picked shipping every time at 0.60 to 0.62, with `confidence` of 0.40 to 0.43. Add a `none_of_these` option and it picked that at 1.00 every time. Three cases do not validate a detector. They do make the case for always giving a Choice a way out.

**Strict complements did not always sum to one.** Back to the earlier objection. I asked ten tickets two pairs of questions, each a condition and its strict negation. Across 20 pairs the answers summed to between 0.93 and 1.19, mean 1.02, with fourteen within 0.05 of one. "Do you offer a student discount?" came back 0.01 and 0.98. TypeSafe's double charge ticket came back 0.74 and 0.45, which is 1.19 again. My guess is ambiguity drives the misses, but I have no ambiguity labels, so it stays a guess. Do not assume complementary questions return complementary numbers.

**On my two prompts, calibration error was about 0.09.** This is the one I cared about. I drew a seeded random sample of 500 human-labeled items from each of two public datasets. [SST-2](https://huggingface.co/datasets/stanfordnlp/sst2) movie review sentences became a Noul: does this express a positive opinion? [AG News](https://huggingface.co/datasets/fancyzhx/ag_news) items became a Choice over four topics.

*Figure: Two reliability diagrams from my own measurements of jev-1.13.0 against human labels, a seeded random sample of 500 items each. Left panel, sentiment as a Noul on SST-2: accuracy 94.0 percent, expected calibration error 0.093, Brier score 0.049. The dots sit below the diagonal at low probabilities and above it at high probabilities, which means the model was underconfident on this sample. When it said about 0.76 the answer was yes 94 percent of the time. Right panel, news topic as a four way Choice on AG News: accuracy 88.2 percent, expected calibration error 0.087. Most of the weight is one large dot at the far right, a bin of 427 answers with a mean predicted probability of 0.995, sitting below the diagonal at 92.5 percent observed accuracy, which means the model was overconfident at the top on this sample. Hollow dots mark bins with fewer than 20 items.*

Figure 3 was a simulation. This one is not.

Accuracy was good for zero effort: 94.0% on sentiment and 88.2% on topics, no examples, no tuning. A thousand calls took about forty seconds.

Calibration error came out near 0.09 on both, in opposite directions. On sentiment the model was underconfident. Where it said about 0.76, the answer was yes 94% of the time. Where it said about 0.15, it was yes 9% of the time. Of the 27 items it scored between 0.3 and 0.5, 18 were positive, which suggests testing a lower cutoff on a separate holdout. It does not tell me which cutoff, because I picked it after looking.

On topics it was overconfident at the top. The highest bin held 427 of the 500 answers, with a mean predicted probability of 0.995 and observed accuracy of 92.5%. One bin down, 25 answers averaged 0.86 and were right 48% of the time.

A side finding on `confidence`. The docs do not give its formula. TypeSafe's official [adapter repo](https://github.com/typesafe-ai/system-one-adapter-python), a drop-in client backed by ordinary LLM APIs, publishes one. Rescale the normalized top probability so a uniform guess is 0 and certainty is 1. Applied to my returned distributions, it differed from the returned `confidence` by at most 0.0133 across all 500 answers. That suggests the hosted model computes something similar without establishing it. On a Choice, `confidence` is unlikely to tell you much that the top probability does not.

The caveats are real. 500 items per task. One wording per question, and it is mine. Both datasets are famous, so they could be in anyone's training data. AG News labels are noisy where business and technology overlap. Several middle bins hold fewer than 20 items. My first pass used the first 500 rows. After noticing that AG News is ordered by topic, I reran everything with a seeded random sample.

So here is what I can say. The speed is real. The zero-shot accuracy is real. The documented failure examples reproduced. And these two prompts were miscalibrated on my samples, by about 0.09, in opposite directions. That does not show Jev's calibration fails in general. It reinforces the boring rule: measure calibration on the workload you intend to run, then fit your thresholds there.

Check my work. Every script and every raw result is in [jev-first-look on GitHub](https://github.com/colinmcnamara/jev-first-look): the [calibration script](https://github.com/colinmcnamara/jev-first-look/blob/main/t3_calibration.py), its [binned results](https://github.com/colinmcnamara/jev-first-look/blob/main/results/t3_results.json), the [per-item raw results](https://github.com/colinmcnamara/jev-first-look/blob/main/results/t3_raw.json), and the [negation pairs](https://github.com/colinmcnamara/jev-first-look/blob/main/results/t5_results.json).

## Against a plain LLM on my own GPU

One more test, because the obvious question is "compared to what?"

I ran the same 1,000 items through `Qwen/Qwen3.8-27B-FP8` on my own GH200, served with vLLM, an open-source inference server. Same seeded samples, same questions, same four parallel requests. Two classic methods: schema-constrained JSON with a self-reported confidence, and a single generated token read from logprobs and renormalized over the allowed answers.

*Figure: Comparison of three methods on the same seeded samples of 500 sentiment items and 500 news topic items: hosted Jev, Qwen3.8-27B with schema constrained JSON and a self reported confidence, and Qwen3.8-27B reading a single token from logprobs. Accuracy is close across all three: 94.0, 95.2, and 95.0 percent on sentiment, and 88.2, 88.2, and 86.8 percent on topics. Expected calibration error is lower for both Qwen methods on sentiment, 0.040 and 0.042 against 0.093 for Jev, and on topics it is 0.084 and 0.059 against 0.087. Jev is faster: about 20 seconds of wall time per 500 items against 47 to 48 seconds for JSON and 35 to 36 seconds for logprobs, and a median of 157 milliseconds for ten questions about one ticket against 467 milliseconds for one Qwen JSON call and 713 milliseconds for ten parallel Qwen one token calls.*

Accuracy was close. Both Qwen methods stayed within 1.4 points of Jev on each task.

Calibration went to the plain LLM. On sentiment, expected calibration error was 0.040 for the JSON method and 0.042 for logprobs, against 0.093 for Jev. On topics it was 0.084 and 0.059 against 0.087. Yes, that means the self-reported confidence I was suspicious of in the first paragraph did better than I expected.

Speed went to Jev, by less than the headlines suggest. In the four-worker dataset runs, Jev finished in about 1.8 to 2.4 times less wall time. On the ten-question ticket from earlier, Jev's median was 157 ms. One Qwen JSON call returning ten booleans took 467 ms. Ten one-token calls fired in parallel took 713 ms. That is about 3.0 and 4.5 times. My GPU sits about 33 ms of network away, which is inside those numbers. The "40x-200x" and "25x" figures are comparisons against frontier models thinking hard. Against a mid-size open model on one GPU, it is a few times.

The caveats are heavier here. One open model. One machine. Two famous datasets that Qwen has plausibly seen, which flatters both its accuracy and its calibration. And I have not compared cost: $0.042 per million tokens with nothing to run, against a GPU I own and operate. The [baseline scripts and results](https://github.com/colinmcnamara/jev-first-look#baseline-an-ordinary-open-llm-on-the-same-items-t6-t7) are in the same repo.

The proper version of this test puts the open-weight models built in Jev's style on the same hardware. That is the follow-on post, and its scripts will land in the repo first.

## What I would still test

[Guo and colleagues showed in 2017](https://arxiv.org/abs/1706.04599) that modern networks are often miscalibrated, and that temperature scaling on held-out data often fixes it. The operative phrase is *on held-out data*. Yours. Here is the full list, with where I stand on each. Nothing on it is specific to this vendor. It is what any decision model owes you.

1. **Partly done, on public data only.** Build a labeled holdout from my own domain. Measure accuracy, ECE, Brier score, and a reliability diagram, per question.
2. **Not started.** Measure accuracy against coverage at every confidence threshold. That curve tells you where to put the gates.
3. **Started, three cases.** Force failures: out-of-distribution text, instructions injected into the state, every option wrong, two intents at once. Always include a `none` option in a Choice.
4. **Not started.** Treat the question text as part of the system under test. Lindfors saw his calibration error change after he added qualifiers to his questions. Re-measure whenever a question changes.
5. **Started, one Qwen baseline.** Test the open-weight models that implement this same style: Eric Zhang's Jev-compatible server, the community RLCD reproduction on a small Qwen, and GLiNER2. Same hardware, same harness, next to a plain logprob baseline and a fine-tuned encoder head. This is the one I most want to run, and it is the follow-on post.
6. **Done from Austin.** Measure latency from my own region, p50 and p99, not the vendor's number.
7. **Done.** Pin `jev-1.13.0`, log the model field on every response, and re-tune thresholds when the version moves. Aliases drift under you.

The builder's version is four lines. Keep your code in control, and leave arithmetic, dates, and policy there. Give the model narrow judgments a knowledgeable person could make in a second. Measure calibration on your own data before you wire a threshold to anything that spends money. Pin the version.

A model that refuses to talk is a good idea. Whether this one is well calibrated on your problem is a question only your holdout set can answer.