A Model That Refuses to Talk

A support ticket lands. Your code needs one thing from it: which queue.

So you call a language model. You ask for JSON and get back tokens shaped like {"queue": "billing", "confidence": 0.9}. You parse it. You branch on it.

Look at what just happened. You needed a decision. You asked a writer. It wrote you a sentence. You turned the sentence back into a decision.

Structured outputs fix the shape of that sentence. They do not fix the mismatch. And that confidence: 0.9 is a generated estimate, not evidence of calibration. Whether it predicts being right is something you would have to measure.

I have been circling this from the other side. The Router Was Right and It Was Never the Model were both about a semantic router, a small component whose whole job is deciding where a request should go. A router is a decision model. It just happens to be one you have to build and tune yourself.

TypeSafe AI shipped a model that takes the other road. It never writes text at all. It is called Jev. The docs describe it as “TypeSafe’s flagship model and the first System One model.” Early access opened on September 15, 2026 from a waitlist, alongside a $40M seed led by DCVC.

Most of this post is a reading of the public docs, papers, and press. Partway through writing it I got early access, so near the end I stop reading and run it. I will tell you exactly which numbers are mine.

The short version: the idea is good, the speed is real, and the docs are unusually candid about where the model breaks. On my two test prompts, expected calibration error was about 0.09, underconfident on one and overconfident on the other. A plain open model on my own GPU was calibrated at least as well, and a few times slower. Measure before you trust.

If you only want the numbers, jump to the measurements.

What Jev does instead

The contract inverts. You send a state, the material a panel of experts would read before judging anything. Then you send typed questions about it. You get typed answers with probability distributions back. No prose, ever.

There are only three question types.

Choice asks which of these options. It returns the winning choice, a probability per option, and a confidence value.

Score asks where this lands on an ordered rubric. It returns a score, a probability per level, and confidence. The score is “a probability-weighted mean of the level numbers,” so it can land at 1.3, between two levels you defined. Write the levels as situations, not degrees. As I write this, the docs show levels written as “0”, “1”, “2” returning a score of 0.55 at confidence 0.33 where descriptive levels return 0.0 at confidence 1.0. The model reads your words. It does not see the numbers.

Noul asks whether something is true. It returns one value from 0 to 1, the probability that the answer is yes, with no separate confidence field.

The structural part I like most: every question is evaluated in parallel and in isolation against the same state. A tenth question does not pollute the other nine. It barely moves latency either. TypeSafe says end-to-end response time is “70ms-500ms” and calls the model “40x-200x faster” than frontier LLMs.

The hard limits are short. Model ID jev-1.13.0. Input costs $0.042 per million tokens and output is free. Context is 64k per request. Text only, English first. Every account gets the same weights, with no customer fine-tuning.

One practical note. The Python package is typesafe-sdk. Do not run pip install typesafe. That installs an unrelated library from April 2010.

That panel is a simulation I built to make the output shape visible. It is not a Jev call. Watch the ticket where the bars go flat and confidence drops. That is how this kind of model says no option is a clear winner.

The training claim: calibrated by design

TypeSafe’s primer tells a three-step story. RLHF optimizes for responses people prefer, and it produced chatbots. RLVR optimizes for verifiable rewards, and it produced reasoning models that are slower and cost more. TypeSafe says its own method, RLCD, optimizes for calibrated decisions. Its argument is that preference training narrows the output distribution, which damages the honest spread of probability a machine consumer needs. The System One name comes from Kahneman: the fast judgment you make before you have reasoned about anything.

So what does calibration actually promise? Across many predictions the model gave 0.8, about 80% should turn out right. That is the whole claim. It is a property of groups of predictions. It does not guarantee that the single 0.94 in front of you is correct, or that calibration holds for the slice of traffic it came from.

The usual score for this is expected calibration error, or ECE. Sort the predictions into bins by what the model said, compare that with what actually happened in each bin, and average the gaps. Zero is perfect. Lower is better. It shows up a lot below.

Simulated again, generated in your browser to teach the shape. Flip the toggle. Accuracy stays the same while calibration falls apart. That is the idea worth carrying out of this post: a model can be right just as often and still lie to you about how sure it is.

Confidence is the lever, if it is honest

The confidence page is where the product gets interesting to me as a builder.

Confidence is a statistic computed from the shape of the probability distribution. Peaked means confident. Flat means unsure. It is not the probability that the answer is correct, and conflating those two will hurt you. You always get the full distribution back, so you can compute your own measure if you prefer one.

The pattern is three bands. High confidence, act. Middle, confirm. Low, escalate to a human or to a reasoning model. What makes it an architecture and not a config value is that the thresholds scale with the stakes. Auto-tagging a ticket and auto-issuing a refund do not deserve the same bar.

Simulated decisions again, not Jev output. Move the handles and watch the counters. Widening what you automate can let more wrong answers ship unsupervised. Measure that trade on your own data, then set each threshold by what a mistake costs.

The page I trust most is their own failure list

TypeSafe publishes a model jaggedness page that lists where Jev 1.13 breaks. Vendors do not usually do this. It is the page I would hand to a skeptical engineer first.

The greatest hits. Jev reads literally, answering what you wrote rather than what you meant. “It struggles with tasks that require numeric precision,” so counting, arithmetic, and date comparison belong in your code. Give it a large state full of irrelevant detail and it “suffers from context rot.” Adversarial text inside the state can move the answer, so prompt injection did not go away just because the output is typed.

One thing the launch coverage blurred. More questions do not cause context rot, because questions run isolated. A bigger, noisier state does. Both are true and they are about different inputs.

Then there is the part I keep coming back to.

Start with the cleaner example. “Is the customer asking for a refund?” returns 0.22 as a Noul. Asked as a yes or no Choice, the same question returns yes at 0.01 with confidence 0.97. Same ticket, same words, different number.

The second is the one TypeSafe calls a question and its negation. “Asking for a refund?” comes back 0.72. “Asking for something other than a refund?” comes back 0.47. Together that is 1.19. A careful reader will object that these are not strict complements, since one ticket can ask for both. Fair.

These are examples the vendor wrote down, not bugs they hid. The lesson is that the probabilities are not one coherent belief state you can do algebra on. Do not carry a threshold tuned on a Noul over to a Choice, and do not expect answers to separate questions to add up like a single distribution.

This is also where “it cannot hallucinate” dies. That is true about output shape only. A valid option from your list can still be confidently wrong, and the CEO acknowledged exactly that on Hacker News.

One design lesson from outside the vendor. Archer Hume ran an independent probe on two synthetic templates and found a position effect.

His setup was deliberately odd: the deciding fact sat inside a reference option the model was told never to select. Accuracy moved with where that option sat: 12 of 16 first, 11 of 16 in the middle, 16 of 16 last. Moved into the shared state, it went 48 of 48. A narrow probe, so a narrow lesson: evidence every option depends on belongs in the state. Options still deserve good descriptions.

This is not a new interface

Typed answers with probabilities is old ground, and saying so is not a knock. If you only came for the measurements, skip to So I ran it.

Constrained decoding already guarantees output shape. Logprob classification, which reads the model’s raw token probabilities instead of its text, already gives you a distribution over labels: map each option to its own single token, read the next-token logprobs after one forward pass, and renormalize. Nothing about that makes the distribution calibrated. Fine-tuned encoder heads have done fast fixed-task classification for years. GLiNER2 supports multiple classification tasks in a single call, and its paper reports CPU latency of 130 to 208 ms as labels grow from 5 to 50.

The reproductions arrived fast. Niels Rogge at Hugging Face published a visual explainer of the likely mechanism, based on a community reproduction. Run the context and schema through once and cache it. Then, for each field, read the probabilities of only the tokens that field allows. Eric Zhang shipped a Jev-compatible API on an open model and reported 64 tasks in <1s. Those reproduce the interface and the caching trick. Nobody has shown that a reproduction matches Jev’s accuracy or calibration.

The CEO points at where the value would sit. Diogo Almeida, a primary author on the InstructGPT paper, replied on X that TypeSafe considers itself “a data research lab!” and that “100% of our data is synthetic.” Han Xiao, VP of AI at Elastic, expects “tool calling and routing move back to discriminative models.”

So the new thing would be general zero-shot calibration, meaning with no examples and no tuning, at this price. That is the unproven part.

What had anyone measured?

Not much, and mostly not against human labels. As far as I can find, TypeSafe has not published an ECE, a Brier score, or a reliability diagram. Its workflow evals grade against answers averaged from two frontier models, not independent labels. On its own invoice eval Jev scored 61.8% against 79.1% for a frontier model, as Anthony Maio reported. Dan Shipper wrote that in Every’s testing Jev was “25x faster and 600x lower priced.” The article adds the useful part: across 12 passages Jev caught six of seven planted defects and Claude Fable 5.1 at high effort caught all seven. Emil Lindfors published an early third-party calibration check, small and scored against another model’s labels.

So I added one of my own.

So I ran it

Five small tests on September 20, 2026, from Austin, against jev-1.13.0 pinned by version, using only public data and my own wording. A first look, not a verdict.

The two documented failure examples reproduced closely. Three runs each. The refund question as a Noul came back 0.21, 0.22, 0.21 against their 0.22. As a Choice, yes came back 0.00, 0.01, 0.00 against their 0.01. The refund pair came back 0.72 to 0.73 and 0.41 to 0.47 against their 0.72 and 0.47. Note the wobble. Identical calls did not return identical numbers.

Ten questions cost about a tenth more than one. After a warm-up, I interleaved 60 calls with one Noul and 60 with ten, on one short ticket. One question: median 143 ms, 90th percentile 191 ms. Ten: median 157 ms, 90th percentile 206 ms. That is SDK round trip from my desk, inside TypeSafe’s “70ms-500ms.” I cannot see what the server caches or how it schedules work. I can say nine extra questions cost me about 14 ms.

Three out-of-scope inputs produced middling confidence. A password reset ticket, a recipe, and keyboard mash, sent to a Choice of billing, shipping, returns. It picked shipping every time at 0.60 to 0.62, with confidence of 0.40 to 0.43. Add a none_of_these option and it picked that at 1.00 every time. Three cases do not validate a detector. They do make the case for always giving a Choice a way out.

Strict complements did not always sum to one. Back to the earlier objection. I asked ten tickets two pairs of questions, each a condition and its strict negation. Across 20 pairs the answers summed to between 0.93 and 1.19, mean 1.02, with fourteen within 0.05 of one. “Do you offer a student discount?” came back 0.01 and 0.98. TypeSafe’s double charge ticket came back 0.74 and 0.45, which is 1.19 again. My guess is ambiguity drives the misses, but I have no ambiguity labels, so it stays a guess. Do not assume complementary questions return complementary numbers.

On my two prompts, calibration error was about 0.09. This is the one I cared about. I drew a seeded random sample of 500 human-labeled items from each of two public datasets. SST-2 movie review sentences became a Noul: does this express a positive opinion? AG News items became a Choice over four topics.

Figure 3 was a simulation. This one is not.

Accuracy was good for zero effort: 94.0% on sentiment and 88.2% on topics, no examples, no tuning. A thousand calls took about forty seconds.

Calibration error came out near 0.09 on both, in opposite directions. On sentiment the model was underconfident. Where it said about 0.76, the answer was yes 94% of the time. Where it said about 0.15, it was yes 9% of the time. Of the 27 items it scored between 0.3 and 0.5, 18 were positive, which suggests testing a lower cutoff on a separate holdout. It does not tell me which cutoff, because I picked it after looking.

On topics it was overconfident at the top. The highest bin held 427 of the 500 answers, with a mean predicted probability of 0.995 and observed accuracy of 92.5%. One bin down, 25 answers averaged 0.86 and were right 48% of the time.

A side finding on confidence. The docs do not give its formula. TypeSafe’s official adapter repo, a drop-in client backed by ordinary LLM APIs, publishes one. Rescale the normalized top probability so a uniform guess is 0 and certainty is 1. Applied to my returned distributions, it differed from the returned confidence by at most 0.0133 across all 500 answers. That suggests the hosted model computes something similar without establishing it. On a Choice, confidence is unlikely to tell you much that the top probability does not.

The caveats are real. 500 items per task. One wording per question, and it is mine. Both datasets are famous, so they could be in anyone’s training data. AG News labels are noisy where business and technology overlap. Several middle bins hold fewer than 20 items. My first pass used the first 500 rows. After noticing that AG News is ordered by topic, I reran everything with a seeded random sample.

So here is what I can say. The speed is real. The zero-shot accuracy is real. The documented failure examples reproduced. And these two prompts were miscalibrated on my samples, by about 0.09, in opposite directions. That does not show Jev’s calibration fails in general. It reinforces the boring rule: measure calibration on the workload you intend to run, then fit your thresholds there.

Check my work. Every script and every raw result is in jev-first-look on GitHub: the calibration script, its binned results, the per-item raw results, and the negation pairs.

Against a plain LLM on my own GPU

One more test, because the obvious question is “compared to what?”

I ran the same 1,000 items through Qwen/Qwen3.8-27B-FP8 on my own GH200, served with vLLM, an open-source inference server. Same seeded samples, same questions, same four parallel requests. Two classic methods: schema-constrained JSON with a self-reported confidence, and a single generated token read from logprobs and renormalized over the allowed answers.

Accuracy was close. Both Qwen methods stayed within 1.4 points of Jev on each task.

Calibration went to the plain LLM. On sentiment, expected calibration error was 0.040 for the JSON method and 0.042 for logprobs, against 0.093 for Jev. On topics it was 0.084 and 0.059 against 0.087. Yes, that means the self-reported confidence I was suspicious of in the first paragraph did better than I expected.

Speed went to Jev, by less than the headlines suggest. In the four-worker dataset runs, Jev finished in about 1.8 to 2.4 times less wall time. On the ten-question ticket from earlier, Jev’s median was 157 ms. One Qwen JSON call returning ten booleans took 467 ms. Ten one-token calls fired in parallel took 713 ms. That is about 3.0 and 4.5 times. My GPU sits about 33 ms of network away, which is inside those numbers. The “40x-200x” and “25x” figures are comparisons against frontier models thinking hard. Against a mid-size open model on one GPU, it is a few times.

The caveats are heavier here. One open model. One machine. Two famous datasets that Qwen has plausibly seen, which flatters both its accuracy and its calibration. And I have not compared cost: $0.042 per million tokens with nothing to run, against a GPU I own and operate. The baseline scripts and results are in the same repo.

The proper version of this test puts the open-weight models built in Jev’s style on the same hardware. That is the follow-on post, and its scripts will land in the repo first.

What I would still test

Guo and colleagues showed in 2017 that modern networks are often miscalibrated, and that temperature scaling on held-out data often fixes it. The operative phrase is on held-out data. Yours. Here is the full list, with where I stand on each. Nothing on it is specific to this vendor. It is what any decision model owes you.

1. Partly done, on public data only. Build a labeled holdout from my own domain. Measure accuracy, ECE, Brier score, and a reliability diagram, per question.
2. Not started. Measure accuracy against coverage at every confidence threshold. That curve tells you where to put the gates.
3. Started, three cases. Force failures: out-of-distribution text, instructions injected into the state, every option wrong, two intents at once. Always include a none option in a Choice.
4. Not started. Treat the question text as part of the system under test. Lindfors saw his calibration error change after he added qualifiers to his questions. Re-measure whenever a question changes.
5. Started, one Qwen baseline. Test the open-weight models that implement this same style: Eric Zhang’s Jev-compatible server, the community RLCD reproduction on a small Qwen, and GLiNER2. Same hardware, same harness, next to a plain logprob baseline and a fine-tuned encoder head. This is the one I most want to run, and it is the follow-on post.
6. Done from Austin. Measure latency from my own region, p50 and p99, not the vendor’s number.
7. Done. Pin jev-1.13.0, log the model field on every response, and re-tune thresholds when the version moves. Aliases drift under you.

The builder’s version is four lines. Keep your code in control, and leave arithmetic, dates, and policy there. Give the model narrow judgments a knowledgeable person could make in a second. Measure calibration on your own data before you wire a threshold to anything that spends money. Pin the version.

A model that refuses to talk is a good idea. Whether this one is well calibrated on your problem is a question only your holdout set can answer.