Tag: Calibration

A confidence number is only worth what it lets you trust. I put nine decision models, open and hosted, through one test: keep only the answers each was 90% sure of, and score them, first on 111 hard AI-written questions and then on 1,599 questions labelled by 100 people each. Some earned the trust. Here is which, and under what conditions.

We keep asking chat models for decisions, then parsing the prose back into something code can branch on. TypeSafe's Jev skips the prose. I spent a few days reading the public material, then got access and measured it myself. Fast and accurate held up. On my two prompts, calibration error came out near 0.09, and a plain open model on my own GPU was calibrated at least as well.