Lesson 04 · Jev 101
Confidence: the difference between a demo and automation
A typed decision is still a guess. Production systems need a control signal: when to act, when to ask a human, when to call a bigger model.
Why prompted “I’m 90% sure” is a weak control
TypeSafe argues RLHF-trained chat models tend to beoverconfident and inconsistent even when you ask them for a confidence field. If a model is right 95% of the time but won’t tell you when it’s in the 5%, you can’t safely automate. TheWhat’s the difference? spells out that contrast.
TypeSafe AI homepage · fetched Sep 20, 2026 · Source“Yet RLHF creates inherent issues such as mode dropping, overconfidence, and lack of reliability. These flaws mean that LLMs require humans-in-the-loop.”
What Jev gives you (company claim)
Every Choice / Score decision ships with probability and confidence estimates trained under RLCD for calibration: higher confidence should mean higher accuracy. Official nuance: calibration is a property across many predictions, not a guarantee on one answer.
Noul is different: official API concepts say Noul has no separate confidence field. The probability is the uncertainty signal. Design thresholds on P(yes) directly.
Your job: validate calibration on your labeled data, then set thresholds.
Secondary Armin Ronacher via TechCrunch · Sep 18, 2026 · Source“if this only comes back with 50% probability, maybe this is a coin toss, and I disregard it. But if it's 95%, sure, then I can do something with it.”
Product design habit
- Pick the decision (Choice / Score / Noul)
- Choose thresholds (e.g. auto if confidence ≥ X and P(yes) ≥ Y)
- Middle band → human or LLM escalate
- Log outcomes; recalibrate
- Pin the model version you tuned against (
jev-1.13.0, not a moving alias)
| Band | What your code should do |
|---|---|
| High confidence, allowed action | Auto-act. Log it. |
| Middle / ambiguous | Escalate to a human, or to a larger generative model if you need language. |
| Low confidence | Do not act. Official guidance: escalate. Don’t paper over it with another chat prompt. |
| High stakes (money, safety, legal) | Human confirm even when the model is sure. Habit pair #34 exists for a reason. |
ChatGPT bridge line: stop asking the chat model tobe the policy. Use a decision model for the policy signal; keep chat for the human-facing words.
What confidence does not fix
- Wrong labels inside a valid schema
- Math, dates, and counts (do those in code)
- Adversarial or injected state
- CJK accuracy gaps — watch the numbers extra carefully