Here's a fun failure I've watched smart teams walk into, including my own. You build an AI classifier, you ask it for a confidence score, and it dutifully returns 0.95 on every single email — the clear ones, the ambiguous ones, the gibberish, all 0.95. It looks confident. It is, in fact, performing confidence. And a confidence score that's always high is exactly as informative as no confidence score at all, except worse, because now you trust it. I call this confidence theater, and it quietly poisons every 'route only if confidence is high' rule you build on top of it.
Why models default to fake confidence
Left to their own devices, language models are people-pleasers. Ask 'how confident are you?' and the model reads it as 'please reassure me,' so it reassures you. Without an explicit definition of what the number means and explicit instruction that low confidence is a CORRECT and desirable output for ambiguous input, you get a flat 0.95 wall. The fix isn't a better model. It's a prompt that defines confidence as a real signal and rewards honesty about uncertainty.
There's also a subtler reason the wall forms. If your prompt only ever USES the confidence score implicitly — you ask for it but nothing in the instructions depends on it — the model has no reason to make it meaningful. The number is just decoration it's been asked to produce, so it produces a pleasant-looking one. The moment you tie a concrete consequence to the score (do X when it's high, do Y when it's low), the model has to actually reason about where the line is, and the numbers start carrying information. Confidence with no downstream consequence is always theater.
How the triage prompt breaks the theater
I rebuilt our classifier with the Build an Inbound Email Triage Classifier with Confidence Scoring prompt on GPT-4o, and the generated prompt did three specific things that forced honest numbers out of it. First, it stated outright that 'a classifier that returns 0.95 for everything is worthless' — putting the failure mode in the prompt itself. Second, it tied a low confidence score to a concrete downstream action (the secondary_category appears only when confidence is below 0.7), so the number had to mean something. Third, it required verbatim key_signals, which means the model has to point at actual evidence — and it's much harder to fake 0.95 confidence when you also have to show your receipts.
- Confidence scores spread out into a real distribution — clear emails landed at 0.9+, genuinely mixed ones at 0.5-0.65, exactly where you'd want a human to look
- Because the secondary_category only populates below 0.7, that field became a built-in 'I'm torn between these two' signal we could act on
- The verbatim-signals rule made every score auditable — if a 0.9 was attached to weak evidence, it stuck out immediately in review
- Spam and gibberish reliably came back low-confidence with requires_human_review = true instead of getting a fake-confident bogus category
Calibration: the part nobody does and everybody should
Here's the test that separates an honest classifier from a theatrical one, and it takes about twenty minutes. Pull a hundred classified emails. Group them by the confidence the model reported — the 0.9-1.0 bucket, the 0.7-0.9 bucket, and so on. Now check what fraction of each bucket the model actually got right. A well-calibrated classifier gets roughly 90% of its 0.9-confidence calls correct and roughly 60% of its 0.6-confidence calls correct. If your 0.9 bucket and your 0.6 bucket have the same accuracy, the number is decorative. When I ran this on the theatrical version, every bucket was about 80% accurate regardless of stated confidence — proof the score meant nothing. After the rebuild, the buckets separated cleanly, and that separation is the only reason a 'route if above 0.8' rule is safe to trust.
Test it the cynical way
If you don't have time for the full calibration check, do the lazy version: run the classifier on a deliberately ambiguous email and look at the number. If it says 0.95 on something you yourself can't confidently classify, the confidence is theater and the whole pipeline is built on sand. A good classifier should be a little bit unsure about the things that are actually unsure. That honesty is the entire value of the score, and it's the thing the prompt has to explicitly teach the model to produce, because the model's default is to reassure you.
Stop trusting numbers your prompt never taught the model to mean. Grab the Build an Inbound Email Triage Classifier with Confidence Scoring prompt on Prompt Dock, run it on GPT-4o, and feed it a few of your nastiest borderline emails on purpose. If it gets appropriately unsure, you've got an honest classifier. If it stays at 0.95, you've got confidence theater — and now you know the difference, and the prompt that fixes it.