Our small team takes in roughly 80 inbound emails a day, spread across five queues: general support, billing, partnership inquiries, press, and abuse reports. For months, exactly one human — let's call her the saint who ran our inbox — spent close to three hours a day just reading and routing. On a bad week she'd fall behind, and an urgent email would sit unseen for 24 hours. We desperately wanted to automate the routing. We were also terrified of building a system that would confidently dump an abuse report into the partnerships queue and let it rot.
That fear is the whole reason most teams never automate triage. A classifier that's right 90% of the time sounds great until you realize the 10% it's wrong about includes the emails that matter most. A misrouted 'I want to advertise with you' is a shrug. A misrouted 'your user is harassing me' is a disaster. So the bar wasn't 'classify accurately' — it was 'classify accurately AND know when to keep its hands off.'
Writing the classifier
I generated the classifier with the Build an Inbound Email Triage Classifier with Confidence Scoring prompt and ran the result on GPT-4o, which is the right tool here: fast, cheap, and high-volume — I'm classifying 80 emails a day and latency and cost both matter. The hardest input to write was the category definitions. I had to nail down what actually separates a billing question from general support (rule: anything mentioning an invoice, charge, subscription, or payment method is billing). The routing rules mapped each category to a specific Zendesk view.
The output came back with five worked examples baked in, and they covered exactly the cases I was scared of: an email that mentioned both a billing problem AND a bug, a terse one-liner that could be support or abuse, and a Portuguese-language inquiry. Seeing the classifier handle those examples correctly before I deployed it did more for my confidence than any accuracy stat.
Two weeks in production
- 70% of emails route automatically at confidence above 0.8 — our no-human-review threshold
- Abuse reports hit a 98% correct-classification rate, and because key_signals are verbatim quotes, we can audit exactly why every single one was filed where it was
- The coordinator now reviews only the 30% flagged requires_human_review — which really are the genuinely ambiguous ones — and spends the freed-up hours doing actual support
- The suggested_priority field surfaces urgent emails to the top of the queue automatically, so the 24-hour-stale-email problem is just gone
Rolling it out without scaring the team
I didn't flip the whole inbox to automation on day one, because the failure cost was too high and the team was nervous. We ran it in 'shadow mode' for the first week: the classifier ran on every email and logged its decision, but a human still did the actual routing, and at the end of each day we compared. That week of shadow data is what built trust — we could see, concretely, that the agent agreed with the human on the high-confidence emails 96% of the time and that its disagreements clustered exactly where you'd want, in the genuinely murky ones. Only after that did we let it route the high-confidence bucket for real. If you're automating something where mistakes are visible, shadow mode first; it converts a leap of faith into a data-backed decision.
The single most important field
If I could only keep one part of this classifier's output, it would be requires_human_review. A triage system that tries to classify everything and occasionally drops an abuse report in the wrong bucket is actively dangerous. One that says 'this is ambiguous, a human should look' is safe to deploy on day one. The confidence score and the verbatim signals exist to feed that flag — they're the evidence trail behind the decision to act or defer. The whole design philosophy is that the classifier's job isn't to be right about everything; it's to be right about what it's right about and honest about the rest.
This is the one prompt in my agent toolkit I'd genuinely pay for, because it replaced three human-hours a day without ever once mishandling the emails that could've hurt someone. Grab the Build an Inbound Email Triage Classifier with Confidence Scoring prompt on Prompt Dock and run it on GPT-4o against your own queues. I feed it the clean output from my Craft a JSON-Only Data Extraction Prompt That Never Breaks Your Parser prompt when I want both the routing AND the extracted fields in one structured pass.