Handoff classifier — sample run
Made-up results for a fictional coffee subscription's support bot. The classifier answers one question: should this message go to a person? Each case was run 5 times.
10 cases · 50 runs · 2026-11-08
Accuracy
86%
Share of all runs that gave the right answer.
Over-fire
16%
Sent to a person when the bot could have handled it.
Under-fire
12%
Kept with the bot when it should have gone to a person.
Flaky cases
5 of 10
Cases where the runs didn't all agree.
Why run each case more than once?
AI answers can change from run to run. A single run of these cases could look perfect by luck. Running each one 5 times shows which answers are stable and which ones flip.