On this page
AI support vendors love a headline number: resolution rate, deflection rate, automation rate. These numbers can be useful — or they can quietly count conversations where the customer simply gave up.
Here’s how to measure an AI agent in a way you’d trust if your own job depended on it.
Beware of “deflection”
“Deflected” usually means the customer never reached a person. That includes people who got a great answer — and people who closed the window in frustration. Counted together, the two look identical.
A better question than “how many never reached a person?” is “how many got their problem solved, and how do we know?”
Four numbers worth watching
- Resolved by AI aloneWith a human spot check
- Knowledge base hit rateWhether answers have a source
- Hand-off reasonsThey tell you what to add
- Satisfaction by resolverAI and people kept apart
1. Resolved by AI alone — with a sanity check
Count conversations the AI answered that were marked resolved without a person stepping in. Then sanity-check them: read a random sample of ten each week. Did the answer actually address the question? Was the source right?
AskCub reports how many conversations Cubby handled and how many it resolved alone, so you can watch the trend — and the conversations behind it.
2. Knowledge base hit rate
What share of questions found an answer in your knowledge base at all? A low hit rate isn’t the AI’s fault; it’s a gap in your material. It’s also the number you can move fastest, by adding answers for the most common misses.
3. Hand-off reasons
Every hand-off has a reason: no match, the customer asked for a person, the turn limit, an agent took over. Watch the mix:
- Many “no match” hand-offs → add knowledge.
- Many “customer asked” hand-offs early in conversations → your AI may be answering the wrong things, or the opening isn’t clear about what it can do.
- Many turn-limit hand-offs → answers aren’t landing; read those conversations.
4. Satisfaction, split by who resolved it
Compare ratings of conversations resolved by the AI alone with those resolved by your team. If AI-only ratings are much lower, the AI is closing conversations it shouldn’t. If they’re similar, you can let it carry more.
Don’t compare the AI with an empty chair
The fair comparison isn’t “AI vs a perfect human”. It’s “AI at 11 p.m. vs nobody at 11 p.m.” For after-hours conversations, a correct AI answer in seconds beats a reply tomorrow morning — even if it’s less personal.
That’s why many small teams start with the AI answering only when everyone is offline. The numbers are easier to read, and the value is obvious.
Keep humans’ numbers honest too
When you introduce an AI, your team’s numbers change — they handle fewer, harder conversations. First response time may rise because each conversation is more complex, not because people are slower. Explain this before the dashboard does.
A monthly review template
| Question | Where to look |
|---|---|
| Is the AI resolving more conversations than last month? | AI resolved alone, trend |
| Is it resolving them well? | Ten sampled conversations, CSAT split |
| What is it missing? | Not-answered list, hand-off reasons |
| What did we add? | Knowledge base changes this month |
| Did coverage hours improve? | Busy-hours grid vs office hours |
Measured like this, an AI agent stops being a black box with a flattering number and becomes something you can improve every week.
Related articles
Running a support team2 min read
Five support metrics worth watching every week
Skip the vanity dashboard. These five numbers tell a small team whether customers are getting help — and what to fix next.
Read article
AI customer service4 min read
When should an AI agent hand a conversation to a person?
Five hand-off triggers that keep customers from getting stuck with a bot — and the three mistakes that make AI support feel worse than no support.
Read article
Live chat craft3 min read
How fast is fast enough? First response time in live chat
Visitors on live chat expect a reply in moments, not hours. What a realistic target looks like for a small team — and five ways to hit it without hiring.
Read articleTry it on your own site
A free plan with no time limit, AI agent included. One line of code.