"Is my chatbot actually helping?" comes down to five chatbot metrics. Containment rate: the share of conversations the assistant finished without a person. Handoff rate, and the reason for each handoff. Booked appointments or qualified leads per week, counted in your calendar or CRM, not in the chat tool. Wrong-answer rate, from reading twenty transcripts a week. Customer satisfaction, from a one-question follow-up by text or email. The first month's numbers are a baseline, not a verdict; you need a second month to see which way they move. And read the five together. An assistant with a high containment rate and a high wrong-answer rate is worse than one that hands off, because it loses customers quietly and in a confident voice.
Still deciding whether to have an assistant at all? Start with what a chatbot can and can't do.
The five chatbot metrics, one at a time
1. Containment rate
What it is: the share of conversations the assistant finished with no handoff to a person: 70 of 100 with nobody stepping in is 70 percent.
Where to get it: most chat tools show it, often as "resolved" or "automated". Check the definition: some count a customer who gave up and closed the window as contained. It measures how often the assistant did not ask for help, not how often it helped.
A worrying reading: either extreme. Very low means it hands off almost everything. Very high, with a wrong-answer rate to match, means it is answering things it should not.
The usual fix: for low containment, gaps in the material or handoff rules that fire too early; for suspiciously high, tighter rules about what it must refuse.
2. Handoff rate, and the reason
What it is: the share of conversations passed to a person, and why. The reason is the useful part: asked for a human, urgent language, outside the material, or not understood.
Where to get it: the same dashboard shows the rate. The reason is usually a tag attached at handoff; if not, read the handoffs, which are short.
A worrying reading: many handoffs for one reason. Ten a week about a service you offer but never wrote down is a gap in the material, not a limit of the software. Handoffs into an inbox nobody opens until Monday are worse than none.
The usual fix: sort by reason and fix the biggest one. How an assistant qualifies leads covers what a good handoff contains.
3. Booked appointments or qualified leads per week
What it is: real appointments on your calendar, or leads in your CRM, that started in chat. This is the number the assistant exists to move.
Where to get it: not from the chat tool. Count in the calendar or CRM, where the booking lives; most can tag the source, with the assistant's conversation summary in your inbox as backup.
A worrying reading: many conversations, few bookings. The assistant is not offering to book, the link breaks on a phone, or the widget sits where people are not ready yet.
The usual fix: walk the booking path yourself on a phone, then check that the assistant offers the next open slot rather than waiting to be asked.
4. Wrong-answer rate
What it is: the share of answers that were wrong: an old price, the wrong hours, a service you do not offer, or medical or legal advice it should have refused.
Where to get it: by reading. Pick twenty transcripts a week, mark each answer right, wrong or correctly refused, and divide. No dashboard measures this, which is why so many owners never see it.
A worrying reading: our rule of thumb: one wrong answer in twenty needs fixing this week; more than that, and the assistant should hand off more until the material is corrected. A wrong price or policy counts double, because the customer will hold you to it.
The usual fix: almost always the material. Write what you would have said, update the document, and retest the question.
5. Customer satisfaction
What it is: whether the person got what they came for, in their words rather than yours.
Where to get it: a one-question follow-up by text or email: "Did you get what you needed? Yes or No." A rating scale gets fewer replies and tells you less.
A worrying reading: most will not reply; that is normal. Watch the share of No among those who do, and read every No transcript. A No after a conversation the dashboard called contained means containment was lying.
The usual fix: the No transcripts tell you: usually a wrong answer, a missing handoff, or a dodged question.
Two numbers that mislead
Total conversations rises when the widget pops open on every page, when the opening message is pushy, and when people are confused. None of that is help. Rising conversations with flat bookings is a reason to look harder.
Average response time was a live-chat number: how long a customer waited for a person. An AI assistant replies in a second whether it is right or wrong. A fast wrong answer is still wrong.
A worked example
Say a med spa's assistant handled 120 conversations last month. These are made-up round numbers, not results from a client.
Say 90 ended without a handoff, so containment is 75 percent. The other 30 were handed off: 12 because the customer asked for a person, 10 about a treatment the material never mentioned, and 8 about a package price the spa does not publish. The calendar shows 14 consultations that started in chat. The owner read 80 transcripts and marked 8 answers wrong, 5 of them quoting last season's price for one package. Of 40 replies to the follow-up text, 6 said No.
Read together: one answer in ten is wrong, mostly about one price, so the fix is a single line in the price list. The 10 handoffs about the unlisted treatment are a material gap; add it and they become contained conversations. The 8 price handoffs are correct, since the spa chose not to publish. The 14 consultations are the number to beat next month, after the two fixes.
The weekly 20-minute review
- Write this week's containment and handoff rates next to last week's. Two minutes.
- Tag each handoff with its reason. Five minutes.
- Count bookings or leads from chat in the calendar or CRM. Three minutes.
- Read twenty transcripts, newest first, marking each answer right, wrong or correctly refused and noting what you would have said for each wrong one. Eight minutes.
- Find the transcripts behind every No reply. Two minutes.
- Pick the one fix that covers the most conversations and send it to whoever maintains the assistant. One a week, so you can see what each one did.
What to change when a number is bad
Four levers, and the number tells you which to pull.
- Material: wrong answers about facts (prices, hours, services, policies). Update the document the assistant answers from.
- Rules: it answers things it should refuse, or refuses things it should answer. Change what it may do and what it says when it does not know.
- Handoff: handoffs fire too late, too often, or to the wrong place. Change the triggers and the destination.
- Placement: plenty of conversations, few bookings, or a widget covering the call button on phones. Move it, or change when it opens.
Switch it off when the wrong-answer rate stays high after two rounds of fixes, when it has given advice it must never give, or when a customer has been misled about money. Put a contact form in its place, fix, retest, relaunch. A week offline costs less than a month of confident wrong answers.
How Deca Digital helps
We build AI assistants with the five numbers in place from day one: handoffs tagged by reason, bookings tagged by source, the follow-up question sent automatically, and a month of reading every transcript with you. After that the weekly review is yours and the material changes are ours. We are based in Celina, Texas, work across North Texas, and build remotely for everyone else.
If you already have an assistant and are not sure it is helping, book a free discovery call and bring last month's dashboard.
Frequently asked questions
What is a good containment rate for a chatbot?
There is no fixed figure that applies to every business, and anyone who gives you one without asking about your wrong-answer rate is measuring the wrong thing. A business that publishes its prices and takes simple bookings can expect higher containment than one where most questions need judgment. Our rule of thumb is to watch the direction rather than the level: containment rising alongside bookings is good, and containment rising while bookings stay flat means the assistant is ending conversations rather than helping.
How many chatbot transcripts should I read each week?
Twenty is enough to see the pattern once the assistant is past its first month. Fewer, and a single bad conversation sways the number too much; more, and the routine stops happening. Read them newest first, mark each answer right, wrong or correctly refused, and write what you would have said for every wrong one. In the first month read as many as you can, because that is when the gaps in the material show up fastest.
Should I switch my chatbot off if the numbers are bad?
Sometimes, and it is not a failure. Switch it off when the wrong-answer rate stays high after two rounds of fixes, when it has given medical, legal or financial advice it should have refused, or when a customer has been misled about a price. Put a contact form in its place, fix the material and rules, retest against the questions it got wrong, and relaunch. A week offline costs less than a month of confident wrong answers.