Automated chat can reply quickly, while a human team can handle exceptions and judgment-heavy requests. To decide how to combine them, compare the work they resolve. A faster greeting or fewer tickets routed to people does not establish that customers received better service.
Define the unit you are measuring
Choose whether a “case” means one conversation or one customer issue, and document how reopened conversations count. An order question that moves from chat to email should not automatically become two successful resolutions. Use the same rules across automated and human queues.
Group cases by issue type and difficulty. Comparing simple tracking questions handled by automation with complex complaints handled by people would make the headline numbers misleading. The AI agents and chatbots comparison provides useful capability tests before a live pilot.
Track five measures together
- Meaningful first response: time until the customer receives relevant help, excluding a receipt acknowledgment. State whether the clock includes non-business hours.
- Resolution time: elapsed time until the issue reaches your defined completed state. Also inspect the slowest cases; an average can hide a long tail.
- Repeat-contact rate: the share of resolved issues that return within a stated window. Keep the window consistent.
- Customer feedback: the response and the response rate, segmented by issue type. A small, self-selected sample should not stand in for every customer.
- Cost per resolved issue: attributable software, usage, staffing, management and review cost divided by completed issues over the same period.
Use our customer support KPI guide to align these definitions with the rest of your reporting.
Separate automation outcomes
Record correct answers, successful actions, appropriate handoffs and unsuccessful attempts. “Contained” can mean the conversation never reached a person; it does not necessarily mean the request was resolved. Review a sample of contained conversations to check the outcome.
Likewise, an escalation is not always a failure. A request outside the approved refund policy may belong with a supervisor. Judge whether the system recognized the boundary and transferred useful context rather than rewarding it for avoiding every handoff.
Run a comparison with similar cases
Start with a defined set of request types. Where practical, allocate comparable cases to the pilot and current workflow, using the same hours, policies and completion criteria. Record promotional periods, outages and fulfillment delays that may affect the comparison.
Review both accuracy and effort. A reply that looks correct may still leave an agent to redo the work. Include time spent correcting answers, investigating failed actions and maintaining knowledge when judging the pilot.
Work through a simple cost example
Suppose a pilot costs $1,200 for software, review and attributable support time, and resolves 600 issues under your definition. Its cost per resolved issue is $2. If another approach costs $900 and resolves 300 comparable issues, its figure is $3.
These are hypothetical numbers, not industry benchmarks or Chad pricing. Before choosing the first approach, compare correctness, repeat contacts and customer feedback. A low initial cost is less useful if the issue returns to the queue later.
Decide what should expand
Keep automation for the categories where it produces accurate, completed work. Improve knowledge where answers are missing, and keep human ownership where requests need discretion. Revisit staffing after measuring the residual workload rather than assuming a fixed percentage of jobs disappears.
If you are piloting Chad, agree on the supported workflows and a review sample before expanding. Pair the pilot with first-contact resolution exercises so the team can see why a case was resolved, escalated or reopened. The result should be a support mix you can explain and operate, not just a better-looking dashboard.
