Automation Metrics
What are Automation Metrics?
- Purpose: Understanding automation effectiveness across multiple dimensions. The system tracks conversation flows through bot interactions, measures meaningful automated responses, and analyzes the success rate of keeping conversations fully automated versus requiring human intervention. Organizations can identify bottlenecks in their automation pipelines, optimize flow designs, and measure the impact of automation improvements on overall customer service efficiency.
- Value: By providing granular visibility into conversation paths, the analytics enable data-driven decisions about flow optimization, content improvement, and resource allocation. Organizations can identify which automated responses are most effective, which conversation patterns lead to escalation, and how different automation strategies perform across various customer segments and communication channels.
- Scope: Part of Analytics.
Key Concepts
Automation Performance tab is split into three different dashboards:
- The first one regarding the automation performance.
- The second one regarding the responses from the bot triggered from user messages.
- And, lastly, the flows and nodes where the conversations went through.
Automation performance
The automation performance dashboard aggregates conversation data across configurable time periods and analyzes key automation success dimensions, including escalation rates, meaningful interactions, AI usage, fallback occurrences, and channel-specific performance.
It uses a stacked analysis framework that segments conversations into performance categories:
This temporal trending capability enables organizations to monitor automation effectiveness over time, identify performance patterns and seasonal variations, measure the impact of bot improvements, and optimize their conversational AI strategies by understanding where automation succeeds or fails.
Bot response inspector
The bot response inspector section provides detailed, interaction-level analytics of individual bot conversations, capturing the granular metrics of how automated systems handle customer interactions.
It tracks specific bot behaviors including Natural Language Understanding events such as intent detection, knowledge base responses, and fallback occurrences when the bot fails to understand user input.
This granular view enables organizations to analyze the quality and effectiveness of individual automated interactions, identify specific points where conversations break down or succeed, understand which flows and nodes perform best, detect patterns in bot failures or successes, and optimize specific conversation paths.
Conversations flow inspector
The last section provides hierarchical flow performance analytics by tracking how conversations traverse through different automation paths and measuring the distribution of traffic across flows and their individual nodes.
It aggregates data at multiple levels (total, flow-level, and node-level) calculating both absolute conversation counts and percentage distributions to show what portion of conversations follow specific automation paths versus the total traffic.
This hierarchical analysis enables organizations to understand which automation flows are most heavily used, identify bottlenecks or underutilized paths within their conversation designs, measure the effectiveness of specific flow nodes in handling customer interactions, and optimize their conversational architecture by understanding traffic patterns and conversation distribution across their entire automation framework.
How to use it?
To use it, first log into your account:
Navigate into the left-side bar and click the Automation Metrics section.
Make the most of your data! Filters are also available.
Examples or Use Cases
A telecommunications company's customer service bot is experiencing high escalation rates for billing inquiries, with 35% of billing-related conversations transferring to human agents instead of their 15% target.
Using the Bot Response Inspector, the team discovers that when customers say "Why is my bill so high this month?" or "My charges seem wrong," the bot triggers fallback responses because its smart intent detection is not detecting them as "billing_inquiry."
The Conversations Flow Inspector analysis reveals that only 45% of conversations entering the "Billing Support" flow complete successfully, with most customers getting stuck at the "Bill Explanation" node because the bot requires exact billing dates that customers don't typically have available.
By retraining the smart intent model to recognize billing-related keywords and emotional indicators, and redesigning the flow to first show current balances before asking for specific details, the team reduces billing escalations to 18% and improves the "Billing Support" flow completion rate to 78%, demonstrating how granular interaction analysis combined with flow performance data enables targeted bot optimization.
Best Practices
Filters available to subset the whole data.