assessing the performance of a virtual assistant using a pipeline of conversation data and analytics
translation of the article from Rasa's blog, Alana Williams, Greg Stephens
Defining Metrics
You created an assistant, set up intentions, launched it – clients interact with it. But how can you determine that everything was successful? How do you even measure success? How do you measure customer satisfaction or determine that an assistant is working as intended? How to understand what exactly needs to be optimized if it is not working efficiently enough?
“Metrics can be compared to dishes on a menu. They must reflect the satisfaction of needs and desires”
Without metrics and analytics, it's impossible to truly determine whether the work you and your development team are doing is actually helping customers or optimizing business processes. I'm sure you're thinking, “We're already counting the number of conversations, sessions, and users. Is anything else needed?
Great question! We believe that for businesses, it would be correct to evaluate the effectiveness of an assistant’s work specifically from the clients. You should get answers to the following questions:
whether clients use an assistant;
Do you plan to use the assistant again?
does the assistant solve their problems;
whether they switch to operators, as well as the percentage of switches from bot to operator.
Customer reviews are the most important indicators.
Collecting customer feedback: customer satisfaction and NPS (Net Promoter Score) scores
One of the best ways to measure customer satisfaction is through feedback. Currently, the CSAT “Customer Satisfaction Score” metric is widely used – an indicator that demonstrates how satisfied the customer is with the interaction with the company. With its help, you can find out how satisfied customers are with the bot’s work. This could be a simple poll at the end of the conversation:
“How satisfied are you with the assistant’s work?”, “Did the assistant help solve the problem?”
You can choose a more specific question that is relevant to your specific bot use case, but the main idea is to understand customer sentiment and how they evaluate communication with the AI assistant. This is also a great way to let customers talk about their experience with additional notes or comments that will help improve the customer experience.
Another alternative to CSAT is NPS or Net Promoter Score. This is again a follow-up question for clients and users at the end of a conversation, just like a CSAT survey, and can be modified to suit your business strategy, but a common question used is: “How likely are you to recommend us to a friend or colleague?”.
Regardless of the tool you choose—CSAT, NPS, or any other methods of collecting customer feedback—conducting a satisfaction survey is a fundamental step that should be taken to achieve successful metrics.
Customer feedback has been implemented, what else do you need to think about?
Another metric we find useful when reporting on business performance is the Containment Rate. Containment Rate is the percentage of conversations processed exclusively by the assistant rather than transferred to a human. This is an important indicator simply because you need to be sure that customers will not be disappointed with the assistant's answers and will not try to switch to the operator.
Imagine that you spent several months creating an assistant to reduce the need for interaction with operators, only to find that only a small percentage of customers were able to use it successfully. Not a very good return on investment.
However, Containment Rate is only one indicator of success; Abandonment Rate and Escalation Rate must be considered along with it. Abandonment Rate is often thought of as the percentage of calls that customers end prematurely, while Escalation Rate is the percentage of calls that are transferred from an assistant to an agent. Both the Abandonment Rate and Escalation Rate are important because they both provide insight into how well the assistant is answering customer questions: are they wasting time wandering through conversation threads to no avail, or are they forced to contact an operator because the assistant was unable to successfully answer theirs? questions.
This is where Rasa Pro Analytics comes in handy.
Analytics helps you visualize and process Rasa assistant metrics using the tools – BI tools, data warehouses – of your choice. Visualizations and analysis of the assistant's performance and conversations allow you to measure the return on investment and improve its effectiveness over time.
Rasa Pro Analytics stores chatbot history and conversation data in a data warehouse of your choice (PostgreSQL, Redshift, BigQuery or Snowflake). Data warehouse and BI tools can be used to report on key performance indicators such as Containment Rate, Abandonment Rate and Escalation Rate.
Charts and dashboards are implemented using Metabase and Tableau. This makes it possible to extract and visualize custom KPIs not available in off-the-shelf solutions. By analyzing these metrics over time, you can identify trends and patterns that can help improve your overall user experience.
Rasa Pro Analytics allows you to measure contact-related KPIs across multiple platforms, including call centers, web and mobile apps, by integrating with the data warehouses and Business intelligence tools of your choice.
By identifying your chatbot's weaknesses, you can take steps to improve performance. For example, if you notice that users are trying to move the conversation to the operator or leaving the conversation prematurely, you can improve Natural Language Processing and conversations to keep users highly engaged and prevent them from leaving the conversation.
Additionally, it's important to know how well your bot's specific conversation threads are performing so you can improve it in key areas. For example, in a bank chatbot you might have a thread that allows you to transfer funds between accounts. To do this, the user must indicate the sender and account, as well as the transfer amount. With Rasa Pro Analytics, we can measure the completion rate of key threads by calculating the percentage of users who started and successfully completed the process.
Finally, tracking key performance indicators can help you evaluate your chatbot's ROI. By understanding how a chatbot performs in terms of user engagement and satisfaction, you can determine the value it provides. If you're investing time and resources into developing and maintaining your own assistant, it's important to be able to measure its impact on the business and demonstrate its value to stakeholders.
Our company page chat platform Rasa
