Interest in and use of chatbots continue to grow among nonprofit organizations as they allow for providing services to and educating constituents at scale in an interactive, engaging way. But once you launch your chatbot and let it interact with users, you will notice that one of its most frequent answers to users is “I’m sorry, I don’t understand”.
This issue doesn't just affect complex bots, or even ones that have Natural Language Understanding (NLU) capabilities. Most social impact organizations use menu-based bots where users select buttons or enter numbers to navigate the chatbot content. Despite the limited spectrum of allowed interactions, users will often try to address the bot with free-text responses that use language the bot was not programmed to handle. These responses, which can comprise as much as 30% of the total interactions with the bot, will trigger a catch-all response (also called a “fallback” or “default” response). Repeatedly getting this response is a frustrating experience for the users, leading them to disengage before they are able to have any meaningful interaction with the chatbot.
Whether you’ve deployed a chatbot or are planning to adopt one, it’s important to develop a strategy for dealing with natural language responses that trigger the catchall in your chatbot. If you collect and analyze them systematically, you can discover quite a lot about your users and their interactions with your bot.
For this post, we’ll focus on a reliable process you can use to work with natural language responses. In a follow-up post, we’ll consider what insights you can get from the data.
How to handle the free-text chatbot responses: A step-by-step guide
You need a clear process to keep your users' messages organized. Our clients saw significant jumps in free responses both after launch and after advertising campaigns. A clear process helps you deal with this volume while keeping you focused on the insights you can gain. At Tangible AI, we use a 5-step process to work through these messages.
Step 1 - Extract
Gather the messages that triggered a catchall response. Natural language responses, basically the responses your users type to your chatbot, can remain hidden if your platform or your organization doesn’t have a way of collecting them. Systems have different capabilities and methods for handling these responses, so it’s important to read your platform’s documentation. Beyond just getting the data, it’s useful to find ways to automate this extraction process to save time and make it easier to repeat.
Step 2 - Cluster
Try to identify groups or clusters of similar queries. Look for common patterns in the messages, similar requests, and synonyms to understand how your users phrase their requests.
Step 3 - Develop Intent Model
Understand what users are trying to do. “Intents” refers to the goal the customer has in mind when typing in a question or a request. The user indicates through intents what they want the chatbot to do, or what information they would like to receive. Your users, whether they are the people your organization serves or your donors and supporters, may make similar requests with different vocabulary, so the synonyms or various sentences you clustered before will help identify these intents. Some messages may relate to existing intents. Or you might get ideas for new intents. During this stage, you need to consider what intents are the most valuable for your users and for your organization’s goals.
Sample intent model.
Step 4 - Label
With the messages clustered and the intents identified, you need to classify the messages. This step involves assigning labels to intents. You should have a clear naming scheme. The consistency in labels will help you keep your bot organized.
Step 5 - Analyze
With the messages organized and labeled, you can analyze the data to find insights into how to use the bot. We will talk extensively about the analysis in our next post. Some of the sample analysis you can carry out:
- What are the most common questions the users ask?
- How are the questions distributed between topics?
- What percentage of users’ messages are spam? How many of the messages relate to navigation?
This 5-step process will help you work systematically with natural language responses. This kind of response may have triggered the catchall, or you may have designed your bot with open questions to elicit user feedback. Regardless, this process organizes these responses so that you can analyze them to get insights into what parts of the chatbot are working and what you could improve. In the next post, we will look at 5 specific insights you can focus your analysis on to find areas for improving your chatbot.
