If you can't measure it, you can't improve it. Chatbottest is an open source guide that helps you identify chatbot's design issues under 7 different categories.
Alma guides you through a short test to get insights about any chatbot. It's a free Chrome extension and it's available to use in Telegram and Facebook Messenger.
Alma only asks 33 questions from a list of 120, a broader model based on our experience. Check the completely open source guide at Github and feel free to collaborate in it.
Based on a heuristic evaluation and our experience, we have created a complete open source guide of instructions and questions. Follow it to test any chatbot design. All these questions will help you find problems in the human-chatbot interaction divided into 7 categories.
Personality
Does the chatbot have a clear voice and tone that fits with the users and with the ongoing conversation?
Onboarding
Are users understanding what is the chatbot about? and how to interact with him from the very beginning?
Understanding
Requests, smalltalk, idioms, emojis... What is the chatbot able to understand?
Answering
What elements does the chatbot send and how well it is doing it? Are they relevant to the moment and context?
Navigation
How easy is to go through the chatbot conversation? Do you feel lost sometimes while speaking with the chatbot?
Error management
How good is the chatbot dealing with all the errors that are going to happen? Is able to recover from them?
Intelligence
Does the chatbot have any intelligence? Is able to remember things? Uses and manages context as a person?