Step 8: Monitoring and Analyzing a Chatbot with Rasa X

In this blog, you will learn how to use Rasa X, a tool for monitoring and analyzing chatbots, to track your chatbot’s performance and user feedback and improve it over time.

Table of Contents

1. Introduction

In this blog, you will learn how to use Rasa X, a tool for monitoring and analyzing chatbots, to track your chatbot’s performance and user feedback and improve it over time.

Rasa X is a web-based application that allows you to interact with your chatbot, view and annotate conversations, test and improve your NLU and dialogue models, and analyze your chatbot’s metrics and trends.

By using Rasa X, you can gain valuable insights into how your chatbot is performing, what your users are asking, and how you can make your chatbot more engaging and effective.

Whether you are a beginner or an expert in chatbot development, Rasa X can help you take your chatbot to the next level.

Are you ready to get started? Let’s dive in!

2. What is Rasa X and Why You Need It

Rasa X is a web-based application that works with Rasa Open Source, a framework for building conversational AI. Rasa X enables you to monitor and analyze your chatbot’s conversations with real users, and use the insights to improve your chatbot’s performance and user experience.

Why do you need Rasa X? Because building a chatbot is not a one-time task, but an ongoing process. You need to constantly test, evaluate, and update your chatbot to make sure it meets your users’ needs and expectations. Rasa X helps you do that by providing you with tools to:

Interact with your chatbot and see how it responds to different inputs and scenarios.
View and annotate your chatbot’s conversations with real users, and identify what works well and what needs improvement.
Test and improve your NLU and dialogue models, and see how your changes affect your chatbot’s behavior and accuracy.
Analyze your chatbot’s metrics and trends, such as the number of conversations, the average conversation length, the intent distribution, the action frequency, and more.
Collect and incorporate user feedback, such as ratings, comments, and suggestions, to make your chatbot more engaging and helpful.

With Rasa X, you can make your chatbot smarter, faster, and more user-friendly.

3. How to Install and Set Up Rasa X

In this section, you will learn how to install and set up Rasa X on your local machine. You will need to have Rasa Open Source installed and a chatbot project ready to connect to Rasa X. If you don’t have these, you can follow the previous steps of this tutorial series to create a simple chatbot using Rasa Open Source.

To install Rasa X, you can use the following command in your terminal:

pip3 install rasa-x --extra-index-url https://pypi.rasa.com/simple

This will install the latest stable version of Rasa X and its dependencies. You can also specify a specific version of Rasa X by adding the version number after the package name, such as rasa-x==0.42.0.

To set up Rasa X, you need to initialize it in your chatbot project directory. You can do this by using the following command in your terminal:

rasa x

This will launch Rasa X in your default browser and open the login page. You will need to create a username and a password for your Rasa X account. You can also use the --rasa-x-port option to specify a different port for Rasa X, such as rasa x --rasa-x-port 5002.

Once you log in, you will see the Rasa X interface, which consists of several tabs and menus. You will use these to monitor and analyze your chatbot’s conversations, test and improve your NLU and dialogue models, and collect and incorporate user feedback.

Congratulations, you have successfully installed and set up Rasa X! Now you are ready to connect your chatbot to Rasa X and start using its features.

4. How to Connect Your Chatbot to Rasa X

Now that you have installed and set up Rasa X, you need to connect your chatbot to it. This will allow you to interact with your chatbot, view and annotate its conversations, and test and improve its models using Rasa X.

To connect your chatbot to Rasa X, you need to follow these steps:

Make sure your chatbot project directory contains the following files and folders: actions.py, config.yml, credentials.yml, domain.yml, endpoints.yml, models, and data.
In your terminal, navigate to your chatbot project directory and run the following command: rasa x. This will start Rasa X and open it in your default browser.
In the Rasa X interface, click on the Models tab and select the model you want to use for your chatbot. You can also upload a new model or train a new model from the Rasa X interface.
Click on the Share your bot button on the top right corner of the Rasa X interface. This will generate a link that you can use to share your chatbot with others. You can also customize the link by adding a name and a description for your chatbot.
Copy the link and paste it in a new browser tab or window. This will open a chat widget where you can interact with your chatbot and see how it responds to different inputs and scenarios.

You have successfully connected your chatbot to Rasa X! Now you can use the Rasa X features to monitor and analyze your chatbot’s conversations, test and improve your NLU and dialogue models, and collect and incorporate user feedback.

5. How to Use Rasa X Dashboard

The Rasa X Dashboard is the main interface where you can monitor and analyze your chatbot’s conversations, test and improve your NLU and dialogue models, and collect and incorporate user feedback. The dashboard consists of several tabs and menus that provide different features and functionalities. In this section, you will learn how to use the Rasa X Dashboard and what each tab and menu does.

The Rasa X Dashboard has the following tabs and menus:

Conversations: This tab shows you the conversations that your chatbot has with real users. You can view and annotate the conversations, filter them by date, intent, action, or rating, and download them as JSON files. You can also replay the conversations and see how your chatbot responds to different inputs and scenarios.
NLU Inbox: This tab shows you the messages that your chatbot receives from real users that are not classified by any of your existing intents. You can review and annotate the messages, assign them to existing or new intents, and retrain your NLU model with the updated data.
Response Selector: This tab shows you the messages that your chatbot receives from real users that are classified by your response selector intents. You can review and annotate the messages, assign them to existing or new responses, and retrain your dialogue model with the updated data.
Story Conflicts: This tab shows you the conflicts that occur in your stories when your chatbot encounters different paths for the same user input. You can review and resolve the conflicts, merge or delete the conflicting stories, and retrain your dialogue model with the updated data.
Analytics: This tab shows you the metrics and trends of your chatbot’s performance and user feedback. You can see the number of conversations, the average conversation length, the intent distribution, the action frequency, the user ratings, the user comments, and more. You can also filter the data by date, intent, action, or rating, and download them as CSV files.

By using the Rasa X Dashboard, you can monitor and analyze your chatbot’s conversations, test and improve your NLU and dialogue models, and collect and incorporate user feedback. You can also use the Test Your Assistant button on the top right corner of the dashboard to test your chatbot’s performance and accuracy with different inputs and scenarios.

5.1. Conversations

The Conversations tab in the Rasa X Dashboard shows you the conversations that your chatbot has with real users. You can use this tab to monitor and analyze your chatbot’s performance and user feedback, and to improve your chatbot’s NLU and dialogue models.

In the Conversations tab, you can see a list of conversations that your chatbot has had with different users. Each conversation has a unique ID, a date and time, a rating, and a comment. You can click on any conversation to view its details, such as the user input, the chatbot output, the intent, the confidence, the action, and the entities.

You can also do the following things in the Conversations tab:

Filter: You can filter the conversations by date, intent, action, or rating, to see only the conversations that match your criteria. For example, you can filter by rating to see the conversations that received positive or negative feedback from the users.
Annotate: You can annotate the conversations by correcting the intent, the entities, or the action, if they are wrong or missing. For example, you can correct the intent if the chatbot misclassified the user input, or add an entity if the chatbot missed it. You can also add comments to the conversations to provide additional feedback or notes.
Download: You can download the conversations as JSON files, to save them for future reference or analysis. You can download all the conversations, or only the filtered ones, by clicking on the download button on the top right corner of the tab.
Replay: You can replay the conversations by clicking on the replay button on the top right corner of the tab. This will open a chat widget where you can interact with your chatbot and see how it responds to the same inputs and scenarios as the original conversations.

By using the Conversations tab, you can monitor and analyze your chatbot’s conversations, test and improve your NLU and dialogue models, and collect and incorporate user feedback.

5.2. NLU Inbox

The NLU Inbox is a feature of Rasa X that allows you to review and annotate the user messages that your chatbot received. By using the NLU Inbox, you can improve your chatbot’s understanding of natural language and make it more accurate and robust.

Why do you need the NLU Inbox? Because your chatbot may not always understand what your users are saying, especially if they use different words, phrases, or expressions than what you trained your chatbot with. The NLU Inbox helps you identify and correct these cases by showing you the user messages that your chatbot failed to classify correctly or confidently.

How do you use the NLU Inbox? It’s simple. You just need to follow these steps:

Go to the NLU Inbox tab on the Rasa X Dashboard.
Look at the user messages that are displayed on the screen. You can filter them by date, intent, or confidence level.
Select a user message that you want to review. You will see the intent and entities that your chatbot assigned to it, as well as the confidence score.
If the intent and entities are correct, click on the Accept button. This will add the user message to your NLU training data and improve your chatbot’s accuracy.
If the intent and entities are incorrect or incomplete, click on the Edit button. This will allow you to change the intent and entities, or add new ones. You can also delete the user message if it is irrelevant or spam.
Repeat the process for the other user messages that you want to review.

By using the NLU Inbox, you can make sure that your chatbot understands your users better and provides them with the best possible responses.

5.3. Response Selector

The Response Selector is a feature of Rasa X that allows you to test and improve your chatbot’s ability to handle retrieval-based responses. Retrieval-based responses are predefined responses that your chatbot can select from a list based on the user’s intent and entities. For example, your chatbot may use retrieval-based responses to answer frequently asked questions, provide chitchat, or handle fallback scenarios.

Why do you need the Response Selector? Because your chatbot may not always select the most appropriate response for the user’s query, especially if the query is ambiguous, complex, or out of scope. The Response Selector helps you identify and correct these cases by showing you the user queries that your chatbot received and the responses that it selected from the list.

How do you use the Response Selector? It’s easy. You just need to follow these steps:

Go to the Response Selector tab on the Rasa X Dashboard.
Look at the user queries that are displayed on the screen. You can filter them by date, intent, response, or confidence level.
Select a user query that you want to review. You will see the response that your chatbot selected from the list, as well as the confidence score.
If the response is correct, click on the Accept button. This will confirm that your chatbot selected the right response and improve its confidence.
If the response is incorrect or irrelevant, click on the Edit button. This will allow you to change the response, or add a new one to the list. You can also delete the user query if it is irrelevant or spam.
Repeat the process for the other user queries that you want to review.

By using the Response Selector, you can make sure that your chatbot provides the most relevant and helpful responses to your users.

5.4. Story Conflicts

The Story Conflicts is a feature of Rasa X that allows you to detect and resolve inconsistencies in your chatbot’s dialogue management. Story conflicts occur when your chatbot predicts different actions for the same conversation path, leading to unpredictable and undesired behavior.

Why do you need the Story Conflicts? Because your chatbot may not always follow the optimal dialogue flow that you designed, especially if you have complex or overlapping stories, rules, or forms. The Story Conflicts helps you identify and correct these cases by showing you the conversation paths that your chatbot learned and the actions that it predicted for each step.

How do you use the Story Conflicts? It’s simple. You just need to follow these steps:

Go to the Story Conflicts tab on the Rasa X Dashboard.
Look at the conversation paths that are displayed on the screen. You can filter them by date, intent, action, or conflict level.
Select a conversation path that you want to review. You will see the actions that your chatbot predicted for each step, as well as the confidence score and the source of the prediction (story, rule, or form).
If the actions are consistent and correct, click on the Accept button. This will confirm that your chatbot followed the right dialogue flow and improve its confidence.
If the actions are inconsistent or incorrect, click on the Edit button. This will allow you to change the actions, or add new ones to the conversation path. You can also delete the conversation path if it is irrelevant or erroneous.
Repeat the process for the other conversation paths that you want to review.

By using the Story Conflicts, you can make sure that your chatbot behaves consistently and reliably in every conversation.

5.5. Analytics

The Analytics is a feature of Rasa X that allows you to measure and visualize your chatbot’s performance and user feedback. Analytics helps you understand how your chatbot is performing, what your users are asking, and how you can optimize your chatbot’s user experience.

Why do you need Analytics? Because your chatbot’s performance and user feedback are crucial indicators of your chatbot’s success and value. Analytics helps you monitor and improve these indicators by providing you with data and insights on:

The number and quality of conversations that your chatbot had with real users.
The distribution and frequency of intents and actions that your chatbot recognized and executed.
The accuracy and confidence of your chatbot’s predictions and responses.
The ratings and comments that your users gave to your chatbot’s responses.
The trends and patterns that emerge from your chatbot’s conversations and user feedback.

How do you use Analytics? It’s easy. You just need to follow these steps:

Go to the Analytics tab on the Rasa X Dashboard.
Look at the charts and graphs that are displayed on the screen. You can filter them by date, intent, action, or rating.
Select a chart or graph that you want to analyze. You will see the data and insights that it provides, such as the number of conversations, the average conversation length, the intent distribution, the action frequency, the prediction accuracy, the response rating, and more.
If the data and insights are satisfactory, click on the Download button. This will allow you to export the data and insights as a CSV or PDF file for further analysis or reporting.
If the data and insights are unsatisfactory, click on the Improve button. This will take you to the relevant feature of Rasa X, such as the NLU Inbox, the Response Selector, or the Story Conflicts, where you can review and improve your chatbot’s performance and user feedback.
Repeat the process for the other charts and graphs that you want to analyze.

By using Analytics, you can make your chatbot more effective, efficient, and engaging for your users.

6. How to Improve Your Chatbot with Rasa X

Now that you have learned how to use Rasa X dashboard to monitor and analyze your chatbot’s conversations, you might be wondering how to use this information to improve your chatbot. In this section, we will show you some tips and best practices on how to use Rasa X to make your chatbot more accurate, reliable, and engaging.

One of the main benefits of Rasa X is that it allows you to collect and incorporate user feedback into your chatbot’s development. User feedback can come in different forms, such as:

Ratings: You can ask your users to rate your chatbot’s responses on a scale of 1 to 5, or use emojis to express their satisfaction or dissatisfaction.
Comments: You can ask your users to leave comments or suggestions on how to improve your chatbot’s responses or functionality.
Corrections: You can ask your users to correct your chatbot’s mistakes, such as misclassified intents, wrong actions, or irrelevant responses.

You can use the Conversations tab in Rasa X dashboard to view and manage the user feedback. You can filter the conversations by ratings, comments, or corrections, and see how your users are interacting with your chatbot. You can also use the NLU Inbox tab to review and annotate the user messages that your chatbot did not understand or classify correctly.

By using user feedback, you can identify the strengths and weaknesses of your chatbot, and make the necessary changes to improve it. For example, you can:

Add more training data or synonyms to your NLU model to increase its accuracy and coverage.
Modify or add more rules or stories to your dialogue model to handle more complex or diverse scenarios.
Update or add more responses or actions to your domain file to make your chatbot more informative or helpful.
Use custom actions or integrations to add more functionality or features to your chatbot, such as booking a reservation, sending an email, or connecting to an API.

After making any changes to your chatbot, you can use Rasa X to test and evaluate the impact of your changes. You can use the Response Selector tab to see how your chatbot selects the best response for each user message. You can use the Story Conflicts tab to see if there are any inconsistencies or conflicts in your dialogue model. You can use the Analytics tab to see how your chatbot’s metrics and trends change over time, such as the number of conversations, the average conversation length, the intent distribution, the action frequency, and more.

By using Rasa X to test and evaluate your chatbot, you can ensure that your changes are effective and do not introduce any errors or bugs.

Rasa X is a powerful tool that can help you improve your chatbot with user feedback, testing, and evaluation. By using Rasa X, you can make your chatbot more accurate, reliable, and engaging, and provide a better user experience.

7. Conclusion

In this blog, you have learned how to use Rasa X, a tool for monitoring and analyzing chatbots, to track your chatbot’s performance and user feedback and improve it over time.

You have learned how to install and set up Rasa X, how to connect your chatbot to Rasa X, and how to use Rasa X dashboard to interact with your chatbot, view and annotate conversations, test and improve your NLU and dialogue models, and analyze your chatbot’s metrics and trends.

You have also learned how to use user feedback, testing, and evaluation to identify and fix the issues and gaps in your chatbot, and make it more accurate, reliable, and engaging.

By using Rasa X, you can make your chatbot smarter, faster, and more user-friendly, and provide a better user experience.

We hope you enjoyed this blog and found it useful. If you have any questions or comments, please feel free to leave them below. We would love to hear from you!

Thank you for reading and happy chatbot building!