Averil Robertson

If you find yourself asking the question "Why is this message being recognized as intent A and not B?" then you're in the right place. Particularly if your AI agent has been live for a while.

The reason why a AI agent is confused between two Intents is usually that the trained expressions under this pair of intents are overlapping. This is when Confusion Matrix comes in.

Confusion Matrix is a great tool to help you understand if the AI model of your AI agent is performing well in terms of intent recognization, and solve the confusion by moving trained expressions between an Intent pair. It is there to show possible inconsistencies in manually trained expressions.

This article covers the following topics:

• Where is Confusion Matrix
• How to Use Confusion Matrix
  • Highlighted expressions
  • Search for a specific intent
  • Advanced filters
• List of Issues vs. Confusion Matrix

Read more about it in Confusion Matrix Explained.

Where is Confusion Matrix? 

From the left side menu, go to Training Center > Confusion Matrix. Once there, you will see two tabs on top: List of Issues and Confusion Matrix. We suggest starting from List of Issues.

How to use Confusion Matrix?

In List of issues, sort by Priority and start from the high and medium ones. Click Solve Issue > Manage Expressions to get to the expressions management view (see below).

The purpose here is to make sure the expressions under the two intents are not overlapped by moving expressions between intents. They can be moved in bulk or one by one.

With our AI superpower, we have made this easier for you by highlighting the expressions that need to be managed. Here are the options of actionable needed from you after going through the highlighted messages carefully:

• Fix 1: Untrain them or move them to the other intent
• Fix 2: Create a new intent with those expressions
  
• Fix 3: Merge the intents if you realize they should be the same
• Fix 4: Help your AI model learn by training more expressions to those two confused intents. Only do this if you are absolutely certain the two intents are very different and should be separated

Once you're done, click Mark as Solved to keep track of your progress. 

Checkmarks are gone once a new model is trained.

Highlighted expressions

Highlighted expressions are the expressions that confuse your AI model. Expressions are highlighted when:

1. The highlighted expressions were trained to incorrectly
2. The highlighted expressions were trained with the correct intent but their words are very similar to another intent’s expressions.

Search for a specific intent

If you want to know if and how a specific intent causes confusion with other intents, you can search for it in the search bar in the top right corner.

Advanced filters

You define the confusion level here more precisely. In addition, if there is more than one intent that you would like to view, they can all be selected here.

All the features can be accessed via the Confusion Matrix tab as well by clicking on one of the cells 

List of Issues vs. Confusion Matrix

Both List of Issues and Confusion Matrix offer a clear overview of which intents are  "confused", meaning potentially having overlapping expressions. 

The only difference is that List of Issues gives you a clear idea of which ones should be tackled first by automatically sorting the issues based on priority High, Medium, and Low.

Whereas in Confusion Matrix put an emphasis on how well the model is. A good model, like the example below, should have a dark line running across the table diagonally. You can read more about it in Confusion Matrix Explained.

What is the Confusion Matrix?

The Confusion Matrix is a visualization of the performance of an AI model, by measure of its intent recognition. In other words, it shows you if your AI model is able to distinctly recognize similar expressions under different intents, or if it is "confused" by it. 

The Confusion Matrix offers you access to the same tools that first class AI scientists are using, allowing you to easily monitor your own AI agent's performance. 

Every Tuesday night, a new confusion matrix is automatically generated, meaning that you will never be looking at one that is more than  a week old if not manually updated. Generally, this is just fine and not cause for any concern.

If your AI agent has recently undergone extensive training, or has had major alterations made (changes to expressions or intents), we recommend doing a new training sooner rather than later.

To retrain a Confusion Matrix, first navigate to Training Center > Confusion Matrix, and then click Re-train in the top-right corner.

When should I use it?

You should use it when you notice AI agent Answered Rate in Analytics is decreasing, we recommend a baseline of 80%, or when you notice that your AI agent is regularly categorizing messages under the wrong intent.

How to read the Confusion Matrix

The two dimensions, X and Y axis, represent the actual and predicted intents, respectively.

In the example above, column 1 and row 1 are the actual and predicted intent Affirmative, row 2 and column 2 are the intent Negative, so on and so forth. 

The darker a cell is, the more a predicted intent overlap with an actual one.

A good model, like the example above, should have a dark line running diagonally across the table.  

However, what you should pay attention to are the colored cells outside of the diagonal as they represent how often messages in an intent are predicted to another. Whenever you see cells with darker colors outside of the diagonal, this is an indicator of a problem in the accuracy of the AI model caused by potential overlaps of expressions between two intents. These can be tackled in a systematic manner in the List of Issues view where all overlapping issues are listed according to priority automatically. 

Related Articles:

Using the Confusion Matrix to improve an advanced AI agent's performance