Analyzing generative reply performance for advanced AI agents

What's my plan?

Add-on

AI agents - Advanced

Generative reply insights are broken down into three key sections:

• Top KPIs
• Knowledge base articles insights
• Knowledge base gaps discovery

Each section is designed to give you insights on your generative reply conversation data so you can make data-driven decisions on where the AI agent is performing well, which topics are bringing you the most inquiries and traffic, as well as the biggest return on your time investment of creating knowledge base articles.

Before we cover the sections, let's showcase the filter criteria that can be defined on the top navigation.

Filters

Filter	Description
Date	Select a time frame
Language	Filters out the conversations in the selected language

About top KPIs

The top KPIs consist of conversations and generative replies.

Conversations

Available only for zero-training and expression-based AI agents.

The Conversations section reports the messaging conversation volumes held by your AI agent. Messaging conversation volumes are broken down by Generative Reply Conversations and Intent-Based Conversations

Widget	Description
Generative Reply Conversations	Conversations that include at least one generative reply or at least one activation of a generative replies block in a dialogue.
Intent-Based Conversations	Conversations designed in the dialogue builder based on intents that didn't include any generative replies.

Generative replies

The Generative replies section reports the rate where a generative replies block yielded a scenario. The section highlights two possible scenarios of generative replies: Response Generated and Not understood.

Widget	Description
Response Generated	The number of times the AI agents generated a response whether small talk or sourced from the imported knowledge base. You can review the breakdown of generative replies by hovering over the metric bar. Small Talk: Total number of generative replies where the answers are not based on the imported KBs. KB Answers: Total number of generative replies that are sourced from the imported KBs.
Not Understood	The metric reviews the number of times the AI agents could not find a relevant context in an imported KB in order to answer a customer inquiry

The breakdown of the metric Response Generated is available from 18th August 2023.

About knowledge base articles insights

The Knowledge base articles section provides more insights on the response generated where the answer was sourced from the knowledge base.

There are two views of the knowledge base articles' insights: the tree map view and the table view.

You can switch between the two views from the button on the top right corner.

Visual	Description
	The tree map view highlights the top most and least used articles by the AI agent to generate a response. You can view the most and least used articles by enabling the quick filters on the top. The tiles are ordered in ascending or descending order based on your filter's selection. Each tile represents an article where: The percentage on the top left corner is the rate where the article was used to generate a response The tile name reflects the article's title More info can be viewed about the article when hovering over it Rate of Usage: the rate at which an article is used to generate a response.  KB Type: the type of the imported knowledge base that the article is part of. KB Name: the name given to the knowledge base when being imported Article URL: external link to the article. Only provided if found
	The table view lists all the imported articles. Each row in the table presents an article where: Article name: is the article title KB Type: the type of the imported knowledge base that the article is part of. KB Name: the name given to the knowledge base when being imported Rate of Usage: the rate at which an article is used to generate a response. Functionalities: Each column can be sorted by clicking on the column header Search for a specific article through the search bar The default max number of articles is 10. You can use the pagination options to suit your preferences

About knowledge base gaps discovery 

Available only for zero-training and expression-based AI agents. Customers using AI agents with agentic AI will be directed to creating use cases from suggestions.

The Knowledge base gaps discovery section is designed to help you find out the topics your AI agent could not find a relevant context in the imported KB(s) in order to answer a customer inquiry. It groups the messages under identified clustered topics where you can make data-driven decisions on the best approach to close these knowledge gaps.

On the left side, the bubble chart presents clusters of your customer messages that were not understood by your AI agent. The bubble chart shows the top 10 topics that are frequently asked by your customers.

The not-understood customer messages are clustered every Sunday at 23:59:59 UTC. Every week at that time, your AI agent will cluster relevant not-understood messages under topics represented by a bubble. You can navigate through the clusters by clicking on the bubble where you view the not-understood messages and the number of messages that belong to the selected cluster.

You can improve the AI agent's knowledge for each topic by either:

1. Creating a new intent through the Intent Assistant feature
   This option is ideal if you have no intent to cover the clustered topic. You can simply follow the steps to create a new Intent, generate expressions for this Intent, and design a flow.
2. Add Messages to an Existing Intent
   If you believe you have an Intent already that covers this topic, you can add more expressions to it and train your model
3. Create an article that covers the missing topic
   You can also create an article in your help center that covers the missing topic.