Optimizing your advanced AI agent performance

What's my plan?

Add-on

AI agents - Advanced

You can optimize your advanced AI agent performance by using analytics to improve the AI model and by building better dialogues to improve customer experience.

This article covers the following topics:

• Using analytics to improve the AI model
• Using smarter dialogues to improve user experience

Using analytics to improve the AI model

You can use analytics to improve the AI model, specifically by taking action to improve AI agent understanding and deflection rate.

Other key metrics you might focus on include: AI agent Understood %, AI agent-Handled %, Escalation %, Automation %.

This section covers the following topics:

• Improving advanced AI agent understanding
• Improving deflection rate

Improving advanced AI agent understanding

You can use the confusion matrix and intents to improve advanced AI agent understanding.

• Review the confusion matrix to understand where there is confusion between intents. The matrix shows whether your AI model is able to distinctly recognize similar expressions under different intents or whether it is confused by them.

  For example, consider "My account has been locked" and " I'm locked out of my account." The first expression is about the company locking the account due to too many sign in attempts, while the second is about the customer forgetting their password or user name.

  You can remove confusion by moving expressions or training intents. If there is a lot of confusion between two intents, you might merge them into one or use conditional blocks to guide users through different paths.

  See Using the confusion matrix to improve advanced AI agent performance.

• Ensure you have enough expressions in your intents. Chat AI agents should have 50-300 and ticket AI agents should have 80-300. The number of expressions should reflect the frequency of use for the intent. Make sure your intent structure is smart by getting rid of unused intents if those exist.

  You can filter "not understood" in the conversation logs to understand whether it’s a training issue (meaning the intents exist but they aren't being recognized) or whether you are missing intents for common issues and need to create them.

  For ticket AI agents, keep in mind the noisiness of incoming messages, which might include forwarded messages, signatures, disclaimers, and so on. You can use entities to sanitize emails so that the "noise" won't be considered in the AI model. Also, make sure the expressions of the intents match the incoming data. The expressions might need to be cleaned up when the incoming messages start being sanitized.

  See About intent training for advanced AI agents.

Improving deflection rate

Any conversation that does not end in escalation is considered deflected. You can review conversation logs and take steps to improve your deflection rate.

Read through conversation logs to see how well dialogues work, paying attention to:

• Do users break dialogues? They might break a dialogue by not using buttons or by asking where to find their order number mid-escalation flow. In that case, you might add intent listening or include free text to guide users through the flow.

• Are they not understanding the instructions? People generally don’t always read long messages. In that case, consider making the message shorter and easier to follow.
• Are they missing key information to help them through the flow? In that case, think about what a live agent would do and add as much of that information in the dialogue as possible.

Take the following actions to improve deflection:

• Adjust the default reply to help manage expectations and guide users through flows. See About system replies for advanced AI agents.
• Conduct content coverage analysis to identify new potential intents that can be automated. See Performing content coverage analysis.
• Use an API integration to automate more conversations. Identify new, suitable use cases for an API to increase automation and improve deflection.

After taking steps to improve your deflection rate, you might review resolutions states as another way to improve your AI agent performance. You can filter on the states escalated and not resolved in the conversation logs to identify troublesome conversations. Look for trends over time.

Using smarter dialogues to improve user experience

Use the following tools to build smarter dialogues and improve user experience:

• Backend integrations. Use backend integrations where possible to fetch data that the advanced AI agent can provide to users.
• Conditional blocks. Use conditional blocks to jump from one flow to the middle of another flow. This allows the AI agent to provide the correct answer if certain keywords are recognized, set the reply differently the second and third time, reduce repetitiveness, and provide less guidance before escalation if the user has been through the flow before.
• Confidence score. Use the native parameter confidence_score as a fall-back in replies where the AI agent might be less confident.

  For chat AI agents, if confidence_score is below 90%, the AI agent can confirm the customer intention another way. For example, “Gotcha. I just want to make sure I understood you correctly. You’ve lost your password and want to generate a new one. Is that correct?” For ticket AI agents, if the AI agent is less confident about a topic, you might exclude the reply and trigger just the actions instead.

• Escalation templates. It's a good idea to manage the escalation process in one centralized place, rather than in each specific flow, to streamline intent replies. Be sure to set operating hours to manage expectations and escalate appropriately.
• A/B testing. Use A/B testing to optimize dialogue flows with data-driven decisions. See Performing A/B testing for advanced AI agents.