In Zendesk QA, whilespotlight insightsflag tickets,rating categoriesscore agents.

What's my plan?
Add-on Quality Assurance (QA) or Workforce Engagement Management (WEM)

Summary: ◀▼

Create AI prompt-based rating categories to score agent behavior in customer conversations using natural language criteria. Customize prompts or use templates to define scoring rules, test and validate prompts, and add categories to scorecards. This feature supports up to 10 active AI categories per account and requires enabling automatic and LLM-based scoring settings for accurate evaluations.

Location: Zendesk QA > Settings > AI

In Zendesk QA, while spotlight insights flag tickets, rating categories score agents.

In addition to the system categories, manual categories, and exact text-match categories, admins and account managers can create AI prompt-based rating categories.

All rating categories allow you to effectively assess customer conversations. However, by using AI prompt-based categories, you can ask targeted questions about conversations using natural language and automatically score them based on your criteria.

Zendesk QA prompt-based AI insights leverage the latest AI models, letting you create custom AI rating categories from scratch or choose from our ready-to-use AI prompt templates. The AI Prompt category criteria define the agent's behavior.

Tip: To ensure proper functionality, both the Automatic scoring with AutoQA and the LLM-based AutoQA account settings must be turned on.

Related articles

  • About AI insights in Zendesk QA
  • Best practices for creating AI insights prompts in Zendesk QA
  • Creating AI prompt-based spotlight insights

Admins and account managers can have up to 10 active AI prompt-based rating categories and AI prompt-based spotlight insights per account. When you reach this limit, you must either mark one as inactive or delete it before you can create or activate more.

To create a custom AI prompt-based rating category

  1. In Quality assurance, click your profile icon in the top-right corner.
  2. Click Settings ().
  3. In the sidebar under Account, click AI.
  4. Click Create AI insight.
  5. Select the Prompt-based AI insight type.
  6. Enter an AI insight name.
  7. (Optional) Click Prompt templates to select a template from our ready-to-use AI prompt templates, then click Use template.
  8. Customize any template to fit your specific needs or create a new prompt from scratch:
    1. In What do you want to detect?, clearly describe what this insight should identify and accomplish.

      Specify that the focus is on the behavior of agents and use objective, measurable language. You can enter up to 2,000 characters.

      Be sure to follow our best practices for writing prompts for AI insights to ensure your insights are accurate and actionable.

    2. Enter an additional scoring prompt, up to 128 characters, that describes what the agent must do to receive a thumbs-up () rating in a category. Be sure to follow our scoring prompt-based AI insights best practices to ensure accurate evaluations.
    3. Enter an additional scoring prompt, up to 128 characters, that describes things that should cause the agent to receive a thumbs-down () negative rating score for this category. For example, if the agent did not offer a demo, and the text does not include “let’s schedule a demo”, the agent receives a negative rating score for this category.
    Note: If AI can't find a clear answer based on the prompts, the category is automatically rated as N/A.
  9. (Optional) If the AI suggestions toggle is turned on in your account settings, click Enhance. The system analyzes your prompt to resolve ambiguity, vagueness, or poor formatting. It also refines your core intent by adding the following details:
    1. The main ask of your prompt
    2. What it is
    3. What it is not
    4. You prompt answer description
    5. An example of a negative prompt score
    6. An example of a positive prompt score
    7. What needs clarifying
    8. Your prompt answer options
  10. (Optional) Click Review the enhanced prompt to see what changed from your original prompt and access detailed suggestions.
  11. (Optional) Click Use original prompt to revert to your original prompt.
  12. (Optional) If the AI suggestions toggle is turned off in your account settings, click Validate to receive feedback and tips on how to improve your prompt without changing it.
  13. Under Suggested usage, select Use as a category on a scorecard.
  14. (Optional) Click Test prompt to confirm that your prompt works correctly and meets the specified criteria.

    A conversation filter view shows the total number of conversations found based on the prompted insights.

    To keep the pool of conversations for testing specific, the filter displays a list of closed conversations created in the last seven days that have more than four replies, by default, with a limit of 100 conversations.

  15. (Optional) Click Select for testing. The conversations found using this prompt are listed in the right-side panel, displayed 10 at a time.

  16. (Optional) If you’ve validated and tested this prompt, then edited it and are testing it again, a prompt history is displayed. This lets you view earlier versions of the prompt and the match rate, which indicates the results evaluated based on that version. Check how often auto review and manual review scores align to get a clearer overview of your prompt performance. You can use this history to select the most suitable prompt version before moving on to its final creation.

  17. (Optional) Test your results:
    1. (Optional) Click Review next to each conversation and manually score them based on the prompt you are creating so you can compare the results and see how often auto review and manual review scores align.
    2. Click Run auto review to see the results of your prompt conditions automatically applied to all listed conversations under Auto.

      If you see many conversations highlighted in yellow, this likely indicates a misalignment between how you want to score conversations and how AI is scoring them. You should consider rephrasing your prompt.

      When rephrasing what you want to detect with your prompt, if your changes are minor, you can click Run auto review again. However, if your changes are more significant, such as modifying it to evaluate the customer instead of the agent, the prompt type might change from category to spotlight. In this case, you'll need to return to step 14 and click Test prompt again.
  18. Click Create category.
  19. After creating it, you must add the category to your scorecards to start using it.
Powered by Zendesk