Setting up autoscoring in Zendesk QA using AutoQA

What's my plan?

AutoQA supports your manual review efforts by automatically evaluating customer interactions for 100% of your ticket volume based on predefined categories. This ensures consistent quality assessments, reduces subjectivity, and saves reviewers time, allowing them to focus on categories that need more attention.

You can learn more in this article or by watching this overview video.

This article contains the following topics:

• Understanding autoscoring categories
• Setting up autoscoring categories
• Customizing AutoQA categories
• Creating new AutoQA categories
• Using autoscoring in the Conversation view
• Using autoscoring to filter conversations

Related articles

• Using Spotlight to filter conversations

Understanding autoscoring categories

There are eight categories that receive automatic scoring in Zendesk QA.

The languages supported for each category vary depending on whether OpenAI is enabled or disabled in the account settings. See Viewing and updating Zendesk QA account settings.

Category	Description	Score	Languages/OpenAI-enabled	Languages/OpenAI-disabled
Greeting	Answer the question “Did the agent greet the customer?”.	Binary: 👍, 👎	All	English, German, Spanish, French, Italian, Dutch, Polish, Portuguese
Empathy	Assesses whether the agent was empathetic towards the customer and their problems.	Binary: 👍, 👎	All	English
Spelling and grammar	Detects grammar mistakes, misspellings, and style errors.	Customizable	English (US), German, French, Polish	English (US), German, French, Polish
Closing	Evaluates if the agent closed the conversation properly, including offering further help and thanking the customer.	Binary: 👍, 👎	All	English, German, Spanish, French, Italian, Dutch, Polish, Portuguese
Solution offered	Identifies whether a solution was proposed in the conversation.	Binary: 👍, 👎 Or "N/A" if not sure	All	Not available*
Tone	Recognizes 27 tones and calculates a score based on their positive or negative weights.	Customizable	All	Not available*
Readability	Analyzes how easily a text can be understood, considering word complexity and sentence length.	Fixed 3-point scale	English	English
Comprehension	Checks if the agent understood the customer's issue, possibly requiring clarifying questions or summarizing the problem.	Binary: 👍, 👎	All	Not available*

*Solution, Tone and Comprehension are only available if you opt in to OpenAI processing.

Setting up autoscoring categories

You can enable AutoQA to automatically evaluate customer interactions based on your selected rating categories. Agents participating in these conversations must be users in Zendesk QA and members of the workspaces where the AutoQA categories are configured.

To set up autoscoring

1. In Zendesk QA, click your profile icon in the bottom-left corner.
2. Select Users, bots, and workspaces.
3. Choose the relevant workspace then click Scorecard.
4. Click the Pencil icon () next to a category to edit it, or click the Create category button to create a new one. See Setting up a scorecard for more information on creating new categories.
5. Turn the Use AutoQA toggle on for the category to start using automatic scores.
6. Select the AutoQA category from the dropdown list.

   This will populate a default name and description for the category, which you can customize to meet your needs.

   

7. Click Update category or Create category.

When a new category is added, only new incoming conversations with a Closed or Solved status are automatically analyzed and rated based on the category conditions. Conversations must contain at least one message from both the customer and the agent, with a minimum word count of 10.

Existing conversations, spam conversations, and historical data on dashboards are not evaluated by AutoQA.

It may take some time for the scores to load due to necessary calculations.

If no matches are found, "N/A" is assigned. AutoQA may also return an "N/A" evaluation for certain categories when there is insufficient information to provide an accurate rating.

Customizing AutoQA categories

You can customize the following AutoQA categories to identify specific keywords or phrases in conversations and automatically rate the agent: Spelling and grammar, Greeting, and Closing.

To customize an existing category

1. In Zendesk QA, click your profile icon in the bottom-left corner.
2. Click the Settings () icon.
3. Under Account in the sidebar, click AutoQA.

   If necessary, click the Sidebar toggle icon () to display the side menu.

4. Click Categories and select one of the following categories to open and edit:
   • Spelling and grammar
   • Greeting
   • Closing

Spelling and grammar

In this category, AutoQA analyzes grammar, spelling, and style mistakes in both agent and bot interactions.

To edit the Spelling and grammar category

1. Click the Add exemptions button.
2. Select the applicable languages from the dropdown menu.
3. Type or paste the word and phrases you’d like AutoQA to allow or ignore as spelling or grammar mistakes, separated by a line break.
4. Click Add exemptions.

Greeting

In this category, you can define approved greetings that align with your brand and service standards. Once added, AutoQA will only detect these approved greetings.

To edit the Greeting category

1. Click the Add greeting button.
2. Select the applicable languages from the dropdown menu.
3. Type or paste each greeting, with a maximum of 300 characters. You can also use the {··} icon to exclude names, company details, or any other sensitive information during AutoQA's closing checks. For example, instead of "Take care, John" you can use "Take care, {··}" to represent any name.
4. Click Add greeting.

Closing

In this category, AutoQA analyzes if the agent or bot wrapped up the conversation.

To edit the Closing category

1. Click the Add closing button
2. Select the applicable languages from the dropdown menu.
3. Type or paste each approved closing, with a maximum of 300 characters. Click the {··} icon to exclude names, company details, or any other sensitive information during AutoQA's closing checks. For example, instead of "Take care, John" you can use "Take care, {··}" to represent any name.
4. Click Add closing.

Creating new AutoQA categories

You can also create custom AutoQA categories using text matching to combine specific phrases and conditions. This allows you to identify agent responses that meet and do not meet your quality standards by looking for specific words or phrases in support interactions and setting up automated scoring based on that.

You can create up to 20 custom AutoQA categories per account.

Custom AutoQA categories are created at the account level and applied to the scorecard at the workspace level. See Setting up a scorecard.

To create a new AutoQA category

1. In Zendesk QA, click your profile icon in the bottom-left corner.
2. Click the Settings () icon.
3. Under Account in the sidebar, click AutoQA.

   If necessary, click the Sidebar toggle icon () to display the side menu.

4. Select Categories and click Create scorecard category.
5. Enter a name for the new category and select if it applies to All conversations or Specific conversations only.

   Selecting Specific conversations allows you to narrow down where your rule applies by source type, conversation channel, and help desk tag.

6. Type or paste your text content, with one word or sentence per row.

   

   You can add multiple rows and use different conditions for complex findings: and, or, is/not, contains/not any of, is/not any of, and variables. To use dynamic content type “/”. You can then use {··} to exclude names, company details, or other sensitive information during AutoQA's checks. For example, instead of "Take care, John" you can use "Take care, {··}" to represent any name.

7. Select how you want to automatically rate the agent. For example, if the agent did not offer a demo, and the text does not include “let’s schedule a demo,” the agent receives a for this category.
8. Click Create scorecard category.
9. To use this rating category, go to your workspace scorecard settings and add it onto a scorecard. See Setting up autoscoring categories.

Using autoscoring in the Conversations view

Conversations with autoscoring categories are marked with a hologram ( ) icon in the Conversations view.

To access AutoQA in Conversations

1. In Zendesk QA, click Conversations in the sidebar.
2. Select the conversation you want to review.
3. AutoQA category autoscores are provided as helpful information under Reviews.

AutoQA category autoscores only count towards the Internal Quality Score (IQS) of a conversation after a reviewer submits the review. See Using the Conversations view.

The Auto Quality Score (AQS) takes into account all AutoQA categories activated in your scorecard settings, even if some categories are not included in your manual QA scorecards.

Autoscores can vary for different agents. As a reviewer, if you disagree with the autoscore, you can select and submit a different rating.

The AutoQA dashboard tracks your teams’ performance across all AutoQA categories. It’s the best place to track trends over time and identify the training needs of your teams and individual agents. See Auto QA Dashboard.

AutoQA evaluates Zendesk and Ultimate bots but does not offer a combined AQS for bots in the Dashboard. This is to avoid penalizing bot builders in potential mismatches for human categories. The combined AQS is only available for agents.

Using autoscoring to filter conversations

AutoQA categories can also be used for filtering conversations. See Filtering conversations.

All AutoQA and custom AutoQA categories are displayed as separate items.

You can use the Conversation view filter to find conversations to review where specific AutoQA categories were applied. For example, you can create a filter to locate conversations where the AutoQA category solution is negatively rated by AutoQA, or where the solution category is rated positively but the sentiment is negative.