Autoscoring automatically evaluates customer interactions based on predefined categories, ensuring consistent quality assessments. It reduces subjectivity and saves reviewers time so that they can focus on categories that require more attention. Agents must be added to Zendesk QA for autoscoring to work. AutoQA is available on the AI suite and Enterprise subscription plan.
You can learn more in this article or by watching this overview video.
This article contains the following sections:
- Understanding autoscoring categories
- Setting up autoscoring categories
- Creating custom AutoQA categories
- Using autoscoring in the Conversation view
- Viewing the AutoQA Dashboard
- Disclaimer
Understanding autoscoring categories
There are eight categories that receive automatic scoring in Zendesk QA:
- Greeting - Answers the question “Did the agent greet the customer?” (binary scale 👍 👎)
- Empathy - Assesses whether the agent was empathetic towards the customer and their problems (binary scale 👍 👎)
- Spelling and grammar - Detects grammar mistakes, misspellings, and style errors (customizable scale)
- Closing - Evaluates if the agent closed the conversation properly, including offering further help and thanking the customer (binary scale 👍 👎)
- Solution offered - Identifies whether a solution was proposed in the conversation (binary scale 👍 👎, or "N/A" if not sure)
- Tone - Recognizes 27 tones and calculates a score based on their positive or negative weights (customizable scale)
- Readability - Analyzes how easily a text can be understood, considering word complexity and sentence length (fixed 3-point scale)
- Comprehension - Checks if the agent understood the customer's issue, possibly requiring clarifying questions or summarizing the problem (binary scale 👍 👎)
Setting up autoscoring categories
To set up autoscoring
- In Zendesk QA, click your profile icon in the bottom-left corner.
- Select Users, bots, and workspaces.
- Choose the relevant workspace then click Scorecard.
-
Create category or click an existing one to edit it.
See Setting up a scorecard for more information on creating new categories. - Toggle on Use AutoQA for the category to start using automatic scores.
- Choose the AutoQA category from the pre-populated list.
This will populate a default name and description for the category, which you can tailor to suit your needs.
- Click Create category.
Creating custom AutoQA categories
You can also create your own AutoQA categories.
Create a custom AutoQA category for finding specific keywords or phrases in conversations and automatically rate the agent. For example, the agent did not offer a demo so the text did not contain “let’s schedule a demo”. The agent gets 👎 for this category.
Custom AutoQA categories are created at the account level and applied to the scorecard on the workspace level.
You can create up to 10 custom categories per account.
When a new category is added, only new incoming closed conversations are automatically analyzed and rated based on the category conditions (and not existing ones). Historical data on dashboards and conversations remain intact.
Similarly to existing AutoQA categories, if no matches are found then N/A is assigned.
Using autoscoring in the Conversation view
In the Conversation view (accessed via the icon in the sidebar), conversations identified as having autoscoring categories present will be marked with a hologram icon, similar to the Auto Quality Score calculation (AQS). AQS is shown alongside your internal quality score in the Reviews tab for easy comparisons. The automated quality score considers all the AutoQA categories activated in your scorecard settings, even if some categories are not added to your manual QA scorecards.
Autoscores can vary for different agents. If a reviewer disagrees with the autoscore, they can click on a different rating and Submit the review.
Reviewers can Highlight to check which part of the conversation was graded. The autoscore will be calculated once the reviewer submits the review, and the result will be visible only on the AutoQA Dashboard.
If reviewers need to find conversations where Zendesk QA automatically found errors (e.g. in grammar), they can use the Conversation view filter – see Filtering conversations for details.
Viewing the AutoQA Dashboard
The Auto QA Dashboard provides information about auto-reviewed conversations.
Disclaimer
The AutoQA scoring will show results for any new (eligible for scoring) closed conversations that have been imported after the feature has been enabled. It might take some time for the scores to load due to necessary calculations.
- AutoQA uses Microsoft Azure OpenAI Service for automatically scoring some categories. Your data is processed under a strict Data Processing Agreement (DPA) only to provide AutoQA to you and is never used to train any OpenAI models.
- Microsoft Azure OpenAI Service enables a wider range of AutoQA categories, out-of-the-box multilingual support, fast-paced addition of new categories, and improved evaluation accuracy. GPT-3.5 shows human-level contextual awareness, benefiting from the latest advancements in AI technology.
To opt out of MS Azure OpenAI processing in Zendesk QA click your profile icon in the bottom-left corner, then select Settings > Account > General.
See AutoQA in Zendesk QA FAQ for common questions about AutoQA autoscoring.