This article offers a high-level overview of how Answer Bot works. It includes the following topics:
How does Answer Bot process natural language?
Answer Bot is powered by Artificial Intelligence which means that it is able to mimic human behavior. Answer Bot uses natural language processing (NLP) to read every article in your help center and to understand the main concept behind each article. Answer Bot then takes all the concepts from all the articles and places them onto a map. Each concept gets its very own “address” on the map so that it lives near other, similar concepts. However, instead of just city, street, and zip code, this address has 500 parts. Whenever a new question comes in, Answer Bot does its best to understand the concept that the question is asking about and use the map to determine the closest existing article.
For example, here are some concepts that Answer Bot might extract from a few questions:
Question | Possible concept |
---|---|
How do I dump my tickets to a file? | Exporting Data |
I’m locked out of my account | Account Access / Password Reset |
How do I create a crane? | Folding Origami Birds |
How does Answer Bot decide which article(s) to recommend?
When an incoming question closely matches with an existing article, they become “neighbors” on the map (as described above) and it’s clear that Answer Bot should recommend the article. However, when the closest match is a few streets over, or in a nearby neighborhood, it becomes less certain that the concepts are related.
The data science team at Zendesk carefully monitors Answer Bot’s performance and has finely tuned this over time by adjusting a “threshold knob”. This threshold is not adjustable by admin or agents, it’s only accessible to the Zendesk development teams. The threshold knob is a global control, meaning it affects all Answer Bot accounts, and is used to determine how closely two concepts must be on the concept map to be considered similar concepts. If the threshold knob is turned up, Answer Bot becomes more conservative and will recommend fewer articles that are more likely to be relevant to the question. However, this means there will also be more questions where Answer Bot does not make any recommendations at all. If the threshold knob is turned down, Answer Bot will recommend more articles, but there’s a higher chance that some of the articles will appear irrelevant to the end user.
Common misconceptions: What Answer Bot doesn’t do
There are some common misconceptions about Answer Bot, and machine learning in general, that can lead to confusion over how they work. In this section, we’ll address these misconceptions and hopefully give you a clearer understanding about what Answer Bot does -- and doesn’t do -- with your data.
It includes the following questions:
- Does Answer Bot learn based on end user feedback? Isn’t that where the machine learning comes in?
- Is Answer Bot’s AI-powered search always better than a keyword search?
- Can I “train” Answer Bot by asking the same question and answer over and over again, and responding with “Yes” or “No” to mark an article as relevant or irrelevant?
- If I add labels to my articles, is that like adding a keyword to the article? Can this be done to boost how often an article is suggested?
- If I can’t improve Answer Bot with the “improve answers” button, how can I improve Answer Bot’s performance?
Does Answer Bot learn based on end user feedback? Isn’t that where the machine learning comes in?
Although Answer Bot is powered by a machine learning model, this does not mean that Answer Bot is constantly learning. Answer Bot’s model does not incorporate feedback in real-time from end users or agents. Therefore, the feedback has no influence on which articles Answer Bot will recommend.
The end user feedback is captured and used in a number of ways:
- It is displayed to agents to provide additional context on what articles were viewed, marked as “not helpful,” or used to resolve a case
- It is exposed in reporting for admin to track Answer Bot’s performance
- It is evaluated by the data science team at Zendesk
If you see that Answer Bot is repeatedly recommending incorrect articles, the best thing to do is modify the title and the first 75 words of the articles to make the main concept more clear. You can also create a list of articles for Answer Bot to draw from by using labels so that Answer Bot’s suggestions will only draw from a sub-set of articles.
Is Answer Bot’s AI-powered search always better than a keyword search?
Overall, we’ve found that Answer Bot’s AI-powered recommendations are more accurate and relevant than a keyword search, especially when the question is asked as a full sentence (instead of one to three words).
However, there are times when a keyword search may work better. For example, when a user asks a single-word question via Web Widget, Answer Bot defaults to using a keyword search, as this is generally more accurate for single-word queries. The exception to this is languages, like Chinese, that do not have explicit word boundaries like spaces.
Can I “train” Answer Bot by asking the same question and answer over and over again, and responding with “Yes” or “No” to mark an article as relevant or irrelevant?
No. Answer Bot will consistently recommend the same articles regardless of any feedback from agents or end users. Answer Bot is specifically built so it doesn’t require any training to get started. It’s already pre-trained to understand natural language. If you test out a phrase/question and Answer Bot is making incorrect recommendations, the best thing to do is modify the title and the first 75 words of the articles to make the main concept more clear.
If I add labels to my articles, is that like adding a keyword to the article? Can this be done to boost how often an article is suggested?
Labels are a great way to create a list of approved articles that Answer Bot can pull from. However, labels do not have an influence on the weighting that Answer Bot gives to each article. For more info on labels, check out this article.
If I can’t improve Answer Bot with the “improve answers” button, how can I improve Answer Bot’s performance?
The best way to improve Answer Bot’s performance is to consider the following:
- Analyze your Answer Bot Activity - Use Explore to see which articles are you best and worst-performing.
- The Structure of Existing Articles - Look at your help center articles and make sure that the content is concise and well organized. Each title should be phrased as a short sentence or a question.
- Content Cues - Use machine learning technology and Guide article usage data to help you discover opportunities and tasks that will improve the health of your knowledge base.
22 Comments
Hi Jeremy Korman
Reading over this article, it sounds like Answer Bot is not designed to get more intelligent.
"Although Answer Bot is powered by a machine learning model, this does not mean that Answer Bot is constantly learning. Answer Bot’s model does not incorporate feedback in real-time from end users or agents. Therefore, the feedback has no influence on which articles Answer Bot will recommend."
"Answer Bot will consistently recommend the same articles regardless of any feedback from agents or end users."
In that case, I am wondering how the Improve Answers feature comes into play. Can you explain more? If I'm understanding this article correctly, it seems like having agents mark articles as off-topic wouldn't be of any value?
Many thanks,
Hannah
Hi @..., thanks for reaching out. Earlier this year when we upgraded the model to support additional languages and jargon, we also evaluated the “improve answers” feature. The analysis showed providing this feedback was eating up agent time, but wasn’t delivering a substantial impact to how well Answer Bot performs. After much consideration, we decided to remove the “improve answers” button on July 31st 2020. We expect that this will save you and your agents time, and allow you to focus on other best practices for maximizing Answer Bot’s performance. We’ve been reaching out to accounts that are using this feature to let folks know about the upcoming change, I’ll double-check that you’re on the distribution list
Thanks for clarifying Jeremy Korman. Could you confirm my understanding here that there are no circumstances under which Answer Bot will get more intelligent the more it's used? Does anything at all impact it (other than optimizing Help Center content)? Thanks.
Hey @..., that's correct, Answer Bot won't automatically improve on its own. However, the more you use Answer Bot, the more data you'll have available in Explore and via Content Cues (if you're on Guide Enterprise). The out of the box dashboard makes it super simple to discover opportunities to improve your help center and boost Answer Bot's performance. Here's an overview of the dashboard.
Hi Team
We did a search using a certain keyword and it's returning an article that has no mention of this word or label - do we know how the answerbot knows what to return?
Thank
Hi Dawn Anderson, thanks for the comment. There are 2 quick things I'd like to point out:
Hi,
Finally a bit of explanation on how Answer Bot works. Can I summarize this as follows:
Answer Bots suggests (almost arbitrarily) 3 answers.
It will not learn that certain answers are correct/incorrect.
There is absolutely no way to teach Answer Bot.
It is us, the agents, that have to learn how each article in the Help Center may have to be written to hopefully fit Answer Bot's (black box) Algorithm.
If we don't adapt the articles to Answer Bot's algorithm then it may keep suggesting the incorrect articles..forever.
Hi Ibrahim, thanks for the comment. Our data science team has done a lot of work to reduce the amount of energy agents and admin should need to spend managing Answer Bot, so I certainly hope the article recommendations don't appear arbitrary! If you haven't done so already, I'd recommend you check out this article on Optimizing your Content for Answer Bot.
Also, I'd add that Answer Bot won't always recommend 3 articles. If Answer Bot only finds 1 relevant article, it'll only recommend the 1 article. Similarly, if there are 0 relevant articles, there will be 0 suggestions. So there could be 0, 1, 2, or 3 suggested articles.
Lastly, you're correct that the best way to influence Answer Bot's recommendations is to adjust the article's title and the first 75 words. However, there are other ways you can guide Answer Bot to search from a more relevant sub-set of articles by using Labels. Here's a comprehensive list of additional resources that you might find interesting!
Hi Jeremy,
agent's time is not reduced by forcing them to figure how to adapt every article to all variations of customer queries which are processed by a black box.
Adapting every article (which are also processed by the black box) to preprocessed queries (by a black box) is like shooting in the dark.
1. Is there a way to aiding/leading agents to write articles which will map to respective query-'neighbourhood'?
2. Is there any way at all of teaching Answer Bot where to search?
Hello Ibrahim,
I'm sorry to hear that you feel like Answer Bot is a black box. The machine learning team has worked extensively with Jeremy to write a clear and understandable article on how the model behind Answer Bot works. If there is a particular part of the article that isn't clear enough we'd be more than happy to try to expand it.
Given that Answer Bot is only processing the first 75 words of the article it is important that each article covers only a single concept OR that the opening paragraph summarises all of the concepts that are covered. I see that your articles tend to cover multiple topics and that the first paragraph doesn't generally elaborate on the topic indicated by the title. This simple addition to your articles would boost Answer Bot performance and may help your users as well.
The Content Cues Product can help you discover areas where the content of your articles might not address your customer's questions. This will help to focus content management efforts and ensure that less time is spent making changes without impact.
Do please submit a ticket if none of this information helps and we can look at your specific use cases, articles and issues.
Arwen
Machine Learning
Thanks for the clarification.
Content Cues is only available to Enterprise plan, right? If so, is there any other similar tool or approach?
Hi Ibrahim, yes, Content Cues is a Guide Enterprise feature. If you're not on Guide Enterprise, you can build a report to see transcripts of what your customers are asking Answer Bot, check out this Explore recipe: https://support.zendesk.com/hc/en-us/articles/360048645274
sad news that the "improve answer bot" button has been removed without notice.
It was helpful.
Hi Frédéric, you should've received a few emails and IPM's about this over the past several months. I believe our first email went out in April, but I'll double-check that you were on the distribution list.
Jeremy Korman Sorry if I missed that anyway it's missing now. I just had a ticket and answer bot was off with all 3 suggestions! Improving gave us at least a feeling of control on answer bot, now we all depend on the learning...
Frédéric it's a common misconception that the feedback provided by agents was having a direct and immediate impact on your performance, and this is part of the reason the feature was removed - we realized it was creating a false expectation. Check out the explanation in the article above about how that feedback was primarily used by our data science team to tune the global model's threshold. We've tried to address this in the misconceptions at the bottom of the article as well.
I see: we decided to remove the “improve answers” button on July 31st 2020. that's too bad; means that the whole idea of answer bot that can get better is not true. Shame!!! I always thought the whole idea was to use the bot to improve and offer better and more relevant answers to the users. Please fix the bot and bring it back!
There are a lot of faults and flaws with answer bot. In original articles we were told it looked at labels which is important for our business, after much testing and back and forth, it has been concluded it doesn't.
The 'train' feature, it's a shame it was removed as it was a good idea, but i'm guessing it did nothing back end. Also this should have been communicated a lot better for people using Answer Bot. To just remove a feature? Were a sample of active users asked before this was removed? It is a paid product after all!
What it does need is a way to inform that it has answered correctly. We rely on end users to do so, but it would be great if we had an option to let it know it's answered correctly.
Answer Bot has a long way to go to be a great product, at the minute it's just ok.
Based on the comments I see here, Zendesk did a really bad job on communicating that a feature we've been using was being deprecated.
I was also not informed of this change, nor did any of the administrators in the system.
We were using the the “improve answers” button as part of our onboarding of new staff, as they would read the suggested articles and rate them accordingly. Removing this button hastily and without proper communication forces me to evaluate other solutions to replace the "AI" that cannot be trained
Hi Angharad Whitburn - thanks for the comment. Answer Bot can use labels as a way to create an allowed list of articles, but they cannot be used as a synonym for keywords. This is a common misconception and this was part of the motivation for directly calling this out in the article above under the "misconceptions" section.
Hi Fernando Duarte - I'm so sorry to hear our comms didn't make it to you. I've just emailed you a copy of the email that was sent out about this change.
It's not a 'misconception'. We asked our customer success representative about use of labels repeatedly when considering using Answer Bot. They advised it read the article text and labels.
This article also suggest labels, and not in the context of limiting articles.
https://support.zendesk.com/hc/en-us/articles/115011212107-Troubleshooting-Answer-Bot-isn-t-suggesting-articles?auth_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhY2NvdW50X2lkIjoxLCJ1c2VyX2lkIjo5NzE5MDQ4MjA3LCJ0aWNrZXRfaWQiOjUyNTE0ODMsImRlZmxlY3Rpb25faWQiOjM2MDM1ODk2MDYzNCwiYXJ0aWNsZXMiOlszNjAwMjI5MTIyMzMsMTE1MDExMjEyMTA3LDM2MDA0MTMyMzY5NF0sInRva2VuIjpudWxsLCJleHAiOjE1ODU0OTk3MjV9.8ZDEJIqjaR3bKLqWlkxR5rK-26Ky4prtdTtmfS3w9RM&solved=0
It has now been clarified after much time spent going back and forth however not corresponding a change to the product is poor service from Zendesk.
Hi Angharad Whitburn - sorry to hear about the confusion and the back and forth in order to get to the bottom this. Thanks for pointing out that article, I've flagged this with our tech-docs team and we'll get it updated ASAP to ensure it's clear that labels do not work as synonyms to boost results. Let us know if you have any other questions.
Please sign in to leave a comment.