Understanding how Answer Bot works with your content

Return to top
Have more questions? Submit a request

22 Comments

  • Hannah Voice

    Hi Jeremy Korman

    Reading over this article, it sounds like Answer Bot is not designed to get more intelligent.

    "Although Answer Bot is powered by a machine learning model, this does not mean that Answer Bot is constantly learning. Answer Bot’s model does not incorporate feedback in real-time from end users or agents. Therefore, the feedback has no influence on which articles Answer Bot will recommend."

    "Answer Bot will consistently recommend the same articles regardless of any feedback from agents or end users."

    In that case, I am wondering how the Improve Answers feature comes into play. Can you explain more? If I'm understanding this article correctly, it seems like having agents mark articles as off-topic wouldn't be of any value?

    Many thanks,

    Hannah

     

    2
  • Jeremy Korman
    Zendesk team member

    Hi Hannah Voice, thanks for reaching out. Earlier this year when we upgraded the model to support additional languages and jargon, we also evaluated the “improve answers” feature. The analysis showed providing this feedback was eating up agent time, but wasn’t delivering a substantial impact to how well Answer Bot performs.  After much consideration, we decided to remove the “improve answers” button on July 31st 2020.  We expect that this will save you and your agents time, and allow you to focus on other best practices for maximizing Answer Bot’s performance. We’ve been reaching out to accounts that are using this feature to let folks know about the upcoming change, I’ll double-check that you’re on the distribution list

    0
  • Hannah Voice

    Thanks for clarifying Jeremy Korman. Could you confirm my understanding here that there are no circumstances under which Answer Bot will get more intelligent the more it's used? Does anything at all impact it (other than optimizing Help Center content)? Thanks.

    0
  • Jeremy Korman
    Zendesk team member

    Hey Hannah Voice, that's correct, Answer Bot won't automatically improve on its own. However, the more you use Answer Bot, the more data you'll have available in Explore and via Content Cues (if you're on Guide Enterprise).  The out of the box dashboard makes it super simple to discover opportunities to improve your help center and boost Answer Bot's performance.  Here's an overview of the dashboard

    0
  • Dawn Anderson

    Hi Team

    We did a search using a certain keyword and it's returning an article that has no mention of this word or label - do we know how the answerbot knows what to return?

    Thank

    0
  • Jeremy Korman
    Zendesk team member

    Hi Dawn Anderson, thanks for the comment.  There are 2 quick things I'd like to point out:

    1. If Answer Bot suggested a particular article then the algorithm detected that the inquiry and article were in the same neighborhood (if we're using the analogy from the article above).  Unfortunately, there's not an easy way to expose exactly what words or phrases made this happen for each inquiry.
    2. Answer Bot uses labels to create a "whitelist" of articles, but it doesn't use labels like keywords (check out the misconceptions above).  Here's another article with more info.
    0
  • Ibrahim

    Hi,

    Finally a bit of explanation on how Answer Bot works. Can I summarize this as follows:

    Answer Bots suggests (almost arbitrarily) 3 answers.

    It will not learn that certain answers are correct/incorrect.

    There is absolutely no way to teach Answer Bot.

    It is us, the agents, that have to learn how each article in the Help Center may have to be written to hopefully fit Answer Bot's (black box) Algorithm.

    If we don't adapt the articles to Answer Bot's algorithm then it may keep suggesting the incorrect articles..forever.

    0
  • Jeremy Korman
    Zendesk team member

    Hi Ibrahim, thanks for the comment.  Our data science team has done a lot of work to reduce the amount of energy agents and admin should need to spend managing Answer Bot, so I certainly hope the article recommendations don't appear arbitrary!  If you haven't done so already, I'd recommend you check out this article on Optimizing your Content for Answer Bot.

    Also, I'd add that Answer Bot won't always recommend 3 articles.  If Answer Bot only finds 1 relevant article, it'll only recommend the 1 article.  Similarly, if there are 0 relevant articles, there will be 0 suggestions.  So there could be 0, 1, 2, or 3 suggested articles.

    Lastly, you're correct that the best way to influence Answer Bot's recommendations is to adjust the article's title and the first 75 words.  However, there are other ways you can guide Answer Bot to search from a more relevant sub-set of articles by using Labels.  Here's a comprehensive list of additional resources that you might find interesting!

    0
  • Ibrahim

    Hi Jeremy,

    agent's time is not reduced by forcing them to figure how to adapt every article to all variations of customer queries which are processed by a black box.

    Adapting every article (which are also processed by the black box) to preprocessed queries (by a black box) is like shooting in the dark.

    1. Is there a way to aiding/leading agents to write articles which will map to respective query-'neighbourhood'?

    2. Is there any way at all of teaching Answer Bot where to search? 

     

    0
  • Arwen Griffioen

    Hello Ibrahim,

    I'm sorry to hear that you feel like Answer Bot is a black box. The machine learning team has worked extensively with Jeremy to write a clear and understandable article on how the model behind Answer Bot works. If there is a particular part of the article that isn't clear enough we'd be more than happy to try to expand it.

    Given that Answer Bot is only processing the first 75 words of the article it is important that each article covers only a single concept OR that the opening paragraph summarises all of the concepts that are covered. I see that your articles tend to cover multiple topics and that the first paragraph doesn't generally elaborate on the topic indicated by the title. This simple addition to your articles would boost Answer Bot performance and may help your users as well. 

    The Content Cues Product can help you discover areas where the content of your articles might not address your customer's questions. This will help to focus content management efforts and ensure that less time is spent making changes without impact.

    Do please submit a ticket if none of this information helps and we can look at your specific use cases, articles and issues.

    Arwen
    Machine Learning

     

    0
  • Ibrahim

    Thanks for the clarification.

     

    Content Cues is only available to Enterprise plan, right? If so, is there any other similar tool or approach?

    0
  • Jeremy Korman
    Zendesk team member

    Hi Ibrahim, yes, Content Cues is a Guide Enterprise feature.  If you're not on Guide Enterprise, you can build a report to see transcripts of what your customers are asking Answer Bot, check out this Explore recipe: https://support.zendesk.com/hc/en-us/articles/360048645274

    1
  • Frédéric

    sad news that the "improve answer bot" button has been removed without notice.

    It was helpful.

    1
  • Jeremy Korman
    Zendesk team member

    Hi Frédéric, you should've received a few emails and IPM's about this over the past several months.  I believe our first email went out in April, but I'll double-check that you were on the distribution list.

    -1
  • Frédéric

    Jeremy Korman Sorry if I missed that anyway it's missing now. I just had a ticket and answer bot was off with all 3 suggestions! Improving gave us at least a feeling of control on answer bot, now we all depend on the learning...

    0
  • Jeremy Korman
    Zendesk team member

    Frédéric it's a common misconception that the feedback provided by agents was having a direct and immediate impact on your performance, and this is part of the reason the feature was removed - we realized it was creating a false expectation.  Check out the explanation in the article above about how that feedback was primarily used by our data science team to tune the global model's threshold.  We've tried to address this in the misconceptions at the bottom of the article as well.

    -1
  • Pertti Kangas

    I see: we decided to remove the “improve answers” button on July 31st 2020. that's too bad; means that the whole idea of answer bot that can get better is not true. Shame!!! I always thought the whole idea was to use the bot to improve and offer better and more relevant answers to the users. Please fix the bot and bring it back!

    2
  • Angharad Whitburn

    There are a lot of faults and flaws with answer bot. In original articles we were told it looked at labels which is important for our business, after much testing and back and forth, it has been concluded it doesn't. 

     

    The 'train' feature, it's a shame it was removed as it was a good idea, but i'm guessing it did nothing back end. Also this should have been communicated a lot better for people using Answer Bot. To just remove a feature? Were a sample of active users asked before this was removed? It is a paid product after all! 

     

    What it does need is a way to inform that it has answered correctly. We rely on end users to do so, but it would be great if we had an option to let it know it's answered correctly. 

    Answer Bot has a long way to go to be a great product, at the minute it's just ok. 

    0
  • Fernando Duarte

    Based on the comments I see here, Zendesk did a really bad job on communicating that a feature we've been using was being deprecated.

    I was also not informed of this change, nor did any of the administrators in the system.

     

    We were using the the “improve answers” button as part of our onboarding of new staff, as they would read the suggested articles and rate them accordingly.  Removing this button hastily and without proper communication forces me to evaluate other solutions to replace the "AI" that cannot be trained 

    3
  • Jeremy Korman
    Zendesk team member

    Hi Angharad Whitburn - thanks for the comment.  Answer Bot can use labels as a way to create an allowed list of articles, but they cannot be used as a synonym for keywords.  This is a common misconception and this was part of the motivation for directly calling this out in the article above under the "misconceptions" section.

    Hi Fernando Duarte - I'm so sorry to hear our comms didn't make it to you.  I've just emailed you a copy of the email that was sent out about this change.

    -2
  • Angharad Whitburn

    It's not a 'misconception'. We asked our customer success representative about use of labels repeatedly when considering using Answer Bot. They advised it read the article text and labels. 

    This article also suggest labels, and not in the context of limiting articles. 

    https://support.zendesk.com/hc/en-us/articles/115011212107-Troubleshooting-Answer-Bot-isn-t-suggesting-articles?auth_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhY2NvdW50X2lkIjoxLCJ1c2VyX2lkIjo5NzE5MDQ4MjA3LCJ0aWNrZXRfaWQiOjUyNTE0ODMsImRlZmxlY3Rpb25faWQiOjM2MDM1ODk2MDYzNCwiYXJ0aWNsZXMiOlszNjAwMjI5MTIyMzMsMTE1MDExMjEyMTA3LDM2MDA0MTMyMzY5NF0sInRva2VuIjpudWxsLCJleHAiOjE1ODU0OTk3MjV9.8ZDEJIqjaR3bKLqWlkxR5rK-26Ky4prtdTtmfS3w9RM&solved=0

     

    It has now been clarified after much time spent going back and forth however not corresponding a change to the product is poor service from Zendesk. 

    2
  • Jeremy Korman
    Zendesk team member

    Hi Angharad Whitburn - sorry to hear about the confusion and the back and forth in order to get to the bottom this.  Thanks for pointing out that article, I've flagged this with our tech-docs team and we'll get it updated ASAP to ensure it's clear that labels do not work as synonyms to boost results.  Let us know if you have any other questions.

    0

Please sign in to leave a comment.

Powered by Zendesk