Service Incident - March 22, 2023 - ZBot Widget - Issues contacting Zendesk Advocacy Customer Support

SUMMARY

On March 22, 2023 from 07:59 UTC to 11:12 UTC, English speaking customers experienced issues contacting Zendesk Advocacy Customer Support via the ZBot Widget, and had to use the web form as a workaround.

Timeline

12:00 UTC | 05:00 PT
We are aware of some customers experiencing issues contacting Zendesk Advocacy Customer Support using our ZBot Widget. Our team is working to resolve the issue as soon as possible. We apologize for the inconvenience caused.

12:18 UTC | 05:18 PT
We appreciate your patience while we investigate and resolve the issue with the ZBot. In the meantime we have enabled the Web Form in our widget so customers are able to contact us. More information in 1h or when we have more details.

13:15 UTC | 06:15 PT
Troubleshooting continues for the issue where some customers got the error “It looks like you don’t have permission to chat with an agent.” while interacting with our Zendesk Customer Support ZBot Widget, even though they were correctly logged in. [1/2]

13:16 UTC | 06:16 PT
Please continue using the Web Form provided in the Zbot now to contact us if you need help. Next update in 2 hours or when we have more details. [2/2]

15:24 UTC | 08:24 PT
We continue our investigation and working toward a full resolution for the issue affecting some customers’ ability to reach out to Zendesk Customer Support via ZBot Widget. Web Form continues available for customers to contact our teams. Next update in 1hr.

16:17 UTC | 09:17 PT
We have resolved the issue preventing interaction with the ZBot Web Widget and you should now be able to interact with the ZBot widget again. We will update you if there are any other changes. Thank you for your patience.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by an unexpected change from a partner service provider responsible for some of the backend ZBot functionality, resulting in conversation failures for English speaking customers.

 

Resolution

To fix this issue, our partner service provider rolled back their change until they could address the issues it introduced.

 

Remediation Items

1. Improve existing playbooks and processes to ensure failover is enabled promptly and proper teams are involved in the investigation.
2. Request more comprehensive change notices for updates from partner service providers.
3. Revisit existing logging and alerting to determine if any improvements can be made.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.