Service Incident - June 2, 2022 - Zendesk ZBot - support.zendesk.com Support widget issues

Summary

Between May 23, 2022 from 01:23 UTC to June 03 21:15 UTC, Zendesk customers may have experienced intermittent issues when trying to reach the Zendesk Support team via the Zendesk ZBot Widget at support.zendesk.com. 

Timeline

17:41 UTC | 10:41 PT

We are investigating reports of intermittent issues reaching Zendesk Support using the Zendesk ZBot Widget at support.zendesk.com. We will provide additional updates as soon as we can.

17:56 UTC | 10:56 PT

We have confirmed an issue affecting a small percentage of conversations with the Zendesk ZBot Widget at support.zendesk.com causing issues reaching Zendesk support. Our team is investigating and we will share any additional information as soon as it's available.

18:21 UTC | 11:21 PT

While our team continues to investigate the issue affecting routing on the Zendesk ZBot Widget to contact Zendesk support, new requests will temporarily be redirected to a submission form rather than messaging conversations.

18:55 UTC | 11:55 PT

We are still investigating the cause of the issues affecting routing on the Zendesk Zbot Widget to contact Zendesk support, and new requests will continue to be directed to a submission form. We will provide additional updates when we have new information to share.

21:02 UTC | 14:02 PT

We are still seeing an issue that is presenting a technical error message in some conversations with the Zendesk ZBot Widget at support.zendesk.com that can cause intermittent issues reaching Zendesk support.

21:03 UTC | 14:03 PT

As an alternative option to reach out to our teams, you are able to now submit a web form ticket choosing the option "Submit a Request".

21:23 UTC | 14:23 PT

We are happy to report that we have resolved the issues affecting access to our Zendesk ZBot Widget for support (support.zendesk.com) and you can now message our support team again after refreshing your browser. Thank you for your patience during our investigation.

22:04 UTC | 15:04 PT

We are no longer seeing errors with the Zendesk ZBot Widget at support.zendesk.com and have now disabled the web form option.

POST-MORTEM

Root Cause Analysis

This incident was caused by insufficient network resource allocation resulting in network traffic reaching its limits, causing issues with access to the Zendesk Zbot Widget.

Resolution

To fix this issue, more resources were allocated and network capacity was increased.

Remediation Items

1. Create additional monitoring and alerting for the type of errors seen in this incident.
2. Further investigation into how limits were reached in this incident.
3. Investigate and backfill any missing user metadata missed during the incident.
4. Expand access to troubleshooting steps and tooling among engineering teams.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.