SUMMARY
Between 19:09 UTC on March 28, 2022 and 19:12 UTC on March 31, 2022, Zendesk customers were unable to contact the Zendesk Support team through the support widget at support.zendesk.com.
Timeline
21:35 UTC | 14:35 PT
We are investigating reports of being unable to reach Zendesk support (support.zendesk.com) through the support widget. We will provide an update as soon as we have more information.
21:42 UTC | 14:42 PT
As we continue to investigate the support.zendesk.com support widget issues, you can reach us via email at serviceincident@zendesk.com.
22:46 UTC | 15:46 PT
We are continuing to investigate the support.zendesk.com support widget issue. You can still email us at serviceincident@zendesk.com. Next update in 60 minutes or when we have more information.
23:45 UTC | 16:45 PT
Our team continues to work towards mitigating the http://support.zendesk.com support widget issue. You can still email us at serviceincident@zendesk.com. Next update in 60 minutes or when we have more information.
00:25 UTC | 17:25 PT
The issue with the support widget on support.zendesk.com has been resolved. Our customers can once again seek help through our chatbot (Z Bot) from within your products. Thanks for your patience while we worked through today's issue.
16:09 UTC | 9:09 PT
We are now seeing a small percentage of requests to the support.zendesk.com support widget being dropped. We will provide an update when we have more information to share.
19:30 UTC | 12:30 PT
We have rolled back a change to the support.zendesk.com support widget that resolves the issue. Please hard refresh your browser for the latest version of the widget.
POST-MORTEM
Root Cause Analysis
This incident was caused by an update which created a race condition when setting conversation metadata in new messaging conversations with the Zendesk Support widget at support.zendesk.com.
Resolution
To fix this issue, we rolled back the update.
Remediation Items
- Introduce additional layers of testing and improve existing widget testing scenarios.
- Revisit and revise response runbook for similar scenarios.
- Create additional alerts and improve existing monitoring.
- Refine on-call and escalation policies to improve response times.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.