On February 16, 2023 from 1:48 to 3:28 UTC, customers across all pods may have experienced delays in Chat and Social Messaging, in some cases the inability to chat with their end users at all. Additionally, Answer Bot experienced some impact: in some cases, the Answer Bot greeting and subsequent responses may have failed to load in the web widget.
03:58 UTC | 19:58 PT (Feb 15)
Since 01:48 UTC, some customers across all Pods may have experienced delayed Chat messages and in some cases being unable to chat with end users. We are in recovery now and agents should be able to chat with end users as normal. A final update will be posted soon.
04:33 UTC | 20:33 PT (Feb 15)
We are happy to report that the issue with delayed chat messages is now resolved. We apologize for the inconvenience and thank you for your patience while we worked through this today.
Root Cause Analysis
This incident was caused by an unexpected error wherein a backend Chat and Messaging service did not reconnect following maintenance on some cloud provider servers.
To fix this issue, we manually reconnected and restarted the backend service.
- Improve monitoring and alerting to catch similar issues much sooner.
- Work with our cloud provider to ensure a full understanding of the root cause and faster solutions to address similar issues.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
Postmortem published February 28, 2023.
Article is closed for comments.