Service Incident - February 16, 2023 - Support & Sunshine Conversations | All Pods - Chat & Messaging in Agent Workspace Delays

SUMMARY

On February 16, 2023 from 1:48 to 3:28 UTC, customers across all pods may have experienced delays in Chat and Social Messaging, in some cases the inability to chat with their end users at all. Additionally, Answer Bot experienced some impact: in some cases, the Answer Bot greeting and subsequent responses may have failed to load in the web widget. 

Timeline

03:58 UTC | 19:58 PT (Feb 15)

Since 01:48 UTC, some customers across all Pods may have experienced delayed Chat messages and in some cases being unable to chat with end users. We are in recovery now and agents should be able to chat with end users as normal. A final update will be posted soon.

04:33 UTC | 20:33 PT (Feb 15)

We are happy to report that the issue with delayed chat messages is now resolved. We apologize for the inconvenience and thank you for your patience while we worked through this today.

POST-MORTEM

Root Cause Analysis

This incident was caused by an unexpected error wherein a backend Chat and Messaging service did not reconnect following maintenance on some cloud provider servers.

Resolution

To fix this issue, we manually reconnected and restarted the backend service.

Remediation Items

1. Improve monitoring and alerting to catch similar issues much sooner.
2. Work with our cloud provider to ensure a full understanding of the root cause and faster solutions to address similar issues.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.