07:34 UTC | 23:34 PT
We're happy to report that all Chat issues have been resolved and we've completed the recovery of Chat transcripts. If you see any discrepancies please reach out to us.
07:00 UTC | 23:00 PT
The issues impacting Chat Real-Time Monitor have been resolved. We continue working to recover the missing transcripts. We will share more updates when available.
05:22 UTC | 21:22 PT
We continue to work on the Chat Monitor issue and recovery of chat transcripts. More updates when they become available.
03:41 UTC | 19:41 PT
We continue to work on the Chat Monitor issues. Other Chat functionalities should be working as normal. We will share more updates when they become available.
01:38 UTC | 17:38 PT
We see that real-time Chat functionality has recovered and transcripts are being created. We are actively working to recover older chat transcripts in this incident. More updates when they become available.
23:37 UTC | 15:37 PT
We've identified the issues affecting Chat Authentication and Chat Transcripts and we're working towards resolution. We'll provide an update when we have additional info to share.
22:27 UTC | 14:27 PT
We have identified and are working to resolve the problem affecting Chat Authentication and Chat Transcripts. Next update in 60 minutes.
21:43 UTC | 13:43 PT
We have confirmed reports of a service disruption affecting Chat Authentication and Chat Transcripts. We're continuing to investigate.
21:20 UTC | 13:20 PT
We are currently investigating reports of access issues for some of our Chat customers. More info to come.
During this incident, a Zendesk Chat cluster went offline due to a bug identified in the stream processing system we are using. This caused features dependent on that cluster to be unavailable, including: agent login, chat saving, ticket creation, chat history, transcripts, real-time monitor, conversion tracking and analytics.
Once we identified the issue, we performed a cluster recovery and implemented configuration updates to lower the risk of future failures. Due to the significant time in bringing the cluster back up, some chat histories and transcripts hit a maximum retry time preventing those chat histories and transcripts from being saved to our database. Chat histories and transcripts that were still available in memory have since been recovered.
To ensure that a similar disruption does not occur again in the future, we have scheduled ongoing work to add failover and chat backup flows alongside additional monitoring and alerts. Internal runbooks have been updated to ensure current failover measures are initiated immediately to prevent chat processing delays.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.