Service Incident - October 10th, 2019 - Chat Web Widget Issues

Summary

On October 10, from 3:48AM UTC to 11:20AM UTC customers who had the web widget embedded on their site but didn’t have help center enabled in their widget would have been unable to receive chats from their users. In most of these cases, the users wouldn't have seen the widget load on their sites.

Timeline

11:43 UTC | 04:43 PT
We have resolved the issue impacting Chat Status on the Web widget. Thank you for your patience.

11:22 UTC | 04:22 PT
We have identified the cause of the issue impacting Chat Status on the Web widget. Our teams are working towards remediation. Thank you for your patience.

09:49 UTC | 02:49 PT
We are actively investigating reports of the Chat widget stating that it is offline for some customers. We will provide further information shortly. 

Root cause Analysis

A feature related to Chat Visitor Registration was deployed. When this feature is turned on, the websocket connection to the chat backend is delayed until a specific action is performed. While the feature was disabled, another deploy enabled the Chat Visitor Registration feature in the widget for certain accounts. Because the websocket connection was delayed for these accounts, the widget didn’t detect when agents were online and wouldn't surface the chat feature to users. Once the issue was identified, a fix was deployed to disable the feature.

Remediation Items

1. Deploy fix to disable feature.
2. Put Chat Visitor Registration feature behind production deploy safeguards.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.