On May 31, 2021 from 00:00 UTC to 01:31 UTC, customers trying to connect to Zendesk Chat via the Web Widget may have experienced connection issues when starting a chat session with an Agent in Ireland (Northern Europe) and US West Coast. Customers using Agent Workspace (Support) would have had some ‘dropped messages’ where tickets may not have been created. These dropped messages were automatically backfilled by our systems.
01:19 UTC | 18:19 PT
We are experiencing Chat connectivity issues for users near the West Coast of the US and Ireland. Our teams are working on a resolution and we will update in 30 minutes.
01:54 UTC | 18:54 PT
We have confirmed that the Chat connectivity issues for users near the West Coast of the US and Ireland has been resolved. Thank you for your patience.
Root Cause Analysis
This incident was caused by a SSL certificate configuration error on two Zendesk Chat mediators in the impacted regions. While updating multiple certificates across Zendesk Chat infrastructure, an assumption was made that new certificates would be deployed in place of old ones by the external certificate management system of these two impacted regions. Even though the correct certificates were uploaded and verified, an error occurred. Service degradation was observed and we moved to address the issue.
To fix this issue, Zendesk engineers removed the expired additional certificates from the certificate list in the 2 affected regions. Recovery was observed thereafter.
- Further improve our certificate validation scripts.
- Additional error logging to capture more verbose logs
- Improved message handling mechanism for mitigating ‘dropped messages’
For More Information
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.