SUMMARY
On June 2, 2022 from 15:14 UTC to 16:32 UTC, Legacy Zendesk Chat customers experienced login issues and were presented with the ‘500 - Internal Server Error’ screen when trying to reach the platform. Another subset of customers using Chat Phase 3/4 (including some Agent Workspace users) may have also experienced intermittent login issues and while trying to add/update/remove account settings.
Timeline
16:20 UTC | 09:20 PT
We are investigating issues logging into standalone legacy Chat accounts. We will provide further updates shortly.
16:34 UTC | 09:34 PT
We have confirmed an issue logging into legacy standalone Chat accounts and our team is investigating. We will provide additional updates as soon as we can.
16:47 UTC | 09:47 PT
We are seeing improvement in the issue affecting log in for legacy standalone Chat accounts and access is returning to normal. We will continue to monitor to ensure full recovery.
17:09 UTC | 10:09 PT
We are happy to report that the issue affecting log in for legacy standalone Chat accounts has been resolved and access has returned to normal. Thank you for your patience during our investigation.
POST-MORTEM
Root Cause Analysis
This incident was caused by a database failover initiated by our provider that resulted in application connectivity issues to the database leading to degradation of several features across the Chat product.
Resolution
To fix this issue, Zendesk Chat application services were redeployed. Stale connections then cleared up and we monitored until full recovery was observed.
Remediation Items
- Improve services to handle database failovers more gracefully [In Progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.