SUMMARY
On October 27, 2020 from 03:03 UTC to 16:01 UTC, a small subset of Zendesk Chat customers experienced login issues, observed empty data in their existing Chat accounts and errors loading the Web Widget.
TIMELINE
15:12 UTC | 08:12 PT
We are investigating an issue in which a small subset of customers are linked to a new version of Chat, appearing as a new account, however your prior Chat account history is safe. To access your existing legacy Chat account, please log in at dashboard.zopim.com.
16:15 UTC | 09:15 PT
We have identified the root cause of this issue and have taken steps to prevent this from affecting further accounts. Our teams are now actively working to restore full functionality for the impacted customers.
04:57 UTC | 21:57 PT
We have restored full functionality for all impacted customers. If you experience any issues, please don't hesitate to reach out.
POST-MORTEM
Root Cause Analysis
This incident was caused by new code deployed to our Account service. The new logic worked off known Chat account types and revealed that a select group of Chat accounts had incorrect account type values in our database. These incorrect account types caused the Account API to direct requests for Chat data to an empty Chat instance, leading to the issues experienced in this incident.
Resolution
To fix this issue, our team reverted the code and cleared the Account service cache.
Remediation Items
- Add test scenarios to cover all Chat account types [Scheduled]
- Include additional code reviewers to vet similar changes in future [Scheduled]
- Add validation to Account service to reject invalid account types [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.