SUMMARY
On January 17, 2022 from 08:11 UTC to 10:30 UTC, some Zendesk Explore customers who are hosted on our US infrastructure might have experienced delays in data refresh as a result of an urgent, unscheduled maintenance.
Timeline
08:55 UTC | 00:55 PT
Due to an urgent, unscheduled maintenance, some of our Explore customers hosted on our US infrastructure might experience a delay in data refresh, or not see up to date data. We will provide further updates as soon as possible.
10:50 UTC | 02:50 PT
We are happy to report that the unscheduled maintenance has been completed successfully and all of our Explore customers should see current data in their accounts.
POST-MORTEM
Root Cause Analysis
This incident was caused by a configuration issue on one of our database clusters.
Resolution
To fix this issue, we redirected reads to a temporary database to let customers see their dashboard during the maintenance. Once the maintenance had been completed, database operations were re-enabled. We continued to monitor as the triggered changes were completed, before full recovery was observed.
Remediation Items
- Improve database monitoring for bloating levels [Done]
- Unbloat other internal tables [In progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.