On January 17, 2022 from 08:11 UTC to 10:30 UTC, some Zendesk Explore customers who are hosted on our US infrastructure might have experienced delays in data refresh as a result of an urgent, unscheduled maintenance.
08:55 UTC | 00:55 PT
Due to an urgent, unscheduled maintenance, some of our Explore customers hosted on our US infrastructure might experience a delay in data refresh, or not see up to date data. We will provide further updates as soon as possible.
10:50 UTC | 02:50 PT
We are happy to report that the unscheduled maintenance has been completed successfully and all of our Explore customers should see current data in their accounts.
Root Cause Analysis
This incident was caused by a configuration issue on one of our database clusters.
To fix this issue, we redirected reads to a temporary database to let customers see their dashboard during the maintenance. Once the maintenance had been completed, database operations were re-enabled. We continued to monitor as the triggered changes were completed, before full recovery was observed.
- Improve database monitoring for bloating levels [Done]
- Unbloat other internal tables [In progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.