SUMMARY
On January 30, 2023 from 04:31 UTC to 09:59 UTC, Zendesk Explore customers on multiple pods experienced reports not loading on Explore dashboards. Some customers were presented with an error message in the Explore user interface.
Timeline
08:52 UTC | 00:52 PT
We are aware and investigating issues regarding Explore dashboards and reports not loading for EU customers on Pods 17, 18 and 29. We appreciate your patience while the teams work on a fix for this.
09:13 UTC | 01:13 PT
We continue troubleshooting the issue and working on a fix for Explore customers unable to load reports and dashboards. At this time, the team is laser focused on the solution. We will update you with more information in 1h or when we have more details.
10:05 UTC | 02:05 PT
Upon further monitoring, we are happy to confirm full recovery from the issue previously preventing Explore dashboards and reports from loading. Many thanks for your patience and partnership.
POST-MORTEM
Root Cause Analysis
This incident was caused by a memory related issue on a database instance.
Resolution
To fix this issue, a new database instance was created in the specific database cluster, as well as rebooted the problematic database cluster. The read queries from Explore started to be processed again. We continued to monitor until recovery was complete.
Remediation Items
- Investigate on an auto-scale policy in the event of memory issues [Scheduled]
- Work with our database provider to understand root cause of memory issues [Completed]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.