SUMMARY
On February 23, 2022 from 15:54 UTC to 19:06 UTC, some US customers may have experienced errors loading queries and dashboards when using Zendesk Explore.
20:51 UTC | 12:51 PT
We are happy to report that the issue affecting Explore query and dashboard loading for some US customers has been resolved, and any delayed syncs have caught up. Thank you for your patience during our investigation.
18:42 UTC | 10:42 PT
We are seeing improvement in the issue causing intermittent Explore dashboard and query timeouts; however, data syncs could be delayed for up to an hour while we recover. We are monitoring the situation.
17:45 UTC | 09:45 PT
Our team continues to investigate intermittent Explore query and dashboard errors affecting some US-based customers. We will provide additional updates as we learn more.
17:16 UTC | 09:16 PT
We are investigating reports of errors when loading Explore queries and dashboards for some US-based customers. We will provide another update as soon as we can.
POST-MORTEM
Root Cause Analysis
This incident was caused by a configuration error as well as an escaped defect.
Resolution
To fix this issue, we had to route customer traffic to a cloned database to fix identified issues on the live database. Once the issues were resolved, we were able to reroute traffic, and monitored as recovery took place and delayed data syncs caught up.
Remediation Items
- Update runbook to handle similar issues more expediently in future.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.