On May 8, 2023 from 13:20 UTC to 15:30 UTC, US-based Explore customers experienced errors when trying to view or filter dashboards and reports in Explore.
14:27 UTC | 07:27 PT
We are actively investigating reports of seeing errors when trying to load or filter dashboards and reports in Explore for customers across multiple Pods. We will follow up with more information soon.
14:54 UTC | 07:54 PT
We have confirmed an issue causing errors in Explore when trying to view or filter reports and dashboards across all pods and our team is working towards a root cause. We will follow up with another update in 30 mins or when new information becomes available.
15:26 UTC | 08:26 PT
After some investigation our team has determined the issue causing errors in Explore when viewing or filtering reports and dashboards is isolated to US-based pods. Our team continues to work towards a root cause.
15:45 UTC | 08:45 PT
We are beginning to see improvement in the errors for US-based Explore customers viewing and filtering reports and dashboards. Please let us know if you continue to experience any issues.
15:59 UTC | 08:59 PT
We are happy to report that the issue affecting US-based Explore customers, causing errors viewing and filtering reports and dashboards has been resolved. Thank you for your patience during our investigation.
Root Cause Analysis
This incident was caused by a connection error reading from the Explore database during a particularly extensive query.
To fix this issue, additional resources were allocated and affected services were restarted.
- Investigate improved handling for similar queries.
- Further exploration for confirmation on cause for query timeout.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.