SUMMARY
On May 12, 2023 from 20:28 UTC to 21:57 UTC, US-based Explore customers experienced errors when trying to view or filter dashboards and reports in Explore.
Timeline
21:28 UTC | 14:28 PT
We are investigating reports of Explore customers receiving errors when trying to load or filter dashboards and reports across multiple pods. We will provide another update as soon as we have more information to share.
21:52 UTC | 14:52 PT
Our engineers have confirmed an issue causing Explore customers to receive errors when trying to load or filter dashboards and reports across multiple pods and are working towards resolution. We will provide another in 30 minutes.
22:15 UTC | 15:15 PT
We are beginning to see improvement in the errors for Explore customers viewing and filtering reports and dashboards. Please let us know if you continue to experience errors with Explore.
POST-MORTEM
Root Cause Analysis
This incident was caused by a connection error reading from the Explore database during a particularly extensive query.
Resolution
To fix this issue, additional resources were allocated and affected services were restarted.
Remediation Items
- Improve monitoring
- Investigate improved handling for similar queries.
- Further exploration for confirmation on the cause of query timeout.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.