Service Incident - May 8, 2023 - Explore | Multiple Pods - Error when loading and viewing dashboards

SUMMARY

On May 8, 2023 from 13:20 UTC to 15:30 UTC, US-based Explore customers experienced errors when trying to view or filter dashboards and reports in Explore.

Timeline

14:27 UTC | 07:27 PT

We are actively investigating reports of seeing errors when trying to load or filter dashboards and reports in Explore for customers across multiple Pods. We will follow up with more information soon.

14:54 UTC | 07:54 PT

We have confirmed an issue causing errors in Explore when trying to view or filter reports and dashboards across all pods and our team is working towards a root cause. We will follow up with another update in 30 mins or when new information becomes available.

15:26 UTC | 08:26 PT

After some investigation our team has determined the issue causing errors in Explore when viewing or filtering reports and dashboards is isolated to US-based pods. Our team continues to work towards a root cause.

15:45 UTC | 08:45 PT

We are beginning to see improvement in the errors for US-based Explore customers viewing and filtering reports and dashboards. Please let us know if you continue to experience any issues.

15:59 UTC | 08:59 PT

We are happy to report that the issue affecting US-based Explore customers, causing errors viewing and filtering reports and dashboards has been resolved. Thank you for your patience during our investigation.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by a connection error reading from the Explore database during a particularly extensive query.

Resolution

To fix this issue, additional resources were allocated and affected services were restarted.

Remediation Items

1. Investigate improved handling for similar queries. 
2. Further exploration for confirmation on cause for query timeout.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.