SUMMARY
On November 14, 2024 from 14:15 UTC to 23:20 UTC, customers using Explore experienced network errors accessing reports and dashboards.
Timeline
November 14, 2024 06:11 PM UTC | November 14, 2024 10:11 AM PT
We are happy to report that the issue causing "Network Error" messages when loading Explore dashboards or reports has been solved. Thank you for your patience during our investigation.
November 14, 2024 03:18 PM UTC | November 14, 2024 07:18 AM PT
Some Explore users may see a "Network Error" message when loading dashboards or reports. If this happens, clearing your browser’s cookies and cache should resolve the issue. We apologise for the inconvenience.
POST-MORTEM
Root Cause Analysis
This incident was caused by a network change that increased the number of headers sent to the Explore service beyond its configured limit. This led to the service failing silently, resulting in the dashboards not loading for customers.
Resolution
To resolve this issue, the network team was paged to investigate the changes. Upon identifying the problem, they reverted the changes that added excessive headers, which restored normal functionality for the Explore Dashboards.
Remediation Items
- Improve existing implementation tools: Review and enhance the tools used for managing header limits in API requests to prevent similar incidents.
- Create additional alerts: Set up alerts for monitoring header counts and other critical metrics to detect issues before they impact customers.
- Add connection limits on specific applications: Implement connection limits on APIs to ensure they do not exceed operational thresholds, reducing the risk of future incidents.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.