SUMMARY
From October 7, 2020 at 15:29 UTC to October 8, 2020 at 04:22 UTC, a subset of Zendesk Explore customers experienced page crashes when attempting to access the Queries Library.
03:35 UTC | 20:35 PT
We are investigating issues with accessing the Queries Library within Explore that may be impacting some customers.
04:36 UTC | 21:36 PT
We have identified the issue preventing users from accessing the Queries Library within Explore and we have rolled out a fix. Please hard refresh your browser if you do not see improvements.
05:04 UTC | 22:04 PT
The issues impacting users from accessing the Queries Library within Explore have now been resolved. We thank you for your patience and apologise for any inconvenience caused.
POST-MORTEM
Root Cause Analysis
The initial root cause was incorrectly identified as a network/third-party issue which resulted in a second incident shortly after. Further analysis of this incident reveals it was caused by the implementation of a new date formatting library in the Explore frontend. The new library was implemented with an incorrect date formatting configuration which led to requests sent to the backend to fail due to a format mismatch.
Resolution
To fix this issue, our engineering team reverted the deploy that contained the new library. The new library has been rolled out again following the incident with the repaired configuration.
Remediation Items
- Additional logging to reveal Graphql API errors in Explore logs [Scheduled].
- Fail gracefully on Graphql calls to prevent full page crashes [Scheduled].
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.