SUMMARY
On June 1st, 2021 from 09:48 UTC to 09:55 UTC, some customers using Zendesk Explore in the EU region experienced a UI message with “Error Code 4” when loading dashboards, which prevented them from seeing results.
10:44 UTC | 03:44 PT
We’re happy to report that all known issues related to the Explore incident have been resolved. We will publish a post-mortem on these events as soon as it is completed.
10:31 UTC | 03:31 PT
We have confirmed a service disruption affecting Explore EU accounts leading to errors loading dashboards. We are working toward resolution and already see stability in the platform.
10:13 UTC | 03:13 PT
We are currently investigating reports of issues in Explore for some customers. Affected customers will see an error code 4. Investigation is underway.
POST-MORTEM
Root Cause Analysis
This incident was caused by a deployment that caused the API to be down while rotating containers with the new code.
Resolution
To fix the issue, old jobs were replaced by the new ones with an up-to-date code version, and with the correct sync, the API became functional again.
Remediation Items
- Create additional alerts with a smaller time frame between seeing errors and alerting engineers.
- Establish a deployment process for this particular edge case.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.