On May 27, 2020 from 00:00 UTC to 16:59 UTC, a subset of Explore customers experienced a delay in datasets refreshing as expected.
17:52 UTC | 10:52 PT
Our team has resolved the issue affecting data delays for some Explore customers. We’re now seeing data refreshes perform as expected.
17:34 UTC | 10:34 PT
Our team has resolved the issue affecting data delays for some Explore customers. We’re continuing to monitor the data refreshes and will provide another update when we are confident we’re in the clear.
14:04 UTC | 07:04 PT
Efforts are continuing to resolve the reported Explore dataset delays. More updates to come
12:58 UTC | 05:58 PT
Our developers continue their work on the previously reported Explore dataset delays. Further updates to follow.
12:11 UTC | 05:11 PT
We continue our investigation of the Explore dataset refresh delays effecting some of our customers. We will post another update as soon as possible.
11:51 UTC | 04:51 PT
We are investigating delayed dataset refreshes for a small number of our Explore customers. More updates to follow.
Root Cause Analysis
This incident was caused by load queries queued up on some data warehouse service clusters which delayed the pipeline and caused data to become stale for customer accounts.
To fix this issue, additional resources were created dedicated to process the pipeline. At 16:59 UTC, all production accounts were unpaused for a dataset refresh, restoring service to normal.
- Add more resources into the production environment
- Investigate and improve efficiency of pipeline processing.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.