On May 17, 2022 from 17:40 UTC to 18:40 UTC, customers on Pod 20 using Zendesk Explore may have experienced outdated real time data.
19:55 UTC | 12:55 PT
Pod 20 Explore Enterprise customers could have seen outdated realtime Support data between 17:40 and 18:40 UTC due to a backend service restart. We apologize for the interruption this may have caused.
Root Cause Analysis
This incident was caused by a surge in the number of support events from a single customer, overloading the data pipeline and resulted in the data processing engine being in a hung state.
To fix this issue, the pipeline was redeployed, and the data engine state cleared. We also made a change to stop the account that caused the surge causing an issue again. After that, we continued to monitor until the Explore metrics caught up.
- Improve how data is processed by the support pipeline to handle such scenarios [In progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.