Service Incident - May 17th, 2022 - Explore | Pod 20 - Outdated real time data

SUMMARY

On May 17, 2022 from 17:40 UTC to 18:40 UTC, customers on Pod 20 using Zendesk Explore may have experienced outdated real time data. 

Timeline

19:55 UTC | 12:55 PT

Pod 20 Explore Enterprise customers could have seen outdated realtime Support data between 17:40 and 18:40 UTC due to a backend service restart. We apologize for the interruption this may have caused.

POST-MORTEM

Root Cause Analysis

This incident was caused by a surge in the number of support events from a single customer, overloading the data pipeline and resulted in the data processing engine being in a hung state.

Resolution

To fix this issue, the pipeline was redeployed, and the data engine state cleared. We also made a change to stop the account that caused the surge causing an issue again. After that, we continued to monitor until the Explore metrics caught up.

Remediation Items

1. Improve how data is processed by the support pipeline to handle such scenarios [In progress]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.