On November 23, 2022 from 14:12 UTC to 20:41 UTC, Zendesk Chat customers across all Pods experienced issues with Chat triggers misfiring, causing them to fire during unexpected times.
19:44 UTC | 11:44 PT
We are aware of an issue causing delays in Chat triggers firing, and our team is already investigating. We will provide additional information as soon as it's available.
20:56 UTC | 12:56 PT
Our engineers have identified the issue causing delays in Chat triggers firing and are working to resolve it. We will provide another update as soon as we have more information.
21:58 UTC | 13:58 PT
We're happy to report that the issue causing delays in Chat triggers firing is now resolved. Thank you for your patience as we worked to resolve the issue.
Root Cause Analysis
This incident was caused by a degradation of the Chat trigger service, resulting in trigger conditions being evaluated incorrectly and firing unexpectedly.
To fix this issue, more resources were allocated, a fix was put in place and the service rebooted. Improvements in latency were noted, and we continued to monitor until recovery was observed.
- Improved alerts so that the team is paged earlier to mitigate impact.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.