From October 19, 2023 at 22:03 UTC until October 20, 2023 at 11:10 UTC, Zendesk Support customers experienced an issue where trigger and automation events were missing from ticket events.
08:41 UTC | 01:41 PT
We are aware of an issue that is causing business rules information not to show in ticket events in Support across all Pods. The business rules are still firing and are working in the background. As a workaround the information can still be seen via API. We will continue our investigation and will update you shortly.
09:17 UTC | 02:17 PT
Our investigation continues into the cause of business rules not showing in ticket events. We will provide another update in 30 mins.
09:43 UTC | 02:43 PT
We believe that we have found the cause of business rules not showing in ticket events and are working on a fix. We will update you in 30 mins or as soon as more information becomes available.
10:13 UTC | 03:13 PT
We continue to work on a fix for the issue with business rules information in ticket events. Expect another update in 30 mins or as soon as information is available.
10:50 UTC | 03:50 PT
We’ll start rolling out a fix across Pods, this might take at least 1h to show up in your account and will require clearing cache and cookies - next update once this is finished.
11:26 UTC | 04:26 PT
The fix has been deployed and the issue has been resolved. Please make sure to clear your cache and cookies to try again. Thank you for your patience while we investigated this, we will now consider this incident as solved.
Root Cause Analysis
This incident was caused by a bug in a recent code change. This bug wasn’t caught before its release to our production environment due to an insufficient test case. The bug related to secondary ticket events and these events were not covered in the test, allowing it to pass and the code change to be released to Support.
To fix this issue, our engineering team corrected the code and released it to the Support production environment.
- Add test to cover the presence of secondary ticket events [Completed]
- Expand smoke testing to cover this trigger scenario [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.