Service Incident - November 23, 2022 - Chat | All Pods - Issue with Chat triggers firing

SUMMARY

On November 23, 2022 from 14:12 UTC to 20:41 UTC, Zendesk Chat customers across all Pods experienced issues with Chat triggers misfiring, causing them to fire during unexpected times. 

Timeline

19:44 UTC | 11:44 PT

We are aware of an issue causing delays in Chat triggers firing, and our team is already investigating. We will provide additional information as soon as it's available.

20:56 UTC | 12:56 PT

Our engineers have identified the issue causing delays in Chat triggers firing and are working to resolve it. We will provide another update as soon as we have more information.

21:58 UTC | 13:58 PT

We're happy to report that the issue causing delays in Chat triggers firing is now resolved. Thank you for your patience as we worked to resolve the issue.

POST-MORTEM

Root Cause Analysis

This incident was caused by a degradation of the Chat trigger service, resulting in trigger conditions being evaluated incorrectly and firing unexpectedly.

Resolution

To fix this issue, more resources were allocated, a fix was put in place and the service rebooted.  Improvements in latency were noted, and we continued to monitor until recovery was observed.

Remediation Items

1. Improved alerts so that the team is paged earlier to mitigate impact.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.