Service Incident - November 16, 2022 - Chat | Multiple Pods - Triggers not firing as expected

Summary

On November 16, 2022 from 13:15 UTC to 16:40 UTC, a subset of Zendesk Chat customers experienced an issue with Chat triggers not firing in a timely manner. 

Timeline

16:14 UTC | 08:14 PT

We are investigating reports of Chat triggers intermittently firing when they should not. More information to follow shortly.

16:29 UTC | 08:29 PT

We have confirmed an issue causing Chat triggers to intermittently fire when their conditions haven't been met. Our team is investigating and we will provide additional information once it's available.

17:25 UTC | 09:25 PT

While we have seen some improvement, our team continues to investigate the issue causing a significant delay in Chat triggers firing. We will provide further updates when we have new information to share.

17:39 UTC | 09:39 PT

We are happy to report that the issue causing a significant delay in Chat triggers firing has been resolved. Thank you for your patience during our investigation.

Root Cause Analysis

This incident was caused by lag in one instance of our Chat trigger service as a result of increased traffic volume. While trigger conditions were being met in our customers’ Chat accounts, the service that performs the trigger actions was delayed in performing those actions. An influencing factor to the extended resolution time is that alerts didn’t page our engineering team as expected.

Resolution

To fix this issue, we redeployed the service on the affected instance that resulted in immediate improvement to the observed lag. Full recovery was observed after the backlog was processed.

Remediation Items

1. Fix service alerts to ensure teams are paged immediately.
2. Investigate the cause of lag in the affected server instance.
3. Investigate mechanism for trigger service to clear consumer backlog.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.