Summary

On January 31, 2024 from 03:52 UTC to 06:52 UTC, Zendesk Chat customers on multiple pods may have experienced failures with Chat triggers impacting features such as chat routing, setting departments and offline messages.

Timeline

06:19 UTC | 22:19 PT
We are investigating an issue affecting Chat trigger creation, update and execution on multiple pods. Customers may experience issues with Chat routing, proactive messaging, setting departments and other related actions. We will provide an update on this issue as soon as possible.

07:53 UTC | 23:53 PT
We have fixed the issue with Chat triggers on all pods and are currently in recovery. Chat trigger actions such as routing, setting departments and offline messages should be working for all new chats. A final update will be posted upon full resolution.

09:17 UTC | 01:17 PT
The fix that was rolled out has resolved all issues with Chat triggers on all pods. We will now consider this incident as concluded. Thank you for your patience while we worked through this.

Root Cause Analysis

This incident was caused by the introduction of a new internal authentication configuration in the Chat trigger service. Despite testing this code change through our staging environment, the issue wasn’t revealed due to the old authentication credentials remaining valid for an extended time period beyond our testing phase.

Resolution

To fix this issue, our team identified the erroneous code change and reverted it.

Remediation Items

  1. Update deployment process to with longer soak times in pre-production environments [Scheduled]
  2. Implement additional alerting [Scheduled]
  3. Investigate mechanism for parallel restarts of hosts [Scheduled]
  4. Update internal runbooks [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

Powered by Zendesk