Service Incident - May 3, 2023 - Support | Multiple Pods - Triggers Page not loading/timing out

SUMMARY

On May 03, 2023 from 15:55 UTC to 18:57 UTC, Zendesk Support customers on multiple pods may have experienced timeouts or could not load the Triggers page in the Admin Center. 

Timeline

16:20 UTC | 09:20 PT
We are investigating reports of issues accessing Trigger pages to edit Triggers in the Admin Center. Further updates to be provided shortly.

16:47 UTC | 09:47 PT
We have confirmed an issue causing errors when loading Trigger pages in the Admin Center. We will provide additional information as we learn more.

17:48 UTC | 10:48 PT
Our engineers have identified the root cause of the issue causing errors when loading Trigger pages in the Admin Center and are working on remediation. We will provide additional information as we learn more.

19:28 UTC | 12:28 PT
Our engineers have deployed a fix for the trigger page not loading. Please let us know if you continue to run into issues.

POST-MORTEM

Root Cause Analysis

This incident was caused by an application performance capacity issue where the accessibility of workers related to the Sunshine Conversations service was unavailable at the time. When that service’s configurator fetching the list of integrations took longer than expected, the client's read timeout value was higher than the worker’s timeout. This resulted in the worker exiting before the client-side timeout, and causing the Trigger definitions endpoint to end abruptly, and not loading the page properly for customers. 

Resolution

To fix this issue, the timeout of the Triggers’ definition was increased and customers were able to access them on the Admin Page soon after. They might have noticed the Channel definition (dropdown) returning an empty list of integrations until the Sunshine Conversations issue was fully resolved.

Remediation Items

1. Adjust the default open and read timeout to the API call. 
2. Create additional alerts for the Channel Trigger definition functionality.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.