Service Incident - May 23, 2023 - Support | All pods - EJD Job Dispatcher stops working (webhooks)

SUMMARY

On May 23, 2023 from 15:03 UTC to 18:39 UTC, Zendesk Support customers on all pods may have experienced delays and failures with webhooks. 

Timeline

17:48 UTC | 10:48 PT
We are investigating reports of significant delays in Support webhooks firing on multiple pods. More information will be posted shortly.

17:59 UTC | 10:59 PT
We have confirmed an issue causing delays in Support webhook delivery across multiple pods and our team is investigating. We will post additional information as soon as we can.

18:26 UTC | 11:26 PT
Our team continues to investigate possible causes for delays in Support webhooks across multiple pods. We will post further updates as we learn more.

19:11 UTC | 12:11 PT
We have found a root cause for the issue causing delays in Support webhooks firing across multiple pods and are beginning to see improvement. Our team will monitor until full resolution.

19:28 UTC | 12:28 PT
We are now seeing recovery of Support webhooks firing across all pods. Our team will continue to monitor performance until full resolution.

19:41 UTC | 12:41 PT
The issue causing delays in Support webhooks firing across all pods is now fully resolved. Please let us know if you continue to experience issues.

POST-MORTEM

Root Cause Analysis

This incident was caused by an erroneous configuration setting preventing the Support webhook queuing processors from refreshing their DNS cache and picking up any new settings. 

Resolution

To fix this issue, we restarted the processors to force a DNS refresh.  Recovery was observed thereafter.

Remediation Items

1. Fix the backend erroneous configuration setting [Done]
2. Review the EJD Dispatcher SDK to avoid similar configuration issues [Done]
3. Improve monitoring and logging for the webhook queuing system [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.