SUMMARY
On March 31, 2023 from 21:31 UTC to 22:36 UTC, Zendesk Support customers on Pod 20 may have experienced up to a 15 minute delay on outbound emails being sent.
Timeline
22:14 UTC | 15:14 PT
We are investigating outbound email delivery delays on Pod 20. Next update in 30 minutes.
22:38 UTC | 15:38 PT
The backlog of outbound emails on Pod 20 has been cleared and outbound email delivery has been restored to normal delivery times. Please let us know if you continue to experience delays.
POST-MORTEM
Root Cause Analysis
This incident was caused by a sudden spike of the outbound email queue on Pod 20, a result of a wave of undeliverable traffic.
Resolution
To fix this issue, the small subset of emails that were queued to deliver to undeliverable addresses were removed from the queue, and all other remaining notifications were processed and delivered following the delay.
Remediation Items
- Examine group notification logic as a short term solution, and rate limiting options for more long term solutions.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
0 Comments
Article is closed for comments.