Summary
From November 13, 2023 at 18:26 UTC until November 14, 2023 at 09:18 UTC, a small subset of Zendesk Support customers on Pods 15, 26 and 28 experienced outbound email notification delays.
Timeline
07:32 UTC | 23:32 PT
We have received several reports of delayed outbound email notifications for Support customers on Pods 15 and 26. This is separate to another ongoing issue with Gmail-specific addresses and seems to be impacting multiple domains. We are investigating this issue at the highest priority and will send the next update in 1 hour.
08:33 UTC | 00:33 PT
We continue to investigate the issue with outbound email notifications for Support customers on Pods 15 and 26. We will provide another update on this in 1 hour.
09:27 UTC | 01:27 PT
We have implemented a fix for the outbound email delays to customers on Pods 15 and 26 and can confirm that Pod 28 was also affected. All emails will be retried. The services have returned back to normal for this incident and we will now consider this as resolved from our side. Thank you for your patience.
POST-MORTEM
Root Cause Analysis
This incident was caused by a bug in our mail transfer system configuration.
Resolution
To fix this issue, our team disabled the feature that contained the bug and implemented alternative options to allow mail to flow out again. Our mail transfer partner resolved the underlying bug within 24 hours.
Remediation Items
- Additional monitoring to pick up outbound mail issues quicker [Completed]
- Mail transfer partner bug resolution [Completed]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.