SUMMARY
May 27, 2024 05:18 PM UTC | May 27, 2024 10:18 AM PT
Between 12:00 and 16:55 UTC on May 27, 2024, Gmail customers may have experienced delayed notification delivery. Email queues have caught up and all notifications have been processed at this time.
Post-Mortem
On May 27 2024, users experienced delays in email notifications due to a loss of temporary configuration settings. This affected the timely delivery of emails but has since been corrected.
Root Cause Analysis
The delay was caused by a loss of temporary fixes applied to manage email delivery, which were erased during routine system maintenance. These temporary fixes were not intended to persist through maintenance cycles.
Resolution
To resolve the issue, a new version of our software was deployed that included a permanent fix to the configuration, restoring normal service for email notifications.
Remediation Items
1. We will ensure that important configuration changes are applied in a way that they remain persistent.
2. Monitoring alerts will be re-enabled to catch similar issues more promptly in the future.
3. We have requested a feature from our software provider to allow dynamic rules to persist through maintenance. This will help prevent similar issues from recurring.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.
1 comment
Jessica G.
Post-mortem published June 17, 2024.
0