SUMMARY
On August 20, 2024 from 00:13 to 10:16 UTC, Support customers on Pods 13, 18, 19, 23, and 25 experienced delays with emails not creating or updating tickets.
Timeline
August 20, 2024 02:12 AM UTC | August 19, 2024 07:12 PM PT
We are aware of an issue with emails not creating or updating tickets across multiple pods. Investigations are currently ongoing and we will provide another update in 30 minutes.
August 20, 2024 02:32 AM UTC | August 19, 2024 07:32 PM PT
We have indications in our logs that the issue causing ticket creation and updates may be related to a traffic spike. We are making progress in our investigations, but have not nailed down a root cause just yet. More information in the next 30 minutes.
August 20, 2024 02:59 AM UTC | August 19, 2024 07:59 PM PT
Traffic spikes in our email system may result in customers experiencing delays in ticket creation or receiving ticket updates. We are looking at ways to reduce the traffic so that the email queue can be processed. Next update expected in 30 minutes.
August 20, 2024 03:31 AM UTC | August 19, 2024 08:31 PM PT
We continue to investigate the issue with our email service that is resulting in delayed ticket updates and ticket creation. Our team is working to mitigate the issue. This issue may affect accounts on pods 13, 18, 19, 23, 25 and 28. Next update in 30 minutes.
August 20, 2024 04:03 AM UTC | August 19, 2024 09:03 PM PT
We are currently testing a potential fix for the Support ticket delays. Once that is confirmed to work as intended, we expect to see a reduction in our queued email backlog, resulting in tickets being created and updated. We will provide another update in 30 minutes. Thanks for your patience while we work through this issue.
August 20, 2024 04:41 AM UTC | August 19, 2024 09:41 PM PT
Our teams are working at the highest priority to fix the issue with email processing affecting Support ticketing. We expect all emails sent during this incident to start reprocessing once the issue is resolved. We will provide the next update as soon as we have progress to share.
August 20, 2024 05:33 AM UTC | August 19, 2024 10:33 PM PT
We continue to work on a fix for the email processing issues on multiple pods. We are rolling out our first potential fix now after successfully testing it. Apologies for the extended impact today; we will provide the next update when we have one. Thank you.
August 20, 2024 07:29 AM UTC | August 20, 2024 12:29 AM PT
Our team has rolled out a fix to all affected pods in an attempt to resolve the Support email ticket issues. We are monitoring our email backlog for improvements.
August 20, 2024 08:41 AM UTC | August 20, 2024 01:41 AM PT
We can see an improvement to our backlog and are continuing to monitor. Once the backlog has cleared, tickets will start creating at a normal pace again. Unfortunately there is no ETA or workaround at this time. If you need to access the emails in the meantime, we recommend checking the email inbox of the linked email address. We will provide another update when we have one.
August 20, 2024 10:27 AM UTC | August 20, 2024 03:27 AM PT
Our team has cleared the backlog, and ticket creation has returned to normal operations. With this, the service incident has been resolved. The backlog has been completed and tickets created or updated in chronological order. All emails impacted by the incident have been reprocessed and none were lost. Please note that you will not need to reconfigure the forwarding integration. We apologise for any inconvenience this may have caused and appreciate your patience.
POST-MORTEM
Root Cause Analysis
This incident was caused by an unexpected bug in email routing logic which created an internal routing loop, causing a backup on email processing queues which led to delays.
Resolution
To fix this issue, we released an update to address the bug in routing logic and allocated additional resources to allow the backlog of emails to process.
Remediation Items
- Improve email routing logic to avoid similar scenarios in the future.
- Revisit alerting to improve response times for growing email processing queues.
- Update documentation and runbooks.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.