Service Incident - September 6, 2022 - Support | All Pods - Emails not creating tickets

SUMMARY

On September 6, 2022 from 16:58 UTC to 18:59 UTC, Zendesk Support customers across all Pods experienced issues with ticket creation from emails. For a subset of customers on Pod 18, the fix was not applied properly and this issue persisted until September 13, 2022 at 16:13 UTC. The subset of unprocessed emails from between September 6 16:47 UTC to September 13 16:12 UTC were processed during a 30 minute timeframe between Sep 13 16:00-16:30 UTC.

 Please note: some tickets submitted during this incident may have been added to the Suspended Tickets view in Support - please make sure to check this for any remaining email tickets.

Timeline

September 6 18:13 UTC | 11:13 PT

We are investigating reports of emails not creating tickets. Further updates coming shortly.

September 6 18:22 UTC | 11:22 PT

We have confirmed an issue with email ticket creation across all pods and our team is working to find a root cause. Additional information will be posted soon.

September 6 18:52 UTC | 11:52 PT

Our engineering team has identified a solution for the issue affecting email ticket creation across all pods, and we are beginning to see improvement. We are monitoring the situation and will provide additional updates as we receive new information.

September 6 19:11 UTC | 12:11 PT

We are happy to report that we have resolved the issue with email ticket creation across all pods. Backlogged emails will create tickets once they are processed. Please check your suspended ticket view for any missing emails.

September 13 21:15 UTC | 14:15 PT

Thank you for your patience as we took a closer look into reports of missing emails persisting after the incident on September 6, 2022. Our engineers found an outstanding issue which caused a subset of additional emails to fail to create tickets. We have put a fix in place to correct this behavior going forward, and emails missed due to this issue will create tickets as a result of the fix. Today our email engineers also attempted to re-process the subset of unprocessed emails that failed to create or update tickets between September 6 16:47 UTC to September 13 16:12 UTC which is why you may be seeing these email tickets and updates only being created now.

We apologize for the interruption and confusion this may be causing you and your team. Please let us know if you have any questions or concerns.

September 14 16:59 UTC | 09:59 PT

One final update. The unprocessed emails from between September 6 16:47 UTC to September 13 16:12 UTC were processed during a 30 minute timeframe between Sep 13 16:00-16:30 UTC yesterday.

If you would like to find all email tickets that were created and updated during that timeframe, you can use the following search queries. Please note that this will also include tickets that were correctly processed at the correct time without delay during that 30 minute time frame.

created>2022-09-13T16:00:00Z created<2022-09-13T16:30:00Z via:mail

updated>2022-09-13T16:00:00Z updated<2022-09-13T16:30:00Z via:mail

Root Cause Analysis

This incident was caused by a configuration issue introduced by an update to a backend process that manages email ticket creation.

Resolution

To fix this issue, our team identified the offending change and reverted it. Once the change was reverted, backlogged emails were processed and created tickets in Support.

Remediation Items

1. Reprocess a subset of emails delivered during the incident that was rejected due to email channel rate limits.
2. Improve runbook and documentation to reduce time to response and resolution.
3. Adjust monitoring and alerting to lower the threshold for response.
4. Revisit training to enable broader response internally.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.