08:17 UTC (18 Jan) | 00:17 PT (18 Jan)
We have successfully reprocessed the delayed emails. If you're still experiencing issues, please reach out to us.
03:34 UTC (18 Jan) | 19:34 PT
We're still working on fixing up the email headers, and reprocessing the delayed emails. We apologize for the inconvenience this may have caused. We will send you further updates on this as soon as possible.
20:55 UTC | 12:55 PT
Correction to the previous message: Outbound e-mails were not affected
20:47 UTC | 12:47 PT
Between 19:49 UTC - 20:25 UTC inbound and outbound email processing in Support for all pods was delayed. We're working on processing delayed emails. We apologize for the inconvenience.
From 19:49 to 20:25 UTC on 17 Jan, any requests coming into Zendesk Support via inbound email were delayed. The cause of this incident was the result of a change made to a configuration that prevents spam emails from being created as tickets. The error in the configuration ultimately caused emails to be rejected completely. Our team was able to quickly identify the configuration change, and that configuration was promptly adjusted to allow the correct requests to be submitted (and spam emails to be appropriately identified and rejected) as originally intended. Though the incident itself was resolved, our teams continued working to reprocess any emails sent during that window, and as of 08:17 UTC on 18 Jan, we have confirmed that those emails have been completely reprocessed.
To prevent this issue from occurring in the future, our team is implementing a thorough vetting process for these types of configuration changes, as well as ensuring the runbook for these operations is updated to reflect that process. Our team is also implementing improved tooling for bulk email reprocessing.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.