19:54 UTC | 11:54 PT
All inbound emails that were delayed have been processed. If you are still experiencing email issues, please reach out to us.
19:18 UTC | 11:18 PT
There was a brief delay in inbound email processing for customers in Support. We are currently monitoring the situation. Please reach out to us if you’re experiencing issues.
During this incident, customers across all pods experienced inbound email delays. This did not affect outbound email processing, though inbound delays also resulted in any notifications sent as a result of the delayed inbound processing. This incident was caused by some backend maintenance work in which two pieces of code still in use in the production environment were deprecated. When that code was read, the code was reactivated, but the parsing queue is on a read-only connection with the database, thus any updates with that code failed and caused, leading to delays in inbound email processing.
Our teams rapidly identified the issue by seeing email processing graphs in our monitoring tool drop, leading to a discovery in exceptions in parsing queue logs. Recovery was fast, and any inbound emails “stuck” in the queue were moved along to process as normal.
To prevent further occurrences of incidents of this nature, our teams are implementing more thorough monitoring tools for the parsing queue, as well as developing a new runbook on best practices with the aforementioned code in situations that require deprecation or utilize read-only databases.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.