SUMMARY
08:07 UTC | 01:07 PT
Issues with Inbound email for POD 15 and 25 have now been fixed. If you continue to experience any problems, send us an email to support@zendesk.com
07:47 UTC | 00:47 PT
Our team is working on inbound email issues for customers on Pod 15 and 25. We have rolled out a fix for Pod 15 and Pod 25. We are continuing to monitor both Pods
07:14 UTC | 00:14 PT
We are observing some issues with inbound email queueing for pods 15 and 25. Our team is currently investigating.
POST-MORTEM
From 2020-09-24 06:17 UTC to 2020-09-24 07:45 UTC, customers experienced delays in inbound emails being converted to tickets. The incident was caused by a failed deployment of our email processing code on pods 15 and 25 which left the existing version continuing to run and processing email, but in a state where it could not be successfully restarted. Before investigations into the failed deployment could be completed an automated job killed the email processing code, which was then unable to be automatically restarted by the system. An explicit redeploy of the existing version was performed but did not resolve the issue. Eventually a number of directories containing all email processing releases and their gems were manually deleted from the affected hosts and a fresh deployment installed. In order to prevent this from happening again in the future, we have rolled back the code change and will be reviewing our current email processing upgrade and deployment process.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.