During this incident, customers on Pod 25 experienced delays on outbound emails between 11:09 UTC and 13:15 UTC.
13:31 UTC | 05:31 PT
We experienced an issue with delays on outbound emails between 11:09 UTC and 13:15 UTC for customers on POD 25. The issue has now been resolved, however if you notice any further delay please contact us.
A Database failover event on Pod 25 lead to resque queues to backup. Many resque queues backed up and many jobs were affected. In particular, all workers in the the outbound_mail queue got stuck working on jobs and were unable to deliver mail until redeployed after just under two hours. All other resque queues were impacted as well, though unlike outbound_mail, all other queues were left with at least one functioning worker and were able to continue to process jobs, albeit at slower rate than normal. Additional workers were added to process the backlog of jobs and the workers auto recovered.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.