13:11 UTC | 06:11 PT
Issues affecting performance on POD14 have been resolved. Thank you for your patience.
12:56 UTC | 05:56 PT
We are seeing improvements with Support on POD14. More information to follow.
12:35 UTC | 05:35 PT
Our Operations team are investigating an issue affecting Support on POD14. More details to follow.
At 12:08 UTC, a database cluster node was restarted, causing the master to failover to a new master node. The failover itself began at 12:09 UTC and was completed two minutes later at 12:11 UTC. Though the failover had been completed, service application errors continued, and we quickly determined that those services were unaware of the database failover and continued trying to work with the original master. Restarting those services at 12:35 UTC allowed them to select the correct master, and the incident was resolved at 12:36 UTC. To prevent this issue from occurring in the future, our team will implement circuit breakers that identify database failovers in service applications that will allow us to failover more quickly without any impact to those applications.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.