19:49 UTC | 12:49 PT
We can confirm Pod 13 is stabilized and access is restored. Post-mortem to follow.
19:34 UTC | 12:34 PT
We are investigating system access issues such as green screens and loading failures in Pod 13 accounts and are seeing some recovery.
POD 13 is an AWS hosted pod. For the MySQL layer the AWS database service is used. The service experienced an underlying hardware failure and an automated failover was initiated. Just prior to the failover until the completion of the recovery customers would have experienced various errors in completing operations. Once the failover and subsequent restart of the AWS database service was completed, services returned to normal without any further intervention. We are working with AWS to investigate why the hardware failure occured in order to prevent this from happening again in the future.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.