21:50 UTC | 14:50 PT
The performance issues on POD 13 have stabilized. Please let us know if you still see issues.
20:48 UTC | 13:48 PT
We are beginning to see improvements in performance on POD 13. We are continuing to monitor the situation and will update shortly.
20:34 UTC | 13:34 PT
Investigation into POD 13 performance issues (incl. green screens and loading delays) is continuing. We will update once we have more info.
20:04 UTC | 13:04 PT
Our team continues the investigation of the ongoing issues affecting POD 13. Please let us know if you see any improvement on your instance.
19:35 UTC | 12:35 PT
While we continue investigating this issue, users may experience green screens and increased loading times on Tickets and in HC.
19:17 UTC | 12:17 PT
We are still investigating performance affecting POD 13. Users will see green screens and loading delays.
18:59 UTC | 11:59 PT
We are investigating reports of decreased performance on POD 13.
Several issues were reported involving services being unable to connect to various endpoints. Based on the error messages reported, it was determined that DNS name resolution was failing or taking an excessive amount of time, causing timeouts and failed connections. The CPUs of dnscache hosts were overcome by the amount of DNS requests and verbose logging. We reduced log verbocity and restarted the DNS service. Services recovered after the change and restart. To prevent this from happening again in the future, we have reduced log verbosity in DNS cache, will optimize DNS performance, and update monitoring for CPU load and DNS resolution.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.