On October 4, 2019 from 20:12 UTC to 20:26 UTC customers on Zendesk Support, Guide & Talk on Pod 19 experienced slow loading, errors, and degraded performance in Support and Guide as well as dropped calls in Talk.
20:42 UTC | 13:42 PT
We experienced a brief issue affecting performance on Pod 19 Zendesk Support accounts. The issue is now resolved. Please let us know if your account is still experiencing issues.
Root cause Analysis
Multiple similar types of Guide originated queries caused query pileups on the database reader nodes. A high number of long running queries resulted in high CPU load on the reader nodes. The services which were using reader nodes of the 19.8 cluster were affected because of high CPU load and latency. Adding a temporary index to the database cluster resolved the issue.
- Add index to optimize slow running queries
- Improved logging and monitoring for queries
- Investigate usage of a circuit breaker when queries run slow
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.