On October 9, 2019 from 09:22 UTC to 09:51 UTC customers on Zendesk Support, Guide & Talk on Pod 17 experienced degraded performance in Support and Guide and dropped calls in Talk.
09:47 UTC | 02:47 PT
We received reports of a service incident on Pod 17. Our team has confirmed that service is now back to normal. Please let us know if you continue to experience any issues
Root Cause Analysis
This incident was caused by a problematic query which caused elevated error rates and response latency in the database cluster.
To fix this issue, the query was killed to reduce the error rate and restore performance.
- Limit pagination to prevent application from generating these types of queries
- Investigate changes to current pagination mechanism
- Investigate the efficacy of rate limits to Gather and Guide endpoints
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.