On July 12, 2019 from 13:44 UTC to 14:56 UTC Zendesk customers on Pod 20 experienced degraded performance in Support and dropped calls while using Talk.
16:32 UTC | 09:32 PT
We’re happy to report the issues affecting Talk and Support on Pod 20 have been identified and are now resolved.
15:28 UTC | 08:28 PT
Pod 20 has been stabilized for Talk & Support products. Our team is still investigating and will continue monitoring the status of this issue until the cause has been identified and resolved.
14:54 UTC | 07:54 PT
We continue to investigate the Talk and Support issues previously mentioned on Pod 20. More info in 30 minutes.
14:32 UTC | 07:32 PT
We are currently investigating reports of latency in Support and dropped calls for Talk customers on Pod 20. Please bear with us, more info to come.
Root cause Analysis
This incident was caused by an issue with an API push endpoint that caused a query pile-up, impacting traffic on Pod 20.
To fix this issue, the hanging queries were killed.
Further load testing will be performed for the API push endpoint and protective mechanisms have been added to prevent this issue from recurring.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.