Summary
On Sunday, October 3, 2021 from 22:21 UTC to 22:53 UTC, Zendesk Talk customers across all Pods may have experienced dropped voice calls. Approximately 35% of the impacted calls degraded to Voicemail and the rest were completely dropped.
Timeline
22:56 UTC | 15:56 PT
We have detected dropped Talk calls across all Pods. Our team is investigating. More to come soon.
23:14 UTC | 16:14 PT
We are seeing improvements with Talk calls across all Pods. Thanks for your patience while our teams continue to investigate the issue. Next update in 30 minutes.
23:51 UTC | 16:51 PT
Our telephony partner has identified and fixed the issue that resulted in dropped calls in Talk. We continue to monitor system stability and will post an update once the issue is fully resolved.
00:06 UTC | 17:06 PT
The issue with dropped calls in Talk has now been fully resolved by our service provider and systems are stable. We apologize for this issue today and appreciate your patience while we worked through it.
Root Cause Analysis
This incident was caused by a capacity issue in our service provider’s infrastructure. This issue occurred when extremely large files on our provider’s infrastructure exhausted server memory making them unserviceable.
Resolution
To fix this issue, our service provider replaced the affected hosts.
Remediation Items
Our service provider has permanently increased capacity to handle large files on the affected infrastructure and reconfigured autoscale capacity. Restrictions around large files are being reviewed while other multi-tenancy protections are planned.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.