Summary
On March 28, 2022 from 15:01 UTC to 15:44 UTC, Zendesk Talk customers on all Pods experienced major degradation through a high percentage of dropped calls.
Timeline
15:19 UTC | 08:19 PT
We are investigating reports of dropped calls across multiple pods. We will provide another update as soon as we can.
15:33 UTC | 08:33 PT
We have confirmed a number of dropped calls across all pods in Talk. We are investigating and will provide additional information as soon as we can.
16:02 UTC | 09:02 PT
Our team continues to investigate dropped calls in Talk for all pods, and we are working with our service provider to find a root cause. We will provide further updates as we receive new information.
16:32 UTC | 09:32 PT
We are seeing improvement in Talk, and calls are being placed successfully. We are monitoring the situation and will provide another update as we reach resolution.
16:50 UTC | 09:50 PT
We are happy to report that the issues causing dropped calls in Talk across all pods have been resolved. Thank you for your patience during our investigation.
Root Cause Analysis
This incident was caused by a capacity issue in our service provider’s infrastructure due to an under-provisioning of their signing and verification service.
Resolution
To fix this issue, our service provider increased the number of hosts that carried the impeded service, leading to our call success rate reaching normal levels once again.
Remediation Items
- (Zendesk) Investigate unrelated database errors that led to unnecessary noise during the initial stages of the service incident [Scheduled]
- (Service provider) Immediate focus on resolving internal technical remediation items from this incident as well as reviewing deployment processes and staff training.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.