SUMMARY
On March 5, 2024 from 19:50 UTC to 22:06 UTC, some Talk customers may have experienced connectivity issues including the inability to make, receive or transfer calls.
Timeline
21:36 UTC | 13:36 PT
We are investigating reports of Talk connectivity issues including the inability to make, receive or transfer calls. Next update in 15 minutes or when we have more information.
22:06 UTC | 14:06 PT
Our engineers continue to investigate reports of Talk connectivity issues including the inability to make, receive or transfer calls. Next update in 30 minutes or when we have more information.
22:32 UTC | 14:32 PT
Our investigation continues into the Talk related issues resulting in the inability to make, receive, or transfer calls. Next update in 1 hour or when we have new information to share.
23:31 UTC | 15:31 PT
Our engineers have rolled back a change and we are seeing some recovery on our end. We will continue to monitor performance over the next 24 hours and provide an update at that time.
March 6 - 10:59 UTC | 02:59 PT
Talk functionality is fully restored and confirmed functioning normally by both customer feedback and our internal monitoring. We appreciate your patience during this time. This incident is now resolved.
POST-MORTEM
Root Cause Analysis
This incident appears to have been caused by a local ISP issue for some customers in the US.
Resolution
To fix this issue, we initially rolled back a recent Talk change; however, our investigation points to an ISP outage which more accurately aligns with the window of impact.
Remediation Items
- Improve monitoring and alerting around outbound call failures.
- Work with our provider to adjust alert thresholds so we can be notified faster in similar situations.
- Improve troubleshooting runbooks to more accurately assess scope.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.