SUMMARY
On December 12, 2023, between 01:34 AM and 04:55 AM UTC, Talk customers experienced issues affecting inbound and outbound calls across all our Pods.
Timeline
03:04 UTC | 19:04 PT
Our Talk service provider is currently investigating inbound and outbound call issues with their carrier partner. Customers may experience errors when making and receiving calls in Talk. More information to come in approximately 1 hour.
03:24 UTC | 19:24 PT
Our Talk service provider is continuing to investigate issues with inbound and outbound calls at their end. We have begun receiving reports of recovery in the service. We are continuing to coordinate efforts with our partner and will provide another update once we receive more information from them.
04:15 UTC | 20:15 PT
Our Talk service provider is reporting partial recovery in voice services. Monitoring continues while partners work toward a full recovery. We will provide another update in 60 minutes or when new information is available.
05:20 UTC | 21:20 PT
We are happy to report that Talk inbound and outbound call issues have been resolved. Thanks for your patience while we worked with our provider to restore service.
POST-MORTEM
Root Cause Analysis
This incident was caused by an issue with our Talk service provider, where an incomplete test configuration was incorrectly deployed by system automation to a portion of their voice call service fleet. This incorrect deployment resulted in a cascade of errors affecting services.
Resolution
To fix this issue, our Talk service provider rolled back all configuration changes. Following their rollback, our engineering teams continued to remediate and recover the services and stuck reservations for several hours.
Remediation Items
- Improve existing monitoring tools for spikes in errors related to the Talk service provider.
- Create additional alerts.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.