SUMMARY
Between 8:37 and 10:07 UTC on November 10, 2023, Talk customers receiving calls from end-users in Denmark, Finland, or Sweden experienced issues with those calls connecting to Talk, or disconnecting shortly after initially connecting.
Timeline
10:53 UTC | 02:53 PT
We are receiving reports of customers having issues with inbound calls to Scandinavian numbers not reaching Talk or disconnecting shortly after initialising. Investigation is underway and we are partnering with our telephony partner. We will provide another update shortly.
11:30 UTC | 03:30 PT
We are continuing to investigate issues regarding inbound calls to Scandinavian numbers not reaching Talk or disconnecting shortly after initialising. Currently, we don’t have any further updates, but we'll provide another update in 1h or when new information becomes available.
12:16 UTC | 04:16 PT
We were able to get confirmation from our telephony partner that an upstream provider is currently experiencing an incident on their network that is causing the reported impact in Talk Scandinavian numbers. The provider has already implemented urgent changes on their side to minimise the impact. We will provide more information as it is available.
13:32 UTC | 05:32 PT
The change implemented by the upstream carrier has resolved the issue and customers can now receive incoming calls to Scandinavian numbers again. We will now consider this incident as resolved, thank you for your collaboration.
POST-MORTEM
Root Cause Analysis
This incident was caused by local partner carriers after an infrastructure update unexpectedly encountered inbound traffic balancing issues.
Resolution
To fix this issue, local carriers increased capacity to handle the influx of incoming traffic until it had levelled off to a manageable state.
Remediation Items
- Update internal runbooks when working with partner providers.
- Revisit escalation policies for partner provider issues.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.