SUMMARY
On June 9th, 2021 from 06:53 UTC to 12:30 UTC, customers experienced inbound call failures to a subset of phone numbers in Sweden, Denmark, Finland, South Korea, Kenya, and Spain with around 24% calls impacted for said countries.
14:43 UTC | 07:43 PT
We’re happy to report that all known issues related to inbound call failures affecting a subset of Talk phone numbers in multiple locations have been resolved. Thanks for your patience throughout this incident.
12:55 UTC | 05:55 PT
We have started seeing recovery for inbound calls to Talk phone numbers in Sweden, Denmark, Finland, South Korea, Kenya and Spain. We will continue to monitor to ensure a full recovery. Please let us know if you are still having any issues.
11:28 UTC | 04:28 PT
We’re expanding the scope of potential impact regarding inbound call failures for Talk, based on customer reports and information from our partner about an upstream carrier failure. We appreciate your patience and will share updates when available.
10:25 UTC | 03:25 PT
We have confirmed reports of a service disruption affecting Talk numbers within the Nordic countries. We’re working with our carrier provider on solutions to fix it. More info to come.
09:57 UTC | 02:57 PT
Our teams are continuing to investigate an issue where some EU numbers are unable to receive inbound calls, as we received reports of other numbers than the ones stated previously. We will provide further information and scope shortly.
09:46 UTC | 02:46 PT
We are currently investigating reports of customers not being able to receive calls for Talk numbers with the +45 and +46 area codes. We will post more information once we learn more.
POST-MORTEM
Root Cause Analysis
This incident was caused by an unwanted error by our telephony partner’s service provider. A network maintenance was being performed which included configuration changes to their IP network. Due to human error, a router configuration was updated with an incorrect IP address range which then resulted in a block of internet connectivity for a specific voice server (Session Border Controller). The provider considered the maintenance as a routine task with no expected downtime and hence did not inform our partner.
Resolution
To fix this issue, the telephony partner’s provider re-instated the original IP network configuration during the maintenance window which restored internet connectivity and allowed calls to connect to their voice server (Session Border Controller).
Remediation Items
Our partner shared the following remediation plans with us:
- They have committed to implement improved monitoring
- They will work with their provider to gain better understanding of maintenance windows.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.