SUMMARY
On January 8, 2021 from 09:05 UTC to 12:35 UTC, some Zendesk Talk customers using phone numbers prefixed with the +4433 area code experienced dropped inbound calls.
11:09 UTC | 03:09 PT
We are receiving reports of some inbound calls to UK numbers not being connected . Investigation is underway.
11:37 UTC | 03:37 PT
We’re continuing to investigate the reports of some inbound calls to UK numbers not being connected, we will share an update once we have additional information.
13:16 UTC | 05:16 PT
We are working with our carrier partner to resolve the issue. We expect to receive another update in 2 hours and we’ll share more information as it becomes available.
14:59 UTC | 06:59 PT
We are continuing to work with our carrier partner to resolve the issue of inbound calls failures to some UK numbers. We expect to receive another update in 4 hours and we’ll share more information as it becomes available.
19:17 UTC | 11:17 PT
We are monitoring the issue with inbound call failures to some UK numbers. We have seen some improvements, and we are continuing to work with our carrier partner. We will post an update when the issue is resolved.
10:32 UTC | 02:32 PT
We have started seeing recovery for inbound calls to our Talk Partner UK Phone Numbers for the last hour after their carrier have implemented some changes. We will continue to monitor to ensure a full recovery. We shall provide more information as it becomes available.
Root Cause Analysis
This incident was caused by a faulty phone number decommissioning job in a downstream provider’s infrastructure. Due to a misconfigured number removal application, the task corrupted a database that stored active UK phone numbers.
Resolution
To fix this issue, the downstream provider restored the affected database table to its previously working state. Service restoration was observed shortly after.
Remediation Items
Service provider
- Review and improve testing process for provisioning tasks [Completed]
Zendesk
- Investigate adding new metrics to view call volume by phone number range [Scheduled]
- Investigate support process improvements to reduce impact time [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.