SUMMARY
On May 4, 2021 from 07:00 UTC to 19:27 UTC, a group of our customers using Zendesk Talk experienced issues receiving calls to their UK phone numbers.
Timeline
19:36 UTC | 12:36 PT
We have just received confirmation that the issue impacting UK phone numbers should now be resolved. Please let us know if calls to these numbers are working as you'd expect or if you're still seeing this issue so that we can take a closer look into your case.
18:38 UTC | 11:38 PT
Our talk partner is seeing recovery for this UK phone number issue. We will continue to monitor and update accordingly. Please let us know if you're seeing improvement on your side as well.
17:36 UTC | 10:36 PT
Our Talk provider is continuing to work with carriers to resolve this issue. We will let you know as soon as a fix is released.
16:38 UTC | 09:38 PT
Our Talk provider has identified the root-cause of this issue and is working with carriers to resolve it as quickly as possible. We will keep you posted as the fix is rolled out.
15:28 UTC | 08:28 PT
We are receiving reports of inbound call failures to a subset of UK phone numbers. Our Talk partner has already identified the cause of the issue and they’re working toward resolution. We will provide more information as it is available.
POST-MORTEM
Root Cause Analysis
This incident was caused by a software issue of one of our telephony partners phone number providers.
This led to a transcoding error with calls, which caused them not to connect properly.
Resolution
To fix this issue, the provider rerouted calls through a backup service which then resolved the issue for customers.
Remediation Items
The provider added improved monitoring for errors of this nature.
The provider will also be upgrading the system where this error occurred, which should remove the potential for recurrence.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.