SUMMARY
13:12 UTC | 06:12 PT
The issue affecting inbound calls to a subset of Spain phone numbers has been resolved, and the service is functioning normally at this time.
16:42 UTC | 09:42 PT
Our telephony provider has confirmed that the issue affecting inbound calls to Brazil numbers is resolved. We are monitoring the issue affecting inbound Spain numbers, but are seeing recovery. We’ll provide another update once we are fully resolved.
15:06 UTC | 08:06 PT
Our provider is seeing improvements with inbound calls to Spanish and Brazilian numbers and we expect to provide another update as soon as more information becomes available.
14:34 UTC | 07:34 PT
Our telephony partner is currently working on an issue impacting incoming calls for Spain and Brazil. We will provide further updates as we receive them.
POST-MORTEM
Our Talk provider experienced 2 separate incidents impacting numbers in Brazil and Spain. Our Talk provider's Phone Number Provider encountered a memory error issue in their Session Border Controller (SBC), due to a sudden increase of concurrent voice calls. This error eventually led to their SBC to reject all new calls resulting in a total outage. After the major issue was addressed, our Talk provider's Phone Number Provider encountered another issue where our Talk provider's allocated capacity reached the limit resulting in partial call rejections. This meant that once the threshold was reached, calls out of that limit started to be rejected. In order to prevent this from happening again in the future, our Talk provider has doubled the memory capacity of their primary SBC and implemented a backup SBC for failover.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.