On October 14, 2019 from 9:23 UTC to 12:03 UTC a subset of agents on Zendesk Talk on Pods 13, 14, 15, 17, 19 and 23 would intermittently receive a call they were unable to answer.
12:35 UTC | 05:35 PT
We have implemented a fix and are currently monitoring the situation with inbound calls. Please do let us know if you are still seeing any issues. Thank you for bearing with us and for your understanding.
11:30 UTC | 04:30 PT
We are continuing to investigate issues with incoming calls affecting some users. More to come
11:03 UTC | 04:03 PT
We are investigating an issue where incoming calls are not reaching Zendesk Support accounts. More to come shortly.
Root Cause Analysis
This incident was caused by a deploy to the agent collision service which included an update to engine.io. Part of the engine.io update included a change in the way payloads are sent, which inadvertently caused Talk agents with special characters in their names (accents, symbols, etc.) or calls coming from countries with special characters, to ring, but fail to connect.
To fix this issue, we rolled back the deployed change to the agent collision service.
- Implement additional Talk tests to include special characters on user names
- Update agent collision service browser tests to cover more use cases
- Investigate potential improvements to better detect these issues
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.