SUMMARY
On November 28, 2023 from 13:14 UTC to 17:14 UTC, Zendesk Talk customers on multiple pods experienced the Talk/phone icon going missing, with some customers seeing Explore reports showing high numbers of missed calls.
Timeline
17:24 UTC | 09:24 PT
Between 14:30 and 17:14 UTC some Talk customers may have experienced an issue where the Talk product and its corresponding product icon were unavailable. We have rolled back the update responsible and this icon is now once again usable within the product.
POST-MORTEM
Root Cause Analysis
This incident was caused by a software fix that contained an escaped defect that was not visible during the testing process. This fix involved a variable that was not always available during the loading of the voice activity, causing the loading to crash and the call icon to not appear in scenarios where agents had third party apps installed.
Resolution
To fix this issue, we performed a rollback to the previous software version, and had confirmation of recovery.
Remediation Items
- Adding a test scenario to check call icon loads when third party apps are present. [Scheduled]
- Adding monitors and alerts when voice activity fails. [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.