Summary
On October 31, 2024 from 10:32 UTC to 14:07 UTC, Zendesk Talk customers on multiple Pods experienced an issue where voice call recordings were not displayed on call tickets.
Timeline
October 31, 2024 12:46 PM UTC | October 31, 2024 05:46 AM PT
We are investigating reports where some Talk recordings aren't showing up in tickets. We will provide more information as it is available.
October 31, 2024 01:19 PM UTC | October 31, 2024 06:19 AM PT
Our teams are actively working to resolve the issue affecting Talk recordings appearing in tickets. We will provide further updates as soon as possible.
October 31, 2024 02:15 PM UTC | October 31, 2024 07:15 AM PT
We’re pleased to inform you that the issue causing Talk recordings not to appear in tickets has now been resolved. We apologise for any inconvenience this may have caused.
Root Cause Analysis
This incident was caused by a network proxy configuration error that resulted in some cross-Pod relay requests being relayed incorrectly. The largest impact was to call recording requests which were unable to be relayed resulting in the errors observed by some of our Talk customers.
Resolution
To fix this issue, our team identified the issue in a recent API deployment and rolled it back to a previously-working version.
Remediation Items
- Additional monitoring and dashboards for the impacted network proxy service [Completed]
- Implement additional directives to better handle similar issues in future [In Progress]
- Add smoke tests to tracking Pod relays [Scheduled]
- Investigate additional SLO to track increase in error rates [In Progress]
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.
1 comment
Dan Beirouty
Post-mortem published November 18, 2024.
0