Service Incident - January 11, 2023 - Talk in Support | All Pods - Call and Voicemail recordings unavailable in tickets

Summary

From January 10, 2023 at 14:41 UTC until January 11, 2023 at 18:58 UTC, some Talk customers across all Pods were unable to retrieve call and voicemail recordings in Support tickets.

Timeline

16:03 UTC | 08:03 PT
We are aware of Talk customers across all Pods facing issues with calls and voicemail recording availability in the ticket interface in Support. Our team is already working on a fix and we appreciate your patience while we get this done as soon as we can.

19:35 UTC | 11:35 PT
We are happy to report that the issue affecting calls and voicemail recording availability in the ticket interface in Support has been resolved. Thank you for your patience during our investigation.

Root Cause Analysis

This incident was caused by a planned backfill that was intended to remove voice recordings on deleted Zendesk accounts in order to fulfill voice recording compliance obligations. Due to a scoping issue in the backfill code, some voice recordings for a small number of active accounts were also removed.

Resolution

To fix this issue, our team implemented a solution in our recording service that recovered the recordings that were inadvertently removed during the incident.

Remediation Items

1. Additional monitors and alerts for fetch recording failures.
2. Review and improve recording delete and recovery mechanisms.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.