Service Incident - November 19th, 2020 - Chat - Attachment deletion errors

SUMMARY

From November 16, 2020 09:30 UTC until November 19, 2020 09:25 UTC, Zendesk Chat customers experienced errors when attempting to delete attachments via the history and conversation screens.

TIMELINE

15:24 UTC | 07:24 PT
We were aware of an issue that resulted in some customers receiving an error when attempting to delete ticket attachments in chat. We are happy to report that this should now be resolved, please reach out if you continue to experience an error.

POST-MORTEM

Root Cause Analysis

This incident was caused by the Zendesk Chat attachment repository being configured to an unexpected state. When attachment deletions are attempted, the Chat application requires two success responses: one from the repository and one from our cache. In this incident, the attachments were deleted successfully from the repository, however the cache received an error due to the unexpected state of the repository. This is the error that was passed on to users when attempting attachment deletions.

Resolution

To fix this issue, our engineers implemented a fix to delete the cached files. This action completely resolved the issue.

Remediation Items

1. Implement mechanism to delete cache when versioned attachment file is deleted [Completed].
2. Audit all other file deletion mechanisms in Chat to ensure they correctly handle [Scheduled].

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.