SUMMARY
On July 14, 2022 this service incident was fired to investigate Support customer reports across all Pods that experience an "Error Loading Message" when clicking on image attachments. This happened from June 29, 12:52 UTC to 23:40 UTC.
Timeline
14:03 UTC | 07:03 PT
We’re currently investigating an issue for multiple Support customers and Pods regarding “Error loading image” while trying to open tickets’ attachments. The workaround is to right-click and open the image in a new tab. We will provide more info in 2h or when we have more details.
16:26 UTC | 09:26 PT
Our team continues to investigate an issue causing an "Error loading image" message when trying to open ticket attachments. Additional information will be provided when we have a substantive update to share.
20:17 UTC | 13:17 PT
Our engineering team will be working on a long term solution for this issue. In the interim, please reach out to Zendesk support via messaging at support.zendesk.com so that our support team can provide you with potential workarounds.
00:22 UTC | 17:22 PT [July 15]
Our team has fixed the issue preventing ticket attachments from loading in Support. Please hard refresh your browser before you attempt to load attachments again. We apologize for the inconvenience caused by this issue.
POST-MORTEM
Root Cause Analysis
This incident was caused by a new code for displaying image thumbnails which were introduced as part of the Malware Scanning Service project. This code changed how images are fetched from the server and, additionally, it contained different request headers which were necessary to fetch images that had been tagged as malware which would normally be denied access.
This new method, and some combination of account settings (having a host mapped domain, enabling secure attachments, we never figured out exactly which), caused requests for images from these accounts to fail and display an error.
Resolution
To fix this issue we returned to using the old method for attachment retrieval, but first had to ensure that malware scanning service requirements could be met by the backend only.
Remediation Items
- Re-implement loading and error states for attachment thumbnails
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.