SUMMARY
On July 01, 2024 from 21:00 UTC to July 02, 2024 02:01 UTC, Zendesk Guide customers experienced errors when loading articles with attachments.
Timeline
July 02, 2024 12:40 AM UTC | July 01, 2024 05:40 PM PT
We are aware of a backend change that may cause customers to experience 500 errors when accessing Guide articles. Investigations are currently ongoing. We will provide more information in the next 15 minutes.
July 02, 2024 12:56 AM UTC | July 01, 2024 05:56 PM PT
We have performed a rollback of some changes as part of troubleshooting. Errors are still present in our logs and we are continuing to investigate. Next update in 30 minutes or as soon as more information becomes available.
July 02, 2024 01:35 AM UTC | July 01, 2024 06:35 PM PT
A change is currently being rolled out. From our initial monitoring, error rates are showing positive signs. The rollout will take some time to affect all pods and we will update when the deployment is complete. Thank you for your patience in the meantime.
July 02, 2024 02:52 AM UTC | July 01, 2024 07:52 PM PT
The software change deployment has completed on all pods and we are noting errors dropping off across the board. We have also tested previously impacted articles loading successfully now, as well as getting confirmation of recovery. Many thanks for your patience as we got to this point.
POST-MORTEM
Root Cause Analysis
This incident was caused by a backend upgrade that resulted in errors when referencing attachments, causing articles with attachments to fail to load correctly for some customers.
Resolution
To fix this issue, a software deployment to address the referencing issues was done. We had confirmation of recovery thereafter.
Remediation Items
- Review our testing process to investigate why this scenario was not identified in preliminary testing. [Scheduled]
- Improve our rollback procedure to ensure a more efficient response for similar future situations. [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.
1 comment
Eugene Khoo
Post-mortem published July 15, 2024.
0