SUMMARY
On May 26, 2023 from 08:40 UTC to 12:33 UTC, some Guide Admin users across all Pods (but mainly on Pods 17, 18 and 29) experienced 5xx errors when trying to access their Help Centre and/or the theme preview customization template.
Timeline
12:13 UTC | 05:13 PT
We are aware of issues with Guide affecting multiple Pods where customers are getting 500 error messages when accessing the product or Theme Preview features. Investigation is underway. Next update in 1h or when we have more info.
12:44 UTC | 05:44 PT
We have rolled a fix for the issue affecting some Guide customers’ access and Admin Theme editing page availability across multiple Pods. Please refresh your browser and if you continue having issues, make sure to clear cache and cookies so the fix is applied to the account.
14:09 UTC | 07:09 PT
We are happy to inform that we have received confirmation from customers that they no longer have issues accessing Guide and its features across multiple Pods. Thank you for your patience and cooperation while we worked on this.
POST-MORTEM
Root Cause Analysis
An upgrade to one of our cache storage mechanism software clients caused a bug that lead to the Help Center theme previewing being inaccessible and some customers' Admins experiencing 5xx errors when accessing Guide.
Resolution
To fix this issue, the change that caused the errors was reverted.
Remediation Items
- Fix the bug. [Completed]
- Implement monitoring for theme preview. [Scheduled]
- Update runbook to include mechanisms for the scenario. [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 Comments
Post-mortem edited 15 June, 2023.
Article is closed for comments.