SUMMARY
On August 9, 2021 from 10:10 UTC to 13:42 UTC, customers from Pods 13, 19, 20, and 23 faced loading issues for theming and admin settings in Zendesk Guide, with some of them seeing a blank screen with 502 Bad Gateway error messages or being unable to access the product.
Timeline
12:45 UTC | 05:45 PT
After receiving reports, we have identified the cause of loading issues in Zendesk Guide when accessing the theming. More information to follow.
13:35 UTC | 06:35 PT
We are happy to report that all known issues related to loading problems for theming and admin settings in Zendesk Guide have been resolved. Thank you for your patience.
POST-MORTEM
Root Cause Analysis
This incident was caused by an upstream (internal server routing configuration) that wasn’t yet deployed into all production Pods. When the first service, in this case, Guide, migrated an existing upstream to a new one, it broke where the version had not been deployed yet.
Resolution
To fix this issue, the team deployed the correct version to all pods making everything consistent.
Remediation Items
- Fixed internal process that prevented a quicker recovery from the bad configuration.
- Created additional documentation that explains further the configuration differences between versions.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.