On June 6, 2022 from 15:54 UTC to 20:18 UTC, Guide customers using SSO experienced blank or white pages when logging into the Help Center.
19:12 UTC | 12:12 PT
We are investigating reports of white screens when accessing Guide. We will provide an update as soon as we have more information.
19:27 UTC | 12:27 PT
We have rolled back a change which resolves the issue. Please refresh your browser for the changes to take effect. Sorry for the inconvenience this may have caused you and your customers.
19:51 UTC | 12:51 PT
We have run into some issues rolling back the change on pods 13, 17, 23, 25 and 27. Our engineers are continuing to work on rolling back the change on those pods and we will provide and update as soon as we have more info.
20:24 UTC | 13:24 PT
All pods have now been rolled back successfully and the issue is now fully resolved. Thank you for your patience as we worked to resolve the issue.
Root Cause Analysis
This incident was caused by a process breakdown wherein a part of an update that had not been fully tested was introduced to production.
To fix this issue, the update was rolled back.
- Improve testing scenarios to include a wider range of use cases.
- Automate detection of out-of-sync update assets.
- Investigate rollback delays to avoid similar issues and speed up resolution in the future.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.