Service Incident - June 6, 2022 - Guide | Multiple Pods - SSO redirects to blank page

SUMMARY

On June 6, 2022 from 15:54 UTC to 20:18 UTC, Guide customers using SSO experienced blank or white pages when logging into the Help Center.

Timeline

19:12 UTC | 12:12 PT

We are investigating reports of white screens when accessing Guide. We will provide an update as soon as we have more information.

19:27 UTC | 12:27 PT

We have rolled back a change which resolves the issue. Please refresh your browser for the changes to take effect. Sorry for the inconvenience this may have caused you and your customers.

19:51 UTC | 12:51 PT

We have run into some issues rolling back the change on pods 13, 17, 23, 25 and 27. Our engineers are continuing to work on rolling back the change on those pods and we will provide and update as soon as we have more info.

20:24 UTC | 13:24 PT

All pods have now been rolled back successfully and the issue is now fully resolved. Thank you for your patience as we worked to resolve the issue.

POST-MORTEM

Root Cause Analysis

This incident was caused by a process breakdown wherein a part of an update that had not been fully tested was introduced to production.

Resolution

To fix this issue, the update was rolled back.

Remediation Items

1. Improve testing scenarios to include a wider range of use cases. 
2. Automate detection of out-of-sync update assets.
3. Investigate rollback delays to avoid similar issues and speed up resolution in the future.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.