SUMMARY
Between May 23, 2022 19:25 UTC to May 24, 2022 20:52 UTC, customers using Zendesk Support experienced a “Something went wrong” error message while accessing sandboxes from the Admin Center.
Timeline
20:26 UTC | 13:26 PT
We are investigating reports of issues accessing sandboxes from the Support admin center. We will provide an update as soon as we have more information.
20:41 UTC | 13:41 PT
We have confirmed an issue when accessing the Admin Center sandbox page resulting in a “Something went wrong” message and 403 error. Our engineers are now working to identify and resolve the issue. Next update in 30 minutes or soon as we have more information.
20:55 UTC | 13:55 PT
Our engineers have rolled out a fix and we can confirm the issue is now resolved. Please refresh your browser for the changes to take effect.
POST-MORTEM
Root Cause Analysis
This incident was caused by a software deploy that contained an escaped defect, where accessing the sandbox with VPN turned off resulted in an error.
Resolution
To fix this issue, we rolled back the deploy, and confirmed that this resolved the issue.
Remediation Items
- Update documentation to add a new step to ensure that the functionality works with and without VPN
- Add additional alerts to spot errors on sandbox endpoints outside of VPN
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.