SUMMARY
On November 22, 2023 from 10:05 UTC to 12:50 UTC, Support customers across All Pods may have experienced an inability to open manage settings’ pages for macros, views, triggers, and automations in the Admin Center. They encountered blank pages while attempting to access these sections. However, it's important to note that business rules were still running in the background and views could still be accessed as normal in Support.
Timeline
10:50 UTC | 02:50 PT
We have received reports from customer being unable to load pages for macros, views, triggers and automations in the admin center. The business rules and views are still working as expected otherwise and continue running in the background. Investigation is underway and we will provide another update shortly.
11:22 UTC | 03:22 PT
We are continuing to investigate customers being unable to access pages for views, macros, automations and triggers in the admin center. This is affecting all pods. We will provide another update in 30 mins.
11:57 UTC | 03:57 PT
We are still actively investigating the root cause of not being able to load some pages in the admin center across all Pods. We don’t have more information to share at this point but we will keep you informed and will provide another update in 30 mins.
12:29 UTC | 04:29 PT
We would like to provide you with an update regarding the service incident that has been affecting the loading of some pages in the admin center across all Pods. As of now, our team is still actively investigating the root cause of the issue and there have been no new developments since our last communication. We appreciate your patience as we work to resolve this issue. We will continue to keep you informed and will provide another update in 60 mins.
13:06 UTC | 05:06 PT
We have reverted a recent change that was affecting access to admin center leading to blank pages, across all Pods. We kindly request you to clear cache and cookies in order to see the latest version of the platform updated. Thanks for your ongoing patience.
POST-MORTEM
Root Cause Analysis
This incident was caused by certain security settings header changes for Admin Center translation assets, which were supposed to improve the way our system handles certain types of data, but it unintentionally blocked some necessary information. As a result, when users tried to access certain parts of the Admin Center, their browsers couldn't get the information they needed, resulting in blank screens.
Resolution
To solve this issue, we rolled back the changes made in the problematic PR, which successfully addressed the problem. Due to certain limitations imposed by the CDN provider's API at the time, this rollback process took more time than we initially anticipated, however, it was fully sorted once the PR was rolled back.
Remediation Items
- Update the Change Management Process when it comes to assets so it has higher impact priority.
- Ensure the assets are only being used by the rightful applications.
- Resolve the CDN provider API rate limit.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.