SUMMARY
On November 7, 2023 from 09:56 UTC to 10:07 UTC, Zendesk Support customers on Pods 15, 18, 23, and 26 experienced errors and blank screens when logging into the product. Customers who were logged in did not experience any impact but anyone reloading the application would be presented with an error message.
Timeline
19:41 UTC | 11:41 PT
Between 9:55 and 10:05 UTC on November 7, 2023, Support customers on Pods 15, 18, 23, and 26 experienced access issues and were unable to load the product. Access has been restored and we are working diligently to complete our root cause investigation.
POST-MORTEM
Root Cause Analysis
The Support user interface relies on Javascript assets that correspond to the UI version deployed, published to our CDN, where the content is then delivered to our customers’ browsers.
This incident was caused by a recent upgrade to the library used to build these javascript assets, where a scenario caused the assets not to be loaded successfully, resulting in errors being experienced by customers.
Resolution
A fix to the deployment process was put in place, the escaped defect was then resolved and redeployed. Recovery was observed thereafter.
Remediation Items
- Prevent asset filename issues from affecting customers when deployments run. [Done]
- Fixing the root cause where the assets don’t correspond to the Support UI version deployed. [Done]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.