On May 30th, 2019 from 13:24 UTC to 15:28 UTC, Zendesk Support and Chat customers on Pod 19 experienced issues with Zendesk apps not loading. Support and Chat admins were unable to load the app manage page during the incident. For agents, apps did not load in Chat or Support (no ticket sidebar or app top bar). A small percentage of agents might have seen apps load but break, either by a iframe 404 in the ticket sidebar or an app itself displaying error messages as it failed to communicate to the App Market backend.
14:10 UTC | 07:10 PT
We are currently investigating Zendesk Apps not loading for some customers in POD19
14:45 UTC | 07:45 PT
We are continuing to investigate Zendesk applications not loading for customers on POD19, more to follow
15:03 UTC | 08:03 PT
We continue to investigate the issue regarding Zendesk Apps not loading for customers on POD19. Next update in one hour.
15:52 UTC | 08:52 PT
We've confirmed that the issues affecting apps not loading on some Pod 19 accounts is now resolved. Affected accounts may need to perform a hard refresh on their browsers to see this working normally.
Root cause Analysis
A code deploy and configuration change updated the URLs app assets were served from. As load increased on Pod 19, increased asset traffic for public apps began to exhaust App work servers, causing the App manage page and Apps to not load.
To fix this issue, the code and configuration change were rolled back and Apps began to load normally again.
- Roll back code and configuration change
- Improve monitoring, alerting, and notification for these types of changes
- Improve documentation around the configuration change to expedite resolution
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.