From October 10, 22:27 UTC until October 11, 04:11 UTC, Support customers with the Tymeshift (3rd-party app) app installed experienced issues with making app secure requests via the Zendesk App Proxy (ZAP) from Tymeshift and causing other Apps to not load.
04:41 UTC | 21:41 PT
We've identified an issue at 22:27 10/10 UTC impacting app functionality to those who have Tymeshift installed in Support. Our team have rolled back a change at 04:11 11/10 UTC. Please perform a hard refresh of your browser to resolve.
Root cause Analysis
This incident was caused by excessive amount of app requests that were introduced by an updated version of the Tymeshift app. Customers with the Tymeshift app installed started experiencing issues with the Tymeshift app as well as other apps they had installed because their app requests were being rate-limited by the Zendesk App Market. In order to resolve the issue, we rolled back the app to its previous version and the issue resolved.
- Revert to the previous version of the Tymeshift app
- Improve rate limiting at the App framework client side level before requests get to the Zendesk App Prox
- Implement better caching for requests going through the Zendesk App Proxy
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.