SUMMARY
On July 29, 2022 from 00:39 UTC to 06:27 UTC, Support customers using specific Zendesk apps were unable to update tickets without disabling apps, receiving multiple errors preventing ticket submission.
Timeline
06:34 UTC | 23:34 PT
We have identified the root cause of the issue preventing ticket submissions in Support and have rolled back the code in question. We have confirmation that this has been resolved. We thank you for your patience and appreciate your continued partnership.
05:06 UTC | 22:06 PT
We have identified that the issues with ticket submissions are related to some installed apps. If you are unable to submit ticket updates, please continue to use the ?noapps workaround. We appreciate your patience as we troubleshoot this. Next update when we have more to share.
04:10 UTC | 21:10 PT
We are continuing to investigate issues with updating tickets in Zendesk Support. At this point, we are still getting to the root cause. In the meantime, please hard refresh your browser to see if this helps at your end. If this does not help, please append ?noapps to the end of the URL as a workaround.
03:34 UTC | 20:34 PT
We have reports from customers experiencing issues when updating tickets in Zendesk Support. We are investigating this as a matter of priority. More to come.
POST-MORTEM
Root Cause Analysis
This incident was caused by a change which made the below apps unable to load the correct version of a JavaScript library from the content delivery network:
- Approve app
- CloudSET Conditionality app
- Formset app
- Field Conditions
- Several private apps
Resolution
To fix this issue, the change was reverted.
Remediation Items
To prevent this in the future, we’ve instituted a policy change to ensure our Zendesk Apps Framework does not have any dependencies which could conflict with users dependencies as marketplace apps and private apps can include utility libraries at different versions than in our SDKs.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.