On March 28, 2022 from 18:09 UTC to March 30, 2022 at 09:59 UTC, some Zendesk Support customers on multiple Pods were unable to submit tickets due to an obscured Ticket Submit button.
18:50 UTC | 11:50 PT
With a recent change, any Zendesk APIs blocked due to incorrect firewall configuration became noticeable. To resolve this issue, please review your firewall configuration to ensure it matches our documentation: https://zdsk.co/3NDojB4
13:10 UTC | 06:10 PT
The workaround of correctly “Configuring your firewall for use with Zendesk” from the article shared previously has been proven effective in order to help stop the issue with updating Ticket Status. Please follow this as we continue investigating more details. Thank you!
09:30 UTC | 02:30 PT
We are investigating Support customer reports’ of inability to submit changes to Ticket Status. A possible workaround is to follow this article’s steps https://support.zendesk.com/hc/en-us/articles/4408842860186 to avoid blocking any Zendesk APIs. We appreciate your patience.
Root Cause Analysis
On 2022-03-28 19:09:47 UTC a change was deployed to update Z BOT widget initialization. The change updated sunshine conversation Widget to call /login on page load, in product and outside of product (auth vs un-authed). Previously /login was called after user interaction in the widget. Updating to login user at page load is a dependency for notify users of unread messages and improve customer experience interacting with the widget.
To fix this issue, the configuration change was rolled back.
- Update Zendesk firewall guidelines
- Investigate failover state
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.