SUMMARY
On June 29, 2022, between 14:20 UTC and 14:32 UTC, Zendesk customers experienced access denied issues to multiple Zendesk products across all Pods.
15:10 UTC | 08:10 PT
The issue impacting access to Zendesk products is now fully resolved. Please let us know if you continue to experience issues.
14:56 UTC | 07:56 PT
We are seeing improvements for the issue with access to Zendesk products. Please refresh your page. We will continue to provide updates.
14:40 UTC | 07:40 PT
Our teams are investigating access issues to Zendesk products. We will provide more information shortly
POST-MORTEM
Root Cause Analysis
The root cause of the issue is a Web Application Firewall (WAF) rule erroneously configured in blocking mode on our CDN. Our configuration was to set this to Dry Run mode. This resulted in 4.4 million blocked requests. Rolling back the changes restored our services. We are working with our CDN provider to understand how this happened.
Resolution
At 14:32 UTC, We rolled back changes and restored the service. At 14:34 UTC, we confirmed our services have recovered from our monitoring systems and also received confirmation from our customers. At 15:12 UTC, all clear was called for this incident.
Remediation Items
- Follow up with CDN vendor to understand the root cause of why the WAF rule was enabled in blocking mode by default
- Zendesk to figure out how we can adequately test the WAF rule in our staging environment
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.