On July 2, 2019, from 13:43 UTC / 06:43 AM PT to 14:08 UTC / 07:08 AM PT, Our customers from all Pods did experience access issues to our platform in the form of 502 errors. Our Status page also did go offline during this service incident.
17:16 UTC | 10:16 PT
We’re pleased to report that status.zendesk.com is now up and running and functioning as expected.
16:26 UTC | 09:26 PT
We are continuing to work on a resolution of the status.zendesk.com page load issue. All Zendesk production services are still working as expected otherwise. We will post a final update when the status page is loading.
15:22 UTC | 08:22 PT
Our third party CDN provider has resolved the issue and services have resumed normal operation. We are still working to resolve issues with the status.zendesk.com page not loading.
14:58 UTC | 07:58 PT
We experienced an outage across all pods impacting, Support, Guide, Talk. Chat, and Explore from 13:43 - 14:08 UTC. Our third party CDN provider has resolved the issue and services have resumed normal operation.
Root Cause Analysis
It has been determined that a Web Application Firewall deploy on our DDoS and abuse protection system's end had resulted in CPU exhaustion in the impacted data centers.
The outage was remediated on our abuse protection system's end by temporarily disabling part of the Web Application Firewall functionality. A fix was then tested and implemented before re-enabling the temporarily disabled functions.
On our end, we are reviewing processes to facilitate quick and effective roll back from our DDoS and abuse protection system if they encounter issues.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.