SUMMARY
On June 6, from 13:29 UTC to 13:37 UTC, Support, Guide, Talk and Sunshine Conversations customers on Pod 23 experienced 5XX server error green screens while processing their request to access the mentioned Zendesk services.
Timeline
15:02 UTC | 08:02 PT
From approximately 13:29 to 13:36 UTC today, customers on Pod 23 may have encountered a green screen “Server Error” message affecting usage across multiple Zendesk Products. This was caused by an increased network congestion and the errors have already subsided.
POST-MORTEM
Root Cause Analysis
This incident was caused by our system experiencing a large-scale attack which led to an outstanding volume of traffic on our DNS provider.
Resolution
Our engineering team was alerted to this incident and although the initial errors went down by themselves, they quickly implemented additional protection rules matching the attack pattern should a recurrence occur.
Remediation Items
- Add additional protection rules to cover this attack pattern. [DONE]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 Comments
Post-mortem published June 16, 2023.
Article is closed for comments.