SUMMARY
June 12, 2024 03:15 PM UTC | June 12, 2024 07:15 AM PT
From June 12, 2024, 11:55 UTC to 12:21 UTC, a subset of Support users experienced login issues, seeing “Access denied” errors, due to a new security rule implemented by our partner. This measure, intended to enhance account protection, inadvertently blocked some legitimate users. The issue was promptly identified, and the security rule has been adjusted. We can confirm this issue is now resolved.
POST-MORTEM
Root Cause Analysis
This incident was caused by a firewall rule that used the request cookie length to block some extreme traffic. However, this trial-and-error method was not accurate enough and inadvertently blocked legitimate sign-in attempts.
Resolution
To fix this issue, once it was identified, the firewall rule was quickly reverted, stopping the blocking of legitimate sign-in attempts. Monitoring and customer feedback were used to identify and resolve the issue promptly.
Remediation Items
Given the nature of the incident, no preventive remediations could have mitigated this. The change was based on extensive data analysis and a human decision to use the best heuristic available to outsmart the extreme traffic.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.
1 comment
Jessica G.
Post-mortem published June 28, 2024.
0