SUMMARY
From March 11, 2021 at 23:27 UTC to March 12, 2021 at 01:28 UTC, anonymous users submitting tickets via Zendesk Help Center encountered 500 Internal Server errors.
Timeline
02:31 UTC | 18:31 PT
We are happy to report that we have resolved the 500 internal server errors impacting ticket forms in Guide. Thank you for your patience.
01:51 UTC | 17:51 PT
Our team has rolled back a change we believe to be the root cause of the issue and are seeing recovery. Please refresh your browser if you’re still experiencing issues
01:20 UTC | 17:20 PT
We are investigating 500 internal errors when trying to access Guide's submit a request page that has ticket forms enabled. More information to follow.
Root Cause Analysis
This incident was caused by a deployed change to production that inadvertently impacted requests using anonymous system user authentication.
Resolution
The deployed change was reverted and the correct production version of our authentication mechanism was restored.
Remediation Items
- Investigate new methods of deploying high risk changes,
- Improve handling of authentication gems during deploys,
- Created additional alerts and monitors to detect issues faster in the future.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.