SUMMARY
From October 22, 2020 at 23:35 UTC until October 23, 2020 at 16:22 UTC, Zendesk Support customers on all Pods may have experienced an issue saving account settings (account admins only).
TIMELINE
15:42 UTC | 08:42 PT
We are currently investigating reports of Support admin settings changes failing to save successfully.
15:54 UTC | 08:54 PT
We have identified the cause of the issues saving Support admin settings and are rolling out a fix. We will provide another update when that rollout is complete. Thank you for your patience!
16:30 UTC | 09:30 PT
We're happy to report that the rollout of the fix is complete. The issues saving Support admin settings are now resolved. Apologies for any inconvenience caused.
POST-MORTEM
Root Cause Analysis
This incident was caused by a bug in a recent Support deploy that introduced strict API validation to Zendesk Support APIs as part of an application framework upgrade. The strict API validation code erroneously stripped admin settings parameters from the settings update requests preventing them from being saved.
Resolution
To fix this issue, our Engineering team rolled back Zendesk Support to the last known working version.
Remediation Items
- Enable simulation mode for additional allowed parameters actions [Scheduled]
- Improve integration test coverage for Support account settings changes. Implement manual tests for changes not covered by integration tests [Scheduled]
- Provide better logging visibility for all Support engineering teams [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.