SUMMARY
On December 15, 2020 from 23:50 UTC to December 16, 2020 00:38 UTC, Zendesk Support customers using Side Conversations experienced an error preventing submission of Side Conversation messages.
TIMELINE
00:49 UTC | 16:49 PT
We’ve received reports of issues in Zendesk Support sending Side Conversations messages on all Pods. We’ve made a change to mitigate the issue and are seeing improvements. We'll monitor this and continue to provide updates once we have them. Thank you.
01:07 UTC | 17:07 PT
We are happy to report that the issues affecting the ability to send messages in Side Conversations has been resolved. Apologies for the inconvenience caused.
POST-MORTEM
Root Cause Analysis
This incident was caused by a software bug in newly-deployed Side Conversations code. A system service didn’t have the required permissions to call a Zendesk API endpoint which resulted in the API calls failing within the send message process. Testing didn’t cover this use case and is a secondary factor influencing the issue.
Resolution
To fix this issue, our team rolled back the culprit code. Monitoring and customer feedback confirmed the issue was resolved shortly after the rollback.
Remediation Items
- Expand risk assessments in Side Conversations deploy process [Completed].
- Implement additional browser tests for Side Conversations [Scheduled].
- Investigate additional alerting [Scheduled].
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.