SUMMARY
On August 05, 2022 from 00:31 UTC to 06:48 UTC, Zendesk Support customers using Side Conversations on multiple pods may have been presented with the ‘This conversation can’t be sent right now’ error message, or experienced missing side conversation threads in tickets.
Timeline
07:06 UTC | 00:06 PT
We have been investigating an issue with Side Conversations errors in Support on multiple Pods. Our team is close to resolving this issue and we will provide an update upon full resolution. Thanks for your patience.
07:44 UTC | 00:44 PT
The roll out of the fix for resolving Side Conversations errors in Support is still happening in the background. We will provide another update once this is complete. As always, thank you for your patience and we appreciate your partnership.
07:55 UTC | 00:55 PT
We are glad to inform you that the issue with Side Conversations errors in Support on multiple Pods has been resolved. Many thanks for your patience as we worked this to resolution.
POST-MORTEM
Root Cause Analysis
An essential software update was applied to the operating system of one of our datastore hostgroups, which resulted in the management tool for that datastore failing.An essential software update was applied to our database management service. This caused health checks to fail, marking the database nodes as unavailable to the Side Conversations application.
Resolution
A fix to the health check management tool was created and deployed. The database service was once again available, processing through the backlog that had built up. We continued to monitor until we had confirmation of full recovery.
Remediation Items
- Create monitoring for database service health check failures [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.