SUMMARY
On October 21, 2021, between 18:58 and 19:51 UTC, customers using Sunshine Conversations were unable to send or receive messages.
Timeline
19:18 UTC | 12:18 PT
We are investigating reports of issues sending and receiving messages in Sunshine Conversations. We will follow up with additional information as soon as it's available.
19:43 UTC | 12:43 PT
Our team continues to investigate issues sending and receiving messages in Sunshine Conversations. We will provide another update as soon as possible.
20:07 UTC | 13:07 PT
Sunshine Conversations messages are now being sent and received as of 19:51 UTC. Thank you for your patience as we worked to resolve the issue. Please let us know if you experience any other issues.
POST-MORTEM
Root Cause Analysis
This incident was caused by a system failure within our message queueing system.
Resolution
To fix this issue, the queueing system was restarted.
Remediation Items
- Reduce connection limit to free up resources and allow the message queueing system to recover.
- Restart overwhelmed nodes in message queueing system.
- Research acceptable fallback measures, to ensure message delivery, and implement those measures.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.