SUMMARY
On June 21, 2022 from 15:00 UTC to 15:53 UTC, non-Agent Workspace Chat customers may have experienced new chats initially not creating tickets in Zendesk Support. These tickets were eventually created in Zendesk after some delay. Chat history on the Real Time Monitoring dashboard may have also been delayed.
Timeline
15:46 UTC | 08:46 PT
We are investigating reports of incoming chats not creating tickets. We will provide additional information as soon as we can.
16:02 UTC | 09:02 PT
We are seeing some recovery as affected services restart, but we are continuing to monitor the situation. We will provide another update within the next 30 minutes.
16:39 UTC | 09:39 PT
We are happy to report that the issue affecting Chat ticket creation has been resolved. Thank you for your patience during our investigation.
POST-MORTEM
Root Cause Analysis
This incident was caused by our backend data store hitting the number of connection limits, causing degradation in the service that relates to ticket integration and creation.
Resolution
To fix this issue, we performed a failover to a separate data node, and also restarted the degraded services to enable them to connect to the new node.
Remediation Items
- Creation of alert when connection limits are near maximum, allowing a bigger buffer of time to address similar scenarios. [Scheduled]
- Create a monitor to alert on ticket integration queue size. [Scheduled]
- Evaluate and implement network monitoring for all chat instances [ Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.