SUMMARY
On May 05, 2022 from 13:50 UTC to 15:45 UTC, a subset of customer agents using Agent Workspace in Zendesk Support may have been unable to accept incoming chats, or respond to open chats.
Timeline
15:52 UTC | 08:52 PT
We are investigating reports of a subset of agents using Chat on Agent Workspace being unable to accept incoming chats, or being unable to respond to open chats. We will provide additional information as soon as we can.
16:35 UTC | 09:35 PT
We are happy to report that the issue causing a subset of agents using Chat on Agent Workspace to be unable to accept incoming chats or respond to ongoing chats has been resolved. Thank you for your patience.
16:03 UTC | 09:03 PT
We are seeing improvement in the behavior causing a subset of agents using Chat on Agent Workspace to be unable to accept new chats or respond to ongoing chats. Please let us know if you continue to experience any issues.
POST-MORTEM
Root Cause Analysis
This incident was caused by a backend configuration and data infrastructure versioning issue.
Resolution
To fix this issue, we first upgraded the data infrastructure version, then did a redeploy of the chat mediator. Errors in our logs subsided thereafter, indicating that recovery was observed.
Remediation Items
- Improve alerts for better resource monitoring [Done]
- Point the monitoring dashboard to the correct data store instance [Done]
- Automate validation processes for software versions [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.