Summary
On November 15, 2023 from 09:43 UTC to 10:44 UTC, some Chat agents may have experienced issues with chats not being reassigned in a timely manner and, in some very limited cases, those chats may have been marked as missed chats.
Timeline
10:46 UTC | 02:46 PT
We are currently investigating an issue that is causing reassignments not to work for Chat and Messaging for some customers across all Pods. We will provide an update on this in 30 mins.
10:55 UTC | 02:55 PT
We have implemented a fix for the issue causing reassignments to fail for customers across all pods and the systems are now working as normal again. We will now consider this incident as resolved.
POST-MORTEM
Root Cause Analysis
This incident was caused by an error in an infrastructure configuration change. Resolution time was delayed due to engineers encountering permissions issues while attempting to roll out the fix.
Resolution
To fix this issue, our engineers deployed a configuration correction to our production environment.
Remediation Items
- Retune existing monitors and alerts for on-call engineers [Scheduled]
- Implement additional approval process to better synchronise staging and production changes [Scheduled]
- Investigate better permissions access for Chat teams [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 comment
Dan Beirouty
Post-mortem published November 22, 2023.
0