Summary
On February 13, 2024 from 15:01 UTC to 16:39 UTC, Zendesk Support customers using Messaging (Agent Workspace) may have experienced errors and other issues when sending and receiving messages.
Timeline
15:32 UTC | 07:32 PT
We are currently aware of reports about customers experiencing issue with sending messages in messaging or receiving unresponsive error messages across all Pods. Investigation is underway and we will provide another update shortly.
15:48 UTC | 07:48 PT
We have rolled back a recent deployment and are seeing improvements for the unresponsive error and issue with sending messages. Please refresh your browser and try again. We will keep working on this until full resolution and will provide another update when we have more information available.
18:26 UTC | 10:26 PT
Following the rollback, we are no longer seeing unresponsive errors and issues sending messages, and this incident is now resolved. Please ensure you have refreshed your browser to see the current version. Thank you for your patience during our investigation.
POST-MORTEM
Root Cause Analysis
This incident was caused by a bug introduced by a recent code change in the Messaging composer. This issue prevented agents from sending messages after the status of a conversation or ticket was changed.
Resolution
To fix this issue, our team identified and reverted the offending change.
Remediation Items
- Add and improve to testing mechanisms for the scenario encountered in this incident [In progress]
- Review and implement improvements to composer logic [In progress]
- Review deploy build process [Scheduled]
- Review monitors for paging internal teams [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 comment
Dan Beirouty
Postmortem published February 21, 2024.
0