SUMMARY

Between December 18, 2023 19:50 UTC and December 19, 2023 05:56 UTC, some customers experienced issues where public or internal comments were intermittently not being saved to tickets. In the case of public comments not saving, it resulted in customers being unable to respond to tickets when the issue presented itself.  Ticket statuses and other properties did update.

Timeline

08:07 UTC | 00:07 PT
We have confirmation that the issue has been resolved following the software rollback. Your patience throughout this process is greatly appreciated. Thank you for your understanding.

06:34 UTC | 22:34 PT
We have determined that this issue would have impacted non-Agent Workspace customers on all pods. After the version rollback, this is no longer replicable in our environment. Please perform a hard refresh of your browser and let us know if this has fixed things at your end.

06:02 UTC | 22:02 PT
We have identified a change that was deployed earlier today and have rolled it back. We are currently monitoring for improvements in symptoms and will provide another update in 30 minutes.

05:25 UTC | 21:25 PT
We are investigating reports of public or internal comments intermittently not being saved to tickets. We are aware of this currently impacting Pods 13, 15, 23, and 25, and are currently investigating further. More information in 30 minutes.


POST-MORTEM

Root Cause Analysis

The incident resulted from a code change that inadvertently introduced a race condition. This race condition prevented ticket comments from being updated whenever an agent had multiple tabs open.

Resolution

Once the issue was identified, the code change was rolled back and performance restored to normal.

Remediation Items

  1. Improve testing to catch this type of race condition
  2. Setup additional alerts to catch this type of race condition
  3. Improve deploy rollback timeliness


FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

Powered by Zendesk