SUMMARY
On November 29, 2023 from 21:16 UTC to 22:24 UTC, Zendesk Support customers across multiple pods experienced issues with the Customer Context Panel displaying the “No users found” error, which may have also impacted the ability to add CCs to tickets.
Timeline
22:48 UTC | 14:48 PT
We identified and resolved an issue accessing the Customer Context panel. Customers may have had issues between 21:16 UTC and 22:24. We apologize for the inconvenience this caused.
23:27 UTC | 15:27 PT
It has come to our attention that the issue impacting the customer context panel also impacted adding CC's to tickets. The impacting change has been rolled back and both issues are now fixed.
POST-MORTEM
Root Cause Analysis
This incident was caused by an escaped defect that was missed in the testing process. A new feature was rolled out where a modal would appear upon ticket submission depending on certain conditions being fulfilled. The defect resulted in the modal loading erroneously in some scenarios, affecting the Customer Context Panel and CCs.
Resolution
In order to address the issue at hand, we implemented a rollback to a previous, stable version of the deployed software. Following this course of action, we observed a successful recovery.
Remediation Items
- Adding a test to cover the customer context scenario for this feature [Done]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 comment
Eugene Khoo
Post-mortem published December 7, 2023.
0