SUMMARY
Between June 22, 2022 from 13:10 UTC to June 24, 2022 18:40 UTC, Zendesk customers may have experienced freezing, lagging and general slowness issues while using Zendesk Support and Chat products.
Timeline
09:34 UTC | 02:34 PT
We’re currently working on resolving the problem behind performance issues across Support & Chat reported by multiple Pod customers. We have rolled back some changes yesterday hoping would mitigate the issue, but seems to have only partially worked. [1/2]
09:35 UTC | 02:35 PT
We truly appreciate your patience while we continue the investigation on the freezing, lagging and general slowness issues in the platform and will update you with further details at 12:30 UTC or when we have them. [2/2]
12:30 UTC | 05:30 PT
We continue working with the relevant teams on gathering details in order to fix the performance issues in the platform across multiple pods. At this point we’re still investigating and we shall update you again at 15:30 UTC or when we have more relevant information to share.
15:15 UTC | 08:15 PT
We thank you for your reports and ongoing patience regarding the performance issues happening across the Zendesk products, especially Support and Agent Workspace. We are continuing our investigation and hope to resolve this issue soon. Again, thank you for your patience and we will update you when we have more information.
17:55 UTC | 10:55 PT
We have rolled out an update which we believe will help to address the performance issues within Support and Agent Workspace. Please let us know if you continue to experience any issues.
23:30 UTC | 16:30 PT
We are continuing to monitor the performance issues impacting Support. We are observing recovery, but please reach out if you are still seeing issues.
16:32 UTC | 9:32 UTC June 28
Our engineers have been monitoring performance since the updates were rolled out last Friday and can confirm the issue is now fully resolved. Please let us know if you continue to experience issues.
POST-MORTEM
Root Cause Analysis
This incident was caused by a new feature release of the Sunshine Conversation Widget that caused latency in Zendesk Support and Chat products. This feature release coincided with the release of other products on June 22, which made it harder to pinpoint which feature was the cause of the performance issues, resulting in delay to resolution.
Resolution
To fix this issue, we rolled back the version release of the Sunshine Conversation Widget. Upon customer feedback, we were able to confirm that the issue was resolved.
Remediation Items
- Creating monitors to detect for keystroke latency in Zendesk Support [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.