Service Incident - March 25th, 2022 - Support | Multiple Pods - Blank ticket screen issue

Summary

From March 24, 2022 at 19:00 UTC until March 25 at 10:00 UTC, a subset of Zendesk Support customers on multiple Pods experienced blank ticket screens when loading tickets in the product.

Timeline

07:05 UTC | 00:05 PT
Our team is investigating reports of slow loading or blank screens when opening tickets in Support for our customers on multiple Pods. More updates to follow.

07:46 UTC | 00:46 PT
We continue to investigate reports of slow loading or blank screens when opening tickets in Support for our customers on multiple Pods.

08:17 UTC | 01:17 PT
Our team continues to investigate the issue with slow loading or blank screens when opening tickets in Support for our customers on multiple Pods. We will share an update once we have additional information.

08:50 UTC | 01:50 PT
We continue to work to find the root cause of the issue with slow loading or blank screens when opening tickets in Support for our customers on multiple Pods. We will share an update within the hour.

10:07 UTC | 03:07 PT
Our engineering teams continue their work on the issue in Support causing slow loading or blank screens when opening tickets currently impacting multiple Pods. We will share an update in an hour or as we find out more.

11:10 UTC | 04:20 PT
Thank you for your patience, we are happy to report that our engineering team has resolved the issue in Support causing slow loading or blank screens when opening tickets. In case you would still encounter this behaviour, please clear your browser cache and cookies. This is needed to ensure your browser loads the most up to date version of Support.

Root Cause Analysis

This incident was caused by a bug in a recent Support user interface optimization update. The change was rolled out to approximately 20% of customers and resulted in a small subset of customers experiencing the issue.

Resolution

To fix this issue, our team switched off the feature gate that was progressively rolling out the change.

Remediation Items

1. Investigate the code defect and iron out the issue before reintroducing the optimization [Scheduled]
2. Improve feature gate identification process and tooling [Scheduling]
3. Update runbooks to improving team paging process [Scheduled]
4. Investigate additional monitoring and alerts to help prevent these types of service interruptions in future [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.