Service Incident - May 6th, 2022 - Support | Multiple Pods - Views not loading

SUMMARY

From May 5, 2022 from 21:17 UTC to May 6, 2022 14:50 UTC, some customers across multiple pods were unable to load their views in the Zendesk Support sidebar. 

Timeline

10:12 UTC | 03:12 PT

We are currently investigating customer reports where some Agents are unable to load Views in Support. This seems to be affecting multiple pods. We appreciate your patience while we check this further.

10:32 UTC | 03:32 PT

We appreciate your patience while we work to resolve the issue affecting Support Views not loading for some Agents. The possible workaround is to access them via “Manage Views” in the Admin Center.

11:01 UTC | 04:01 PT

We’re actively working to find the root cause and fix the issue where Views in Support are not loading for some Agents across multiple Pods. We shall provide another update in 1h or when we have more details.

12:03 UTC | 05:03 PT

We have identified the potential cause of this issue affecting Support Views and are working on a solution to fix it. We will send another update once the fix has been rolled out.

13:48 UTC | 06:48 PT

Our team has deployed a fix for the issue where some Agents were unable to load Views in Support. We continue monitoring for stability. Please refresh your browser and clear cache to see the update applied.

15:57 UTC | 08:57 PT

We are happy to report that the issue affecting the ability to load Views in Support is now resolved. Thank you for your patience during our investigation.

POST-MORTEM

Root Cause Analysis

This incident was caused by a deploy that introduced a regression related to views execution during a rollout on the previous day.

Resolution

To fix this issue, this deployment was rolled back on all pods, and the problematic library dependency update was reverted.  Error levels began to subside and customers were able to load views again.

Remediation Items

1. Create additional alerts for high rate of new errors. [To Do] 
2. Fix the regression introduced in the deploy. [In Progress]

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.