SUMMARY
On February 17, 2022 from 09:10 UTC to 12:00 UTC, Support customers on Pod 17 experienced errors when accessing view counts of tickets on their account Views list.
Timeline
10:29 UTC | 02:29 PT
We are currently investigating reports of Support customers on Pod 17 having issues loading Views. More details to come.
10:47 UTC | 02:47 PT
We appreciate your patience while we work through Support issues on Pod 17 regarding ticket numbers not appearing in Views. Our team is actively working on remediation items for this.
11:29 UTC | 03:29 PT
Our team continues investigating the issue for ticket count not showing in Views, for a subset of Support customers on Pod 17. We shall provide more information in 1h.
12:29 UTC | 04:29 PT
We are happy to report that the issue impacting Views in Support for our customers on POD17 have now been resolved. You might need to clear your browser cache and cookies for this to take full effect. We apologize for the inconvenience.
POST-MORTEM
Root Cause Analysis
This incident was caused by an unsuccessful network connection where a bad host was taking too much traffic in the view execution service. This made it overwhelming to the point that it could not connect to load the ticket count anymore in the Views.
Resolution
To fix this issue and reduce the backlog of failed attempts, we temporarily disabled the view count endpoint to a small subset of traffic in Pod 17 and the host was taken out of rotation to help improve the flow and circulation/traffic of execution attempts. With those actions, the backlog started to go through, without affecting new requests. Once the numbers decreased in the backlog, the view count endpoint was re-enabled and the issue was resolved, with customers being able to load and see View numbers again.
Remediation Items
- Reduce the view execution service backlog size by adding it to a mesh in order to fastly process the information [In Progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.
1 Comments
Post-Mortem published March 9, 2022.
Article is closed for comments.