Service Incident - August 27th, 2020 - Agent Workspace Performance Issues

SUMMARY

On August 27th, 2020 from 07:50 UTC to 13:56 UTC, some Zendesk Support customers using Agent Workspace on all Pods experienced performance issues across multiple features of the product.  

TIMELINE

12:35 UTC | 05:35 PT

We are investigating connection issues when attempting to access the Support interface for some customers on Pod 17, updates to follow.

14:05 UTC | 07:05 PT

We have rolled back a change which was impacting performance on our Zendesk Support product for some customers. Please hard refresh your browser if you do not see improvements.

14:41 UTC | 07:41 PT

We’re happy to report that we have resolved the issue with performance issues in the support interface. Please log out and log back in if you continue to see issues. Thank you for your patience.

POST-MORTEM

Root Cause Analysis

This incident was caused by a code change that sought to optimize timestamps in the Support frontend. A specific dependency in the code caused specific functions to loop resulting in the symptoms customers observed (e.g. tickets unable to load, submit or close).

Resolution

To fix this issue, the code change was identified and reverted.

Remediation Items

1. Add end-to-end tests to cover more scenarios such as the ones encountered in this incident [Scheduled].
2. Review other render flows across the product and refactor them [Scheduled].
3. Share lessons learned with other engineers to prevent similar incidents in future [Scheduled].

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.