Service Incident - November 17th, 2020 - Support Ticket Fields displayed incorrectly

SUMMARY

On November 17, 2020 from 08:16 UTC to 10:24 UTC, Zendesk Support customers on all Pods experienced unordered ticket fields on their ticket forms.

TIMELINE

09:53 UTC | 01:53 PT
We are aware of customers experiencing an issue with how ticket fields are displayed in the support interface, we are currently working to understand the cause of this and will keep you updated as we find out more.

10:27 UTC | 02:27 PT
We have identified the cause of the issue with ticket fields displaying in the incorrect order within the support interface and we are working to have this resolved, we will continue to provide updates.

11:02 UTC | 03:02 PT
We are happy to report that we have resolved the issue with ticket fields displaying in the incorrect order, please refresh your browser and reach out if you continue to experience an issue.

POST-MORTEM

Root Cause Analysis

This incident was caused by a new Support deploy that sought to optimize the time taken to load ticket fields in Support ticket forms. The new code changed the ticket fields array to order by all ticket fields on the account instead of the intended list of ticket fields on the loaded ticket form. This resulted in the incorrect order of ticket fields that the customers were seeing on the Support ticket forms.

Resolution

To fix this issue, the Zendesk Engineering team reverted the deploy that caused the issue.

Remediation Items

1. Add automated tests targeting ticket field ordering in Support ticket forms [Scheduled].
2. Explore deploying similar changes in Support behind a gate in future [Scheduled].
3. Investigate a more incremental Support release strategy [Scheduled].

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.