Service Incident - February 27th, 2020 - Ticket Form changes not saved

SUMMARY

On the 26th of February, between 20:11 UTC and 15:04 UTC when submitting a ticket update, some of our Support customers experienced issues with changes made to the Ticket form not saving, and reverting to the previous value. There was also no indication of the change in the Ticket Events.

TIMELINE

15:46 UTC | 07:46 PT
Our development teams have resolved the issue where changes to Ticket Forms in Zendesk Support would not save.

14:51 UTC | 06:51 PT
Our development teams are working to resolve an issue where changes to Ticket Forms in Zendesk Support may not save. We'll provide an update in one hour.

POST-MORTEM

Root Cause Analysis

On the 26th of February a deploy containing code for Contextual Workspaces was rolled out to parts of our customer base. In the new system, if specific conditions are met, Support would mistakenly undo any exchange of ticket forms, discarding any inputted information.
This meant that in the case an account had no Contextual Workspaces set up, Support reverted to the previous ticket form. 

Resolution

Once our Developers identified the deploy causing the behaviour, they rolled the change back and normal functionality was restored.

Remediation Items

Our Team is committed to adding an automatic ticket test feature to prevent an issue like this from reoccurring.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.