Service Incident - August 6th, 2020 - Ticket Creation Errors

SUMMARY

On August 6, 2020, from 22:50 to  23:11 UTC, Agents experienced errors in the Support UI preventing tickets from being created and updated. Ticket creation that was attempted via API would have failed and tickets were not able to be created via the api/v2/tickets endpoint. Talk, Chat and incoming emails to Support encountered errors but were backfilled when service was restored. 

00:35 UTC | 17:35 PT

We’re happy to report that the issue in ticket creation for our Support product have been fully resolved and we have backfilled all email and Chat tickets. Thank you for your patience.

23:19 UTC | 16:19 PT

We have rolled back a change that resulted in increased error rates during ticket creation for our Support product from 22:50 to 23:11 UTC.

POST-MORTEM

Root Cause Analysis

A feature change behind a feature flag was rolled out that resulted in tickets being unable to be created. This feature was intended to allow for rate limiting on the /api/v2/tickets endpoint to remediate a recently identified issue that was affecting email delivery to a subset of customers. 

Resolution

Once the root cause was identified, the change was rolled back and ticket creation returned to normal functionality. Email and Chat tickets that were not created were then backfilled.

Remediations

1. Rollback code change
2. Improve feature flag change management process and testing
3. Update feature flag tool to enforce gradual rollout
4. Update feature flag documentation, and change runbook

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.