Service Incident - October 23rd, 2019 - Chat and Support Ticket Suspensions

SUMMARY

On October 23, 2019 from 08:34 UTC to 08:49 UTC customers on Zendesk Support and Chat across all Pods experienced Chat transcript tickets and legitimate Support tickets being sent to the Suspended queue as they were marked as Spam upon creation.

16:53 UTC | 09:53 PT
In regards to the earlier issue with suspended tickets, the backfill of suspended Support tickets has been completed and those tickets should now be visible in your normal ticket queues. Thanks again for your patience!

12:15 UTC | 05:15 PT
We have started the backfill of Support tickets that were inadvertently suspended earlier today. They should gradually show up in your normal ticket queues. We will provide an update when the process has completed. Thank you for your patience.

10:58 UTC | 03:58 PT
Our teams continue to investigate the issue which inadvertently marked some Chat & Support tickets as spam. At this time, Chat tickets have been backfilled and should be visible in your normal ticket queues. We will provide an update in 60 minutes.

10:02 UTC | 03:02 PT
These tickets can be found in your Suspended Ticket Queue. Our teams continue to investigate. We will provide an update in 60 minutes

10:01 UTC | 03:01 PT
We experienced an issue with our Spam Filter which inadvertently marked some Chat & Support tickets as spam. This occurred over a 15 minute period from 08:34 UTC to 08:49 UTC this morning.

Root Cause Analysis

This incident was caused by a new pattern being added to our spam prevention tool which was present in the tickets that were then Suspended.

Resolution

To fix this issue, we removed the new pattern from the spam prevention tool.

Remediation Items

1. Backfilled Chat and Support tickets sent to Suspended queue upon creation
2. Investigate new methods to capture rapidly changing spam patterns
3. Additional causes of suspension to be listed in Suspended queue

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.