Service Incident - February 7th, 2020 - Pod 20 Ticket creation and outbound email delays

Summary

On February 7, 2020 from  02:46 UTC - 03:37 UTC Support customers on Zendesk Support on Pod 20 experienced delays with ticket creation and outbound email notifications. 

Timeline

04:10 UTC | 20:10 PT

We observed temporary drop in performance between 02:46 UTC - 03:37 UTC, this affected ticket creation and outbound email notifications in Support on POD 20. We now have a resolution on this incident.

Root Cause Analysis

An incorrect caching configuration value for a deployment caused email processing and other background jobs in Pod 20 to be delayed for up to 30 minutes as the worker processes were crashing on startup.

Resolution

Once the caching configuration value was fixed, tickets creation and outbound email notification were restored. In order to prevent this from happening again in the future, we will be implementing improved monitoring and logging.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.