On February 7, 2020 from 02:46 UTC - 03:37 UTC Support customers on Zendesk Support on Pod 20 experienced delays with ticket creation and outbound email notifications.
04:10 UTC | 20:10 PT
We observed temporary drop in performance between 02:46 UTC - 03:37 UTC, this affected ticket creation and outbound email notifications in Support on POD 20. We now have a resolution on this incident.
Root Cause Analysis
An incorrect caching configuration value for a deployment caused email processing and other background jobs in Pod 20 to be delayed for up to 30 minutes as the worker processes were crashing on startup.
Once the caching configuration value was fixed, tickets creation and outbound email notification were restored. In order to prevent this from happening again in the future, we will be implementing improved monitoring and logging.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.