21:50 UTC | 13:50 PT
We are happy to report that the Pod 20 issue is now resolved and that email processing, API ticket creation and talk are now fully recovered.
21:16 UTC | 13:16 PT
We are investigating reports of Support email delays, ticket creation API errors, and Talk dropped calls for accounts on Pod 20.
From 2019-11-11 19:14 UTC to 20:30 UTC, Customers experienced delayed Inbound and Outbound emails, errors when creating tickets via API, and Talk dropped calls.
The incident was the caused by an internal script that was run to create user, organization, and ticket data. The script was stopped, rate limits were applied, and additional workers were added to process work queues as performance returned to normal.
In order to prevent this from happening again in the future, we will implement these types of changes in staging with a different API endpoint.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.