On July 3 2019, at 20:49 UTC / 2:49 PM PDT, Our customers on Pod 19 experienced slow performance and inability to load Support, Talk, and Chat, receiving green screen errors.
22:12 UTC | 15:12 PT
Pod 19 performance issues are resolved, all products are functioning as normal. We apologize for the inconvenience this may have caused.
21:45 UTC | 14:45 PT
Performance issues have improved on Pod 19 with Support, Talk, and Chat products and we're continuing to monitor performance.
21:17 UTC | 14:17 PT
We are currently investigating performance issues in Pod 19 with Support, Talk, and Chat products. More information shortly.
Root Cause Analysis
A number of API servers became unresponsive as they got overloaded after a chain reaction that started with a configuration upgrade of our code deployment system on Pod 19.
It has been determined that an automated upscaling process did not operate as intended causing a capacity issue on the API server level.
The automated upscaling process was temporarily disabled and manually managed to help free up resources which got the API servers back on track.
-Additional logging to monitor our code deployment system has been added
-Automated upscaling system reconfigured to prevent such a scenario to occur again in the future
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.