On November 19, 2019 from 22:13 UTC to 23:19 UTC Zendesk Support customers on Pods 14 and 18 saw 502 errors within the Admin Settings pages and experienced issues with certain features such as attachments and ticket merging.
01:02 UTC | 17:02 PT
We are happy to report that we have resolved the issue impacting Support performance for accounts on pod 14 and 18. We apologize for the inconvenience caused.
23:49 UTC | 15:49 PT
We are seeing an improvement with the issue impacting Support performance for accounts on pod 14 and 18. We will continue to monitor the situation and provide an update when fully resolved.
23:15 UTC | 15:15 PT
After some initial investigation we can confirm that Pod 18 is also impacted along with Pod 14. Our teams are working to resolve the issue. We will provide an update in 30 mins.
22:45 UTC | 14:45 PT
We are currently investigating an issue affecting Support product performance on pod 14. We are assessing the scope of this issue and will provide an update in 30 minutes.
Root Cause Analysis
This incident was caused by a configuration error surfaced during a manual instance rotation. An instance was terminated, reducing running instances to 0, and the AutoScalingGroup was unable to spin up a new instance as the AMI was not found.
To fix this issue, the launch template was updated with the new AMI which spun up new instances and restored service to normal.
- Create additional alerts when currently used AMIs are missing or when instances fail to start
- Automate instance rotation with checks in place to ensure adequate resources are available
- Investigate deployment process automation improvements and runbook updates
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.