01:24 UTC | 17:24 PT
We experienced elevated Support search errors from 23:21 - 23:51 UTC. A fix has been deployed and search availability is back to normal.
00:21 UTC | 16:21 PT
Support search performance is now stable across all Pods. We will continue to monitor the situation and provide an update when fully resolved.
23:50 UTC | 15:50 PT
Our engineers are investigating intermittent search failures in Support across all Pods. More info to follow.
On November 22, 2019 from 23:21 UTC to 23:51 UTC customers on Zendesk Support across all Pods were unable to generate Search results or had an inconsistent Search experience.
This incident was caused by an authentication failure for the Search Service after a credential rotation. The issue was remedied by reverting to the previous credentials before updating the proxy hosts in each pod sequentially, verifying that authentication requests were again successful.
To prevent a recurrence of this issue, we are updating the runbook for the credential rotation process and upgrading the Search Service to send all traffic through a single endpoint.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.