05:11 UTC | 22:11 PT
We are happy to announce that search services have been restored to Pod 14. If you are still experiencing the issue, please reach out to us.
04:38 UTC | 21:38 PT
The issue with search on Pod 14 is currently stable. We are continuing to monitor the situation. Next update in one hour.
04:01 UTC | 21:01 PT
Our team has mitigated the issue. Users on Pod 14 should be seeing an improvement in Search. Our team is continuing to monitor.
03:25 UTC | 20:25 PT
Our team is currently investigating search issues on Pod 14. More information to follow.
On June 7 4:59 UTC, June 8 02:50 UTC, June 8 22:43 UTC, and June 11 14:30 UTC, customers on Pod 14 experienced multiple incidents resulting in search becoming unavailable and causing 502 errors resulting in service unavailability.
Our developers took several actions in order to mitigate the search performance issues including:
- Isolating a slow search index shard to dedicated data node
- Migrate Guide search traffic to a separate endpoint so that Support traffic isn't impacted
- Disable cjk_bigram token filter for Korean languages
- Added circuit breakers to Guide to further reduce the impact of search overloading
- Split the article index into multiple indices to compartmentalize abusive accounts
Our developers have identified the root cause to be an elasticsearch CJK bigram filter used to optimize Korean language search queries. Our developers have since disabled the CKJ bigram filter on Pod 14.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.