Service Incident - August 22nd, 2019 - Search Issues on Pod 14

SUMMARY

On August 22, 2019 from 12:49 UTC to 15:00 UTC customers on Pod 14 experienced an inability to search or high latency in retrieving results from search in Support and Talk calls may have dropped when transferring between agents.

16:02 UTC | 09:02 PT
We experienced a brief recurrence of the issue from 15:58 UTC to 16:02 UTC and apologize for any interruption of service. We expect service to be fully restored again and again appreciate your patience. 

15:53 UTC | 08:53 PT
We are happy to announce that the Search issues impacting Support on Pod 14 are now resolved. Thank you for your patience.

14:50 UTC | 07:50 PT
Our teams continue to investigate the Search issues impacting Pod 14. The following areas may be degraded during this time: Zendesk API, searching for users, tickets, tags & transferring calls on Zendesk talk. More information to follow.

14:03 UTC | 07:03 PT
Our teams continue to actively work on the Support Search issues currently being reported on Pod14. More info in 60 minutes or earlier. Thanks for bearing with us.

13:30 UTC | 06:30 PT
We are still investigating reports of search issues in Support on Pod14, more info to come soon.

13:14 UTC | 06:14 PT
We are currently investigating reports of search issues in Support on Pod14, more info to come soon.

Root cause Analysis

This issue was caused by problematic queries to a Search API endpoint. 

Resolution

The excessive traffic was blocked to restore Search service to Pod 14. 

Remediation Items

Create additional alerts and update handling for long query terms.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.