Service Incident - May 19th, 2020 - Pod 19 Search Issues

SUMMARY

On May 19, 2020 from 21:35 UTC to 21:57 UTC customers using Zendesk Support on Pod 19 experienced degraded performance when using Search.

TIMELINE

00:04 UTC | 17:04 PT
Search issues in Support on Pod 19 are now resolved. Thanks for your patience while we worked through this issue and apologies for the inconvenience this may have caused you and your team.

23:44 UTC | 16:44 PT
Support search functionality on Pod 19 is now stable. Our team is continuing to monitor the situation and will provide another update once we've confirmed the issues are fully resolved.

22:42 UTC | 15:42 PT
We have identified the root-cause of this search issue and are actively taking steps to prevent it from recurring.

22:11 UTC | 15:11 PT
We’ve received reports of search issues affecting Pod 19. We’re actively investigating and will provide an update within the hour.

Root Cause Analysis

This incident was caused by a significant uptick in search timeouts due to excessive memory intensive queries on Pod 19.  

Resolution

To fix this issue, we increased capacity and restarted the affected search nodes while rate limiting the problematic queries.

Remediation Items

We are Investigating new mechanisms to manage memory intensive queries to avoid recurrence of this issue.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.