SUMMARY
On February 06, 2023 from 05:14 UTC to 07:20 UTC, Zendesk Support customers on Pod 25 experienced sporadic errors when performing search operations.
Timeline
06:13 UTC | 22:13 PT
We are investigating errors with the search service in Support on Pod 25. Customers may experience issues searching in multiple features in the Support product. More information to come.
06:27 UTC | 22:27 PT
We are seeing recovery after increasing resource capacity. Customers may continue to experience sporadic search errors in Support until full recovery is complete. Next update in 60 minutes or when we have more information to share.
07:29 UTC | 23:29 PT
We are glad to inform you that the issue affecting search in Zendesk Support has been resolved. Many thanks for your patience and continued partnership.
POST-MORTEM
Root Cause Analysis
This incident was caused by database resources being exhausted from heavy Search API volume usage on this specific database cluster.
Resolution
To fix this issue, more data nodes were assigned to speed up recovery of the cluster. We began to observe recovery, with metrics beginning to normalise. We continued to monitor until we received confirmation of recovery.
Remediation Items
- Update database cluster configuration for better memory allocation and performance [Scheduled]
- Improve database indexing to manage database cluster health [In Progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.