SUMMARY
On August 16, 2021, customers on Pod 23 using Search in the Help Center first experienced errors between 12:34 and 12:59 UTC, and then incomplete or missing results between 13:00 and 17:05 UTC.
Timeline
13:04 UTC | 06:04 PT
We are currently investigating search issues impacting the Help Center on Pod 23. We are working toward resolution.
13:24 UTC | 06:24 PT
We continue to work on degraded search performance impacting Help Center for customers on POD 23. More information to follow.
13:55 UTC | 06:55 PT
We are seeing improvements in the degraded search performance impacting customers on POD 23, and are continuing to work on a full resolution. We will provide more information as it is available.
14:40 UTC | 07:40 PT
We are continuing to work on the degraded search performance impacting customers on POD 23. We will post more information as we get it.
15:51 UTC | 08:51 PT
We are seeing improvements in results from Article searches but other search types may still see incomplete results on Pod 23 while we continue to work. We will provide more information as soon as it becomes available.
17:14 UTC | 10:14 PT
Article, Community, User Mention, and Guide Admin search are now fully restored for Pod 23. Thank you for your patience as we worked to resolve this.
POST-MORTEM
Root Cause Analysis
This incident was caused by an incorrectly scoped command run during scheduled decommissioning of unused infrastructure.
Resolution
To fix this issue, the affected processes were restarted.
Remediation Items
- Improve recovery runbooks and processes.
- Surface additional identifying information to better clarify impact during decommissioning process.
- Investigate methods to improve recovery speed.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.