Summary
On December 15, 2025, between 13:03 and 13:44 UTC, an alert was received indicating elevated processing errors across multiple pods, resulting in a stall of the search indexing service.
Timeline
December 15, 2025 15:26 UTC | December 15, 2025 7:26 AM PT
On December 15, 2025, between 13:03 and 13:44 UTC, we identified an intermittent stale data issue in the search indexer service affecting customers across all pods for approximately 30-40 minutes. Our engineers have resolved the root cause, and search indexing has resumed normal operation. We appreciate your patience and encourage you to contact us if you continue to experience any related issues.
Root Cause Analysis
The incident was caused by a recent change that incorrectly set account status values to lowercase “deleted” instead of uppercase, leading to indexing failures.
Resolution
To resolve the issue, engineers reverted the account state values to the proper format, which restored real-time indexing across all pods. Additionally, backfill operations were initiated and completed to recover search data missed during the 30-40 minute incident.
Remediation Items
- Strengthen tools used to manage whale data to prevent errors and improve stability.
- Develop comprehensive runbooks to guide the use and maintenance of whale-related tools.
- Enable the indexer to store the most recent whale data temporarily to ensure continued accuracy.
- Establish a regular and documented schedule for whale data cleanup.
- Investigate automating the removal of accounts from search index tables after 70 days in line with data retention policies.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. If you have additional questions about this incident, contact Zendesk customer support.