Service Incident - August 24th, 2020 - Search errors in Support on Pod 23

SUMMARY

On August 24th, 2020 from 5:03 UTC to 5:14 UTC, Zendesk Support customers on Pod 23 may have experienced errors when using Search.

TIMELINE

05:49 UTC | 22:49 PT
We encountered a brief issue with search functionality in Support between 05:03 - 05:14 UTC impacting all accounts on Pod 23. Users attempting to search would not have received any results during this period. The issue has since been fully resolved.

POST-MORTEM

Root Cause Analysis

This incident was caused by a misconfigured staging environment in Pod 23. The new environment was intended to be “non-live” however the roles for the new stage were configured incorrectly causing a Search service deploy to be rolled out to the live cluster. This erroneous deploy caused the issues some customers experienced on Pod 23.

Resolution

While our team was monitoring this deploy, they noticed the errors on Pod 23 and proceeded to fix the deployment stage and roll out the correct deploy to production to fix the issue.

Remediation Items

1. Consolidate standardization of deployment process across all projects in Search service [Scheduled].

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.