SUMMARY
On June 9, 2025 from 7:13 UTC to 10:10 UTC, customers experienced issues with messaging triggers not functioning as expected, along with difficulties performing searches using the search bar in Zendesk Support.
Timeline
June 09, 2025 08:52 AM UTC | June 09, 2025 01:52 AM PT
We are currently aware of the issue affecting support search and messaging trigger problems on Pod 25, which began today at 07:20 UTC. The symptoms include some customers’ messaging triggers not working as expected and difficulties in performing searches using the support search bar. Our engineering team is actively investigating the matter. We will provide an update within the next 30 minutes.
June 09, 2025 09:36 AM UTC | June 09, 2025 02:36 AM PT
Our engineers are still actively investigating the contributing factors to this issue where customers are unable to search in the Support dashboard and getting 503 API errors in Messaging. We apologize for this inconvenience and appreciate your patience as our team works to mitigate this issue. We will send updates as it becomes available.
June 09, 2025 10:58 AM UTC | June 09, 2025 03:58 AM PT
We have received information from our engineers that the issue with the Support Search has subsided at this time. Our team is continuously monitoring the issue to identify the contributing factors that led to this issue. We appreciate your patience and recommend you to follow this communication for significant updates as we will provide it as it becomes available.
June 09, 2025 05:45 PM UTC | June 09, 2025 10:45 AM PT
We have remained stable since recovering from the search issues experienced in Support today. Our team will continue to monitor the situation to ensure no further issues are encountered, and we will provide additional information when we have a substantive update to share. Thank you for your continued patience.
June 11, 2025 01:26 AM UTC | June 10, 2025 06:26 PM PT
We are happy to report that the search issue in Support is now resolved. We apologize for the delayed closure message for this incident and appreciate your patience while we worked through this issue. Thank you.
Root Cause Analysis
This incident started when one of the data servers slowed down briefly around 07:13 UTC on June 9. This caused the system to become overloaded and produce more errors. As a result, automatic retry attempts kept flooding the system with repeated requests, keeping it under heavy stress for almost three hours.
Resolution
The issue resolved naturally without any intervention as the volume of requests declined later in the day, aligning with the end of the business day for the affected customers.
Remediation Items
- Conduct a follow-up meeting with the engineering teams involved to discuss extra protections for apps beyond standard limits.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.