Summary
On October 25, 2025 from 02:00 UTC to 15:40 UTC, Zendesk Support (Agent Workspace) and Sunshine Conversations customers on Pods 15 and 19 experienced various issues in those products including messaging failures, ticket creation delays and other performance issues. These symptoms occurred at different time periods within the impact window.
Timeline
October 25, 2025 02:30 UTC | October 24, 2025 7:30 PT
October 25, 2025 03:00 UTC | October 24, 2025 8:00 PT
We have identified the issue causing delays and 5xx errors for SunCo on pod 15 and are in the process of remediation. We will provide an update within 30 minutes or when we have more information to share.
October 25, 2025 03:00 UTC | October 24, 2025 8:00 PT
We are starting to see recovery in the latency and error rate impacting Sunshine Conversations in pod 15. We will update when we observe full recovery or in one hour.
October 25, 2025 03:15 UTC | October 24, 2025 8:15 PT
We are able to now confirm that the latency and error rates have returned to normal for messaging on pod 15. We apologize for the disruption.
October 25, 2025 03:45 UTC | October 24, 2025 8:45 PT
Although messaging latency and 5xx errors in sunshine conversations have returned to normal we are observing and working to resolve delays in ticket creation. We will provide an update in 30 minutes or less.
October 25, 2025 04:00 UTC | October 24, 2025 9:00 PT
We apologize for the confusion, but unfortunately we are noticing increased latency and errors again on pod 15 sunshine conversations. We have also identified an issue impacting pod 19, however this is impacting ticket creation. These issues are related and we will provide prompt updates as we learn more.
October 25, 2025 05:00 UTC | October 24, 2025 10:00 PT
We are observing stability once again in sunshine conversations messaging on pod 15. We continue to investigate the issue causing ticket creation delays for some customers on pod 19.
October 25, 2025 07:45 UTC | October 25, 2025 00:25 PT
Our engineering teams are still working to resolve the issue impacting pod 19 ticket creation delays. We apologize for the continued disruption.
October 25, 2025 14:25 UTC | October 25, 2025 07:25 PT
After several hours of working to restore normal ticket creation on pod 19 we are starting to see our backlog of ticket creation jobs get caught up. We do still observe delays on ticket creation, but we hope to have an update shortly confirming normal ticket creation timing.
October 25, 2025 15:15 UTC | October 25, 2025 08:15 PT
While monitoring for full recovery, we continue to see the backlog of message and ticket creation processing moving in the right direction. We also wanted to share an update on the scope; the delays in ticket creation are also having an impact to ticket assignment. We apologize for the confusion and frustration this additional impact has caused. We will provide an update in one hour (or less).
October 25, 2025 16:00 UTC | October 25, 2025 09:00 PT
We continue to see positive results processing through the backlog of messages, and aim to confirm in our next update full recovery for message ticket creation and assignment on Pod 19. Some customers may still experience delays and messages not routing to agents at this time. We apologize for the lengthy impact and appreciate your understanding.
October 25, 2025 17:00 UTC | October 25, 2025 10:00 PT
We are very pleased to report, after monitoring, the backlog of unprocessed messaging tickets remains clear. We thank you greatly for your patience while we worked to resolve this issue. Please reach out if you have any questions.
October 28, 2025 01:52 UTC | October 27, 2025 18:52 PT
We are pleased to inform you that the issues with Sunshine Conversation, including SunCo errors and latency on Pod 15, as well as ticket creation delays on Pod 19, have been resolved. We appreciate your patience and cooperation throughout the resolution process.
Root Cause Analysis
This incident was caused by a sudden surge in Messaging SDK traffic. This unusually high request volume caused database and server resource exhaustion leading to delays in event processing. This was exacerbated by cross-service latencies, especially in Support. As a result, customers experienced service slow-downs, errors, and significant delays in receiving and sending messages as well as ticket creation delays. These latencies also impacted associated API endpoints that may have caused impact to apps and integrations leveraging those services.
Resolution
To fix this issue, our teams increased infrastructure capacity by scaling up our databases and processing systems to handle the traffic and clear our messaging backlogs. In addition, we adjusted system configurations to improve processing speed and to increase system resiliency for these sudden traffic spikes.
Remediation Items
- Improve system monitoring and alerting to improve detection times.
- Enhance infrastructure scalability so our systems can better handle sudden increases in demand.
- Review and optimize rate limit strategies to prevent platform overload.
- Strengthening cross-region coordination to reduce the risk of delays when one part of the system is impacted.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.