SUMMARY
On March 5th, 2024, between 1:53 AM and 2:12 AM UTC, a subset of Support customers on pods 13 and 20 experienced slow connection speeds or error messages while accessing their Zendesk account. This was caused due to networking issues with our CDN vendor
Timeline
05:09 UTC | 21:09 PT
We are happy to report that the performance issues in Support on pods 13 and 20 are now resolved. Investigations currently point to a content delivery network issue in the U.S. region. More information will be provided in our post-mortem report in due course. Thanks for your patience through today’s issues.
03:07 UTC | 19:07 PT
We are investigating performance issues in Support on Pods 13 and 20. Symptoms may include green error screens, message ticket loading issues and general slowness loading pages. While partial recovery has been observed in some areas, we are still getting to the bottom of the issue. Please reach out to us if you are still experiencing issues. More to come in 30 minutes.
02:41 UTC | 18:41 PT
We are investigating reports of errors and performance issues in Support on multiple pods. More information to come shortly.
POST-MORTEM
Root Cause Analysis
Customers using Argo Smart Routing being routed through Portland saw increased failure rates when our CDN vendor disabled a bad transit in Portland which caused congestion in the location. This was due to a transit provider experiencing packet loss in the United States.
Resolution
Our CDN vendor rerouted a large volume of traffic to mitigate the issue then resolved the issue at 2:12AM UTC and performance returned to normal levels.
Remediation Items
- Improve CDN vendor monitoring and alerting
- Review internet resilience monitors
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.