SUMMARY
On March 23th, 2021 from 14:40 UTC and 15:59 UTC, Zendesk agents and end-users with traffic routing through Ashburn, VA, US East region experienced elevated error rates or degraded performance accessing Zendesk Support, Guide and Talk products.
Timeline
20:41 UTC | 13:41 PT
From 14:50 UTC - 15:59 UTC, Support customers across all pods may have experienced elevated error rates and degraded performance. This was due to an issue caused by our CDN provider having network performance issues but is now resolved
Root Cause Analysis
This incident occurred when our Content Delivery Network (CDN) provider was experiencing a network performance issue in Ashburn, VA and rerouted traffic to other point of presences (POP).
Resolution
To fix this issue, our CDN provider implemented a fix and was monitoring the results at 15:52 UTC. Our monitoring systems indicated the performance issue was resolved and stable at 15:59 UTC and connectivity to Zendesk Support, Guide and Talk products returned to normal for the affected customers. At 16:14 UTC, our CDN provider called an all clear for this incident.
Remediation Items
- Improve detection time for regional connectivity issues
- Change Zendesk internal service calls to go via AWS backbone rather than via CDN to reduce the impact of CDN incident
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.