SUMMARY
On December 25, 2021, from 13:10 UTC to 16:45 UTC some customers within the APAC (Asia Pacific) Region reported an increased level of HTTP 5xx errors when trying to access Zendesk, resulting in the inability to access the platform for the above mentioned period.
Timeline
16:45 UTC | 08:45 PT
We are happy to report that users getting server errors when trying to access Zendesk is now resolved. Please clear cache and cookies if you continue to see issues. Thanks for your patience!
16:16 UTC | 08:16 PT
Our service provider has identified an issue and are currently working on a fix. This issue is impacting users in the Asia Pacific region. We’re starting to see some improvements and are continuing to monitor.
15:39 UTC | 07:39 PT
We are receiving reports of users getting a server error when trying to access Zendesk. The investigation is underway.
POST-MORTEM
Root Cause Analysis
This incident was caused by a networking issue at our CDN provider’s end that diminished its ability to perform data transfer.
Resolution
To fix this issue, a network routing change was done at our CDN partner’s end, and recovery was observed once this change had propagated to all the affected networks.
Remediation Items
- Our partner will distribute their resources more evenly, relying less on specific data centers.
- Our partner will increase their network traffic capacity to reduce congestion.
- Investigate why their automatic tools did not work as expected.
- Build a better alerting system for similar behaviour.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.