SUMMARY
On May 31, 2023 from 05:10 UTC to 08:56 UTC, customers on multiple pods may have experienced slowness or latency when using various Zendesk products.
Timeline
08:34 UTC | 01:34 PT
We are aware of latency and slowness issues with loading Support and Chat in Agent Workspace across multiple Pods. Investigation is underway.
09:03 UTC | 02:03 PT
We identified an issue that seems related to a DNS provider ongoing incident. The team is currently working on a possible workaround to get customers to load. Customers may face latency and slowness issues with loading most Zendesk Products, on multiple pods. Update in 30 min.
09:22 UTC | 02:22 PT
Our team is seeing improvement on logs for loading products across multiple regions and Pods. We kindly ask you to clear cache and cookies and confirm if the situation has gotten better. New updates as we learn more. Thanks for your patience!
10:13 UTC | 03:13 PT
We are happy to confirm that customers are being able to load and access Zendesk products across multiple Pods. The team reverted the temporary workaround as our CDN* (corrected) provider has also confirmed the incident resolution on their end. Thank you!
POST-MORTEM
Root Cause Analysis
This incident was caused by Zendesk’s CDN partner experiencing connectivity issues in the Asia Pacific region, which impacted our customers’ ability to connect to Zendesk services.
Resolution
To fix this issue, a failover was initiated to bypass the CDN in the affected region. After our CDN partner had applied a fix, the CDN bypass was reverted. We continued to monitor and noted that the system was stable.
Remediation Items
- Improve the bypass procedure for more expedient response timing [Scheduled]
- Review adding additional metrics to improve monitoring of similar errors [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.