Summary
On October 19, 2022 from 09:30 UTC to 15:49 UTC, some Zendesk Support customers in the APAC region were unable to access the product.
Timeline
12:55 UTC | 05:55 PT
We are currently investigating reports of some customers not being able to load Zendesk. We will follow up with another update once we learn more.
13:17 UTC | 06:17 PT
We are currently working to understand the root cause of some customers on Pods 23, 25, 27 and 29 not being able to load Zendesk. We have been able to observe improvements for some customers. We will provide another update in 30 minutes, thank you for your patience
13:57 UTC | 06:57 PT
We would like to confirm that the issue with customers not being able to load Zendesk only affected customers based in APAC, meaning multiple Pods. This has now been mitigated and we continue to monitor. We will provide another update in 60 minutes or as soon as we know more
15:09 UTC | 08:09 PT
We’re continuing to work to resolve the issue that is causing some customers to be unable to load Zendesk. We are aware that some customers are still impacted. We will provide another update once we have more information, thank you for your cooperation.
18:18 UTC | 11:18 PT
We have applied a fix for the affected area and are beginning to see improvement in the access issues affecting customers based in the APAC region. Please let us know if you continue to experience any issues.
19:35 UTC | 12:35 PT
We are happy to report that the issue causing some customers to be unable to load Zendesk in the APAC region is now fully resolved. Sorry for the inconvenience this may have caused you and your team.
Root Cause Analysis
This incident was primarily caused by a local ISP issue resulting in agent traffic routed to our content delivery service to fail. A secondary factor impacted our attempt to failover to backup routes in a timely manner due to a bug in our SSL certificate management system.
Resolution
To fix the secondary factor, our team deployed updated SSL certificates to allow failover to backup routes.
Remediation Items
-
Additional monitoring and alerts for certificate changes and errors.
-
Improve visibility of certificate fetching and customer site-to-CDN connectivity.
-
Review certificate adoption strategy.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.