Service Incident - June 23, 2023 - Chat & Support (Agent Workspace) | All Pods - Access issues for European agents

SUMMARY

On June 23, 2023 from 05:15 UTC UTC to 08:20 UTC, Chat & Support [Agent workspace] customers across multiple Pods, based in Germany or using a specific ISP provider from the European region, experienced an inability to access and login into the aforementioned products, as well as long loading times, “Page unresponsive” and white screen errors. 

Timeline

07:03 UTC | 00:03 PT
We are investigating reports of Support access issues. More to come.

07:19 UTC | 00:19 PT
We continue to investigate localized Zendesk access issues in the European region. Agents may see blank pages in Support or be unable to login. Next update in 30 mins.

07:57 UTC | 00:57 PT
Our CDN provider has confirmed there’s an ongoing network issue on their end that affects our customer’s login to Zendesk across multiple products and Pods. More information in 1h or as we learn more.

08:37 UTC | 01:37 PT
We have implemented a workaround for our customers across multiple Pods, but mostly affected in the European region, who had issues logging in into their Support accounts. Our CDN provider continues working to fully address this issue. Thank you for your ongoing patience.

09:17 UTC | 02:17 PT
We are happy to inform you that our CDN provider has confirmed a fix was applied on their end. We are monitoring and will soon revert the workaround we implemented when we ensure full resolution for the issues affecting multiple Zendesk products’ usage, across multiple Pods and mainly in the European region.

10:03 UTC | 03:03 PT
We have now reverted the workaround as our CDN partner fully addressed the issue on their end and our monitoring shows stability. Please clear cache and cookies and let us know if you see any further issues. Thank you for your help and patience.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by a third-party route leak of a German ISP's prefixes, resulting in high latency and degraded performance for our CDN providers' accounts behind this ISP. 

Resolution

To fix this issue, at the same time as raising it with our CDN provider, we triggered a manual DNS failover which alleviated almost instantaneous the negative impact the customers observed. 

Remediation Items

1. None from Zendesk’s side.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.