SUMMARY
On April 19, 2022 from 12:55 UTC to 13:35 UTC, customers using any Zendesk products in the EU region around Berlin, Germany or nearby colos, would have experienced increased latency and connectivity errors. This was caused by issues on our global network provider’s end.
Timeline
13:58 UTC | 06:58 PT
Between 13:05 and 13:36 UTC, our global network provider encountered issues that affected platform availability and connection for some of our Support customers in the EU region.
14:28 UTC | 07:28 PT
Our global network provider experienced an issue in one of their hubs between 13:05 and 13:36 UTC, resulting in Zendesk platform unavailability and connection issues for some of our customers in the EU region, across multiple products.
POST-MORTEM
Root Cause Analysis
This incident was caused by our global network provider in the Berlin, Germany location having a power issue on-site, which caused them to lose access to all their devices on the site.
Resolution
To fix this issue, their infrastructure teams reached out to the site to get the issue checked and resolved. Once power was restored, our global network provider resorted to access to their edge network device and was able to disable anycast. Once anycast was disabled, customers would have been diverted to neighboring nearby locations in Germany and able to connect successfully.
Remediation Items on Zendesk’s end
- Implement alerts for percentage of 5xx errors by colo. [Done]
Recommendations on our Network Provider’s end
- Continue investigating why anycast was advertised after a cold reboot of the network device.
- Replace legacy power distribution units in their network configuration.
- Investigate further with the Berlin location on power issues.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.