Service Incident - May 14th, 2019 - Connection Issues on Pods 13, 14, 19, and 20

SUMMARY

On May 14th, 2019 from 14:37 UTC to 15:18 UTC customers on Support & Guide on Pods 13, 14, 19, and 20 experienced slowness or saw a “This site can't be reached” browser error when navigating to Support & Guide across the US and Brazil.

Timeline

16:03 UTC | 09:03 PT 
We are happy to report slow performance affecting customers in pods 13,14,19, and 20 is now resolved.

15:49 UTC | 08:49 PT 
Performance issues affecting pods 13,14,19, and 20 have begin to stabilize. Please let us know if you continue to experience issues.

15:15 UTC | 08:15 PT 
Our teams are continuing the investigation of slow performance affecting some customers. We have determined that this may be affecting customers on Pods 13, 14, 19, and 20.

14:51 UTC | 07:51 PT 
We are investigating performance slowness reports in several parts of the US and Brazil. More info to come shortly.

Root cause Analysis

This incident was caused by an issue at a local Content Delivery Network (CDN) point-of-presence (PoP) that serves Zendesk site assets to customers. A fiber path was disrupted which resulted in rerouted traffic and caused a degradation of service or intermittent connection.

Resolution

The issue was resolved once our CDN provider had restored their fiber path.  

Remediation Items

We continue to implement additional alerting tools between Zendesk and our CDN provider to improve communication between our engineering teams to reduce impact duration.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.