Service Incident - August 12th, 2020 - Connectivity Issues Pods 14, 19, and 23

SUMMARY

On August 12, 2020 from 01:53 UTC to 02:12 UTC, agents and end-users whose internet traffic routes through specific content delivery network regions experienced service degradation or were unable to access Zendesk products (Support, Guide, Talk, Chat, Sunshine, Explore) across Pods 14, 19, and 23.

Timeline

03:40 UTC | 20:40 PT
Services for all products on Pods 14, 19 and 23 have been restored. Thank you for your patience and co-operation.

02:41 UTC | 19:41 PT
We are currently investigating increased error rates across Pods 14/19/20/23. We will provide another update when available.

Root Cause Analysis

This incident was caused by an outage with our content delivery network that resulted in an overload of nearby datacenters and impacted traffic in the northeastern US and Montreal, Canada.

Resolution

Our content delivery network provider resolved the hardware issue to restore traffic across all impacted regions. 

Remediation Items

Investigate methods of increasing network connectivity observability and ways to mitigate regional failures.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.