SUMMARY
On December 15th, 2021 from 15:10 UTC to 16:00 UTC, many customers on Pods 13 and 20 (US-WEST-2 region) experienced network connectivity issues and were unable to connect to all Zendesk products. Chat customers on multiple pods were also unable to access Zendesk Chat.
Timeline
15:34 UTC | 07:34 PT
Our teams are investigating an outage impacting customers on Pod 13 & 20. More info to come shortly.
16:01 UTC | 08:01 PT
We have confirmed a network connectivity issue which is affecting customers’ ability to connect to the Zendesk and other partners across the US Service region. We continue to work with our providers to mitigate or restore service for affected users. Our support portal is also affected by this, so customers may get delayed response from our support team.
16:28 UTC | 08:28 PT
This incident impacting Support customers on pods 13 & 20 and Chat customers across multiple pods is now resolved. Please clear cache and cookies if you continue to see issues. Thanks for your patience!
POST-MORTEM
Root Cause Analysis
This incident was caused by our cloud service provider experiencing internet connectivity issues to the US-WEST-2 region.
Resolution
Our cloud service provider identified the root cause of the internet connectivity issue and worked it to resolution. We verified that all Zendesk services were back to normal from 16:08 UTC.
Remediation Items
- Performed Zendesk incident retrospective to identify improvements we can make to our system resilience to vendor outages. [Scheduled]
- Improve our business continuity plan to third party vendor failures, including identity and access management, monitoring, and alerting services. [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.