On May 28, from 13:54 UTC to 16:21 UTC, all customers using the Zendesk Web Widget, regardless of Pod, saw a 502 error on the Web Widget instead of expected behavior such as surfacing articles, ticket forms, or allowing new chats.
17:05 UTC | 10:05 PT
We experienced a brief partial outage with web widget functionality. We have rolled back a recent change that caused the issue. If you're still experiencing issues with web widget functionality, please reach out to us.
Root cause Analysis
This incident was caused by a configuration change to the Web Widget to redirect traffic to our new Content Delivery Network provider. Our CDN provider identified an error in their internal tooling that sets the cache key, which caused the 502 errors as requests were not being cached and an increase in requests hit the EKR nodes directly.
To fix this issue, we rolled back the change to restore traffic to its original path as we investigated a fix with our CDN provider.
- Improve monitoring tools to detect these types of issues faster
- Improve communication across internal teams and CDN provider
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.