On October 22, 2019 at 18:31 UTC to October 23, 2019 at 01:04 UTC, Zendesk Support, Talk and Guide customers on all Pods experienced increased latency and intermittent failures when resolving Amazon Web Services (AWS) DNS queries.
03:36 UTC | 20:36 PT
We’re happy to report that the issue regarding the DNS resolution has now been resolved. Thank you for your patience while we worked through this and we apologize for the inconvenience.
01:58 UTC | 18:58 PT
We are seeing improvements in DNS resolution that caused errors in multiple Zendesk products. If you continue to see issues, please provide the agent's location to our Support team to aide further investigations. We continue to monitor and will provide an update when available.
01:01 UTC | 18:01 PT
We are currently investigating issues with DNS resolution related to our CDN provider. This may result in errors across multiple Zendesk products. We will be providing more information as soon as it becomes available.
Root Cause Analysis
This incident was caused by DNS resolution issues between our CDN provider and AWS.
To fix this issue, we redirected queries from amazonaws.com and zdassets.com to a different DNS cache and availability region not impacted by the DNS resolution issues, which resolved the issue.
- Failover queries to amazonaws.com and zdassets.com to a different region.
- Investigate Automatic TTL with our CDN provider for origin dns records.
- Evaluate failover to different AWS regions or serving different DNS cache.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.