SUMMARY
On July 31, 2021 from 00:54 UTC to 2:41 UTC, customers on Pods 13 and 20 who had hostmapping configured experienced DNS errors trying to access multiple products.
Timeline
06:38 UTC | 23:38 PT
We are happy to report that the issue causing DNS errors on Pod 13 & 20 have now been resolved. Thank you for your patience.
03:01 UTC | 20:01 PT
We have identified the issue causing DNS errors on Pod 13 & 20. And customers have confirmed recovery. We are investigating the cause.
02:30 UTC | 19:30 PT
We've received reports of users encountering a DNS error message on Pod 13 & 20 when trying to access their help centre. Customers are now starting to see recovery. More details in 15 minutes.
02:14 UTC | 19:14 PT
We’ve received reports of users encountering a DNS error message when trying to access their help centre. Our team is investigating, more scope in 15 mins
POST-MORTEM
Root Cause Analysis
This incident was caused by a defect causing a delay in an automated job used to update DNS records.
Resolution
The incident self-resolved.
Remediation Items
- Improve network monitoring to better detect similar shifts in traffic.
- Include additional testing for host mapped domains.
- Improve latency handling in automated jobs.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.