SUMMARY
On July 12, 2021 from 22:01 UTC to 22:25 UTC, some customers using Zendesk Support on Pods 19 and 23 experienced new inbound emails delays between the mentioned times, and that backlog of emails was cleared soon after, processed according to the timeline below.
Timeline
23:08 UTC | 16:08 PT
The email processing delay affecting some inbound emails on pods 19 and 23 has been resolved. Inbound emails sent between 22:01 and 22:25 UTC were successfully processed as of 22:42 UTC.
22:42 UTC | 15:42 PT
We are investigating an issue where processing of some inbound emails between 22:01 and 22:25 UTC are delayed on pods 19 and 23. All new emails are processing as expected.
POST-MORTEM
Root Cause Analysis
This incident was caused by an incorrect configuration of our DNS host being pushed out to the Pods in question.
Resolution
We manually addressed the affected DNS hosts that had received this incorrect pushed configuration with the correct IP addresses to fix this issue.
Remediation Items
- To inform any changes made to data outside the original scope, our team will create an internal alert for the tool that initially pushed the wrong configuration.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.