SUMMARY
On February 3-4, 2022 between 23:36 UTC and 01:12 UTC, Support customers on pods 28 and 29 experienced inbound email delays.
Timeline
01:25 UTC | 05:25 PT
We have detected inbound email delays on Pods 28 and 29 that occurred between 23:36 UTC and 01:12 UTC. This may have resulted in delays in ticket creation and ticket updates (via email channel) in Support on those Pods. Our systems have recovered and processed the email backlog. We apologize for any inconvenience this caused and will follow up with a post-mortem in due course.
POST-MORTEM
Root Cause Analysis
The incident was caused by a configuration change to our mail processor that resulted in a zone file being omitted and causing DNS lookups to return wrong results.
Resolution
The code change was reverted and email processing returned to a normal state.
Remediation Items
- Review and optimize mail processing change management
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.