UPDATE November 19, 2019
During this incident, email DKIM (Domain Keys Identified Mail) signatures were not being attached to outgoing email notifications. This resulted in the unsuccessful delivery of some emails. The issue resulted from a code deployment that was immediately reverted upon discovery on 2019-11-17 03:55 UTC.
DKIM signatures are used to associate a domain name with an email message, thus vouching for its authenticity. For customers who have a DMARC (Domain-based Message Authentication, Reporting & Conformance) policy that is set to reject, or quarantine emails that do not include the requisite DKIM and SPF (Sender Policy Framework) records, emails sent during this period may have been rejected by the receiving mail servers.
We are not automatically resending these notifications to avoid confusion with previous ticket updates and potential duplicate notifications. For impacted accounts using DKIM, we have already notified account owners via email of this incident. If you would like Zendesk to attempt to resend these notifications, please have your account owner email firstname.lastname@example.org.
04:05 UTC | 20:05 PT
Our team has identified and resolved the cause of outgoing email notification issues on all pods. If you're still experiencing issues, please reach out to us.
02:48 UTC | 18:48 PT
We are currently investigating reports of issues with Outgoing Email Notification for multiple customers on all pods. Our team are proactively working towards a fix and will update you as soon as we have one available.
From 2019-11-15 19:06 UTC to 2019-11-17 03:55 UTC, DKIM signatures were not being attached to outgoing email notifications for Zendesk Support customers using that Email feature. For customers that have a DMARC policy that is set to reject emails that do not include the requisite DKIM and SPF records, emails sent during this period would have most likely been rejected by the receiving mail servers. ("Most likely rejected" because it is up to the receiving email server to honour the DMARC record.)
The issue was caused by a code change made to the Zendesk outgoing email processor. Once the issue was identified, the code was rolled back and outgoing emails were delivered moving forward.
In order to prevent this from happening again in the future, we’ve rolled back the code change and will be improving our email QA testing suite, email deliverability monitoring and alerting, and improving our email backfill process and runbook when these types of issues occur.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.