SUMMARY
On June 11, 2021 from 0:00 UTC to 14:02 UTC, customers using strict TLS settings in their outbound mail servers were unable to send mail to Zendesk accounts on all Pods due to an expired TLS certificate in Zendesk’s email infrastructure.
Timeline
14:34 UTC | 07:34 PT
We’re happy to report that all known issues impacting email delivery for customers with strict TLS settings have been resolved. We will publish a post-mortem as soon as it’s completed.
14:15 UTC | 07:15 PT
We're seeing improvements in email delivery for customers with Strict TLS settings. We're continuing to monitor as we work to fully resolve the issue.
13:54 UTC | 06:54 PT
We have identified a Transport Layer Security (TLS) issue impacting email forwarding towards Zendesk. We will continue to update as we investigate further.
13:39 UTC | 06:39 PT
We have identified an issue impacting email delivery for some customers related to expired certificates. We will update with more information as it becomes available.
Root Cause Analysis
There was a breakdown in process during scheduled SSL certificate renewal that caused an internal registry to not be updated with the new key, even though it had been renewed.
Resolution
To fix this issue, our internal certificate registry was updated with the renewed SSL certificate which restored functionality to inbound email hosts.
Remediation Items
- Revisit certificate renewal process with additional testing and review.
- Update auditing of internal registries and process for certificate error alerting.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.