SUMMARY
On July 3rd 2021 at 11:45 UTC certificates for *.zdusercontent.com and *.apps.zdusercontent.com expired which resulted in Zendesk Apps and Ticket Attachments not loading. *.zdusercontent.com and *.apps.zdusercontent.com certificates were updated at 14:20 UTC and 14:40 UTC respectively and recovered Ticket Attachments and Zendesk Apps delivery.
Timeline
15:02 UTC | 08:02 PT
We’re happy to report the issue affecting attachments and apps has now been resolved. Thank you for your patience.
14:53 UTC | 07:53 PT
We are seeing improvements in being able to access both attachments and apps and are seeing normal service return. We are continuing to monitor.
14:28 UTC | 07:28 PT
We are seeing improvements with being able to access attachments, we are still working to resolve the issue affecting apps. We will continue to provide updates
14:06 UTC | 07:06 PT
We’re continuing to work on remediations for the incident affecting ticket attachments and apps, we will continue to provide updates as we find out more.
13:42 UTC | 06:42 PT
We believe we have discovered the root cause of the incident affecting apps, and this will also affect ticket attachments. We are actively working on the issue and will post another update when we have more information.
13:18 UTC | 06:18 PT
Zendesk apps are currently not loading on some accounts. We are investigating the issue
Root Cause Analysis
This incident was caused by an internal certificate expiration.
Resolution
To fix this issue, our certificates were updated at the CDN vendor.
Remediation Items
- Add SSL tests to proactively monitor the validity and expiration of all Zendesk SSL/TLS certificates.
- Automate all manual certificate renewal steps.
- Improve Zendesk monitoring and alerting so we can identify the issues and pinpoint the root cause faster.
- Review our Incident Management Severity levels to ensure the right level of response for impact to our customers.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.