Service Incident - November 9, 2022 - Guide | Multiple Pods - Help Center returning unformatted data

SUMMARY

Between November 09, 2022 from 01:20 UTC to November 10, 2022 16:07 UTC, Zendesk Guide customers across multiple pods experienced their Help Center returning unformatted data.

Timeline

15:30 UTC | 07:30 PT

We’re currently investigating reports across multiple Pods of Help Center returning unformatted data. As a workaround, clearing cache & cookies and hard-refreshing browsers should resolve the issue while we continue to work on a permanent fix. Next update in 2h.

17:28 UTC | 09:28 PT

We have implemented a potential fix in response to reports across multiple Pods of Help Center returning unformatted content. Please let us know if you continue to experience issues after clearing cache & cookies and hard-refreshing your browser.

18:59 UTC | 10:59 PT

Our attempted fix did not have the full desired effect, and our team is still investigating ongoing intermittent issues loading Help Center assets and pages. We will keep providing updates current with our investigation. Please let us know if you are still experiencing any issues

Nov 10 - 12:25 UTC | 04:25 PT

We appreciate your patience while we continue working on the formatting issue affecting loading Guide Help Center pages for customers across multiple Pod’s. Please make sure to update your browser, as well as clearing cache&cookies.

Nov 10 - 15:48 UTC | 07:48 PT

Due to the intermittency of this issue regarding assets not loading properly on Guide Help Center pages for customers across multiple Pods, we carry on with the investigation and work to fully fix it. In the meantime, the workaround provided remains helpful to most customers.

Nov 11 - 10:20 UTC | 02:20 PT

We are happy to confirm that our latest implemented fix for the issues with assets not loading properly in Guide Help Center pages is working as expected. We truly appreciated your patience while we worked through this.

POST-MORTEM

Root Cause Analysis

Our CDN provider migrated Zendesk to a new version of their SSL for SaaS product, which resulted in a change in the behaviour of how requests should be formed, a scenario that had not previously been encountered.

Resolution

To fix this issue, we made a change to our CDN provider’s certificate configuration settings to revert to the previous behaviour for certificate validation. 

Remediation Items

Zendesk has assessed that no further remediation items are needed for this scenario.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.