SUMMARY

From February 20, 2024 around 18:00 UTC to April 3, 2024 12:00 UTC Guide customers across all Pods experienced an issue where multiple search engines were displaying the incorrect favicon (the small icon associated with a website, typically displayed in browser tabs) for their Zendesk Guide Help Centers. Customers using various search engines, such as Google and DuckDuckGo, noticed that the favicon associated with their Help Center was incorrect, while others like Bing and Yandex displayed the correct one. 

 

Timeline

March 12 - 14:46 UTC | 07:46 PT
We are aware of the unexpected favicon display issues within various search engine results for multiple customers’ Help Centers. Our engineering teams have been investigating and have not found any code change within our products to explain the behavior. We are continuing to investigate any changes to how favicons are displayed within these search engines and have dedicated teams working alongside our external partners to resolve the issue as quickly as possible. Should you have any questions or additional information that might aid our efforts, please do not hesitate to reach out to us. We will share further updates when available and thank you for your patience and understanding.

March 14 - 11:38 UTC | 04:38 PT
At this time, we have no significant updates to share regarding the unexpected favicon display issues affecting various search engine results for multiple customers’ Help Centers. We sincerely appreciate those who have contributed troubleshooting tips and insights; however, we have not yet been able to fully narrow down the root cause for it. Please know that our teams remain deeply committed to resolving this matter, and we are exploring all avenues to find a solution. We will provide new updates as soon as we have more details. Thank you for your continued patience and support.

March 19 - 14:21 UTC | 07:21 PT
Our team has been working tirelessly to identify a potential fix for the ongoing favicon display issues that have been affecting various search engine results for multiple customers’ Help Centers. We are currently finalizing the details of a possible fix and, pending a thorough review, we aim to deploy it by tomorrow. While we continue testing its effectiveness, we remain cautiously optimistic. We will keep you informed of any new developments. We appreciate your continued patience and cooperation as we diligently work to resolve this issue.

March 21 - 11:41 UTC | 04:41 PT
Following yesterday’s deployment of the proposed fix to a small number of customers for testing purposes, we have observed positive results. Encouraged by this outcome, we are now proceeding with the deployment of the fix across the remaining pods. We ask for your continued patience as we work to implement this solution. Please stay tuned for further updates, and thank you once again for bearing with us during this process.

March 21 - 13:26 UTC | 06:26 PT
We can confirm that the fix has been implemented across all accounts. To expedite the update of search engine results, we kindly request that you initiate a reindexing of your URLs with Google. You can find the necessary steps to request this at the following link: https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl. Your collaboration on this step is crucial and greatly appreciated, and we thank you for your continued support as we finalize the resolution of this issue.

March 25 - 13:38 UTC | 06:38 PT
Based on our previous update last week, a fix for the favicon display issues that have been affecting various search engine results for multiple customers’ Help Centers has been implemented across all accounts. To expedite the update of search engine results, we requested customers to initiate a reindexing of their URLs with Google—and with any other search engines, where possible. However, we would like to note that this process can take some time to yield results. As we gather confirmations from customers, we remain thankful for your patience throughout.

April 2 - 12:58 UTC | 05:58 PT
As we continue to monitor the Help Center favicon issue across multiple search engines, we’ve observed Googlebot-Image/1.0 accessing the correct favicon path and substantial traffic for favicon files since our last fix was implemented, despite not being able to filter favicon-specific traffic from other theme assets. We kindly request and recommend you ensure your favicon also adheres to Google’s guidelines (https://developers.google.com/search/docs/appearance/favicon-in-search#guidelines) on format and size. Thanks for your ongoing patience.

April 5 - 11:47 UTC | 04:47 PT
As more customer confirmations come in regarding the appearance of their correct favicons in search engine results, we are confident that our implemented fix, combined with the reindexing requests to search engines (which may take additional time to process), and ensuring compliance with the recommended favicon size in our Help Center branding guidelines (https://support.zendesk.com/hc/en-us/articles/4408824139546-Branding-your-help-center), are effectively leading to the resolution of this incident. In light of these developments, we are marking this incident as stable and will not be providing further updates apart from the post-mortem in the future. We appreciate your cooperation, patience, and understanding throughout the resolution of this issue.

 

POST-MORTEM

Root Cause Analysis

This incident was caused primarily because the incorrect favicon display was identified as a violation of the guideline that states there should be "one favicon per hostname". For instance, a Zendesk Help Center could’ve had two different public favicons accessible under the same hostname: one for the Help Center pages and another for the authentication pages. This inconsistency led to confusion when search engine crawlers indexed the site, resulting in the wrong favicon being displayed in search results.

Resolution

To fix this issue, the following actions were taken:

  • We've updated the /favicon.ico route to point to the favicon set in the Help Center live theme of the corresponding subdomain.
  • If a subdomain does not have a Help Center, the /favicon.ico route returns the default Zendesk Support icon.
  • We've added caching to the /favicon.ico route to optimize performance.

Remediation Items

  1. Customer Action: 
    1. Customers are advised to request a re-index of their Help Center with Google to ensure that the correct favicon is displayed in search results.
    2. Customers should be aware of the importance of having a consistent favicon setup and how to properly request re-indexing with search engines as per the multiple guidelines shared throughout this incident (linked articles).
  2. Zendesk Action: 
    1. Monitoring Changes: Zendesk will continue to monitor any changes made by search engines that could impact favicon indexing and display.
    2. Long-term Review: Zendesk will conduct a long-term review of the favicon setup and indexing process to understand why a spike in reports occurred and to prevent future incidents.

Additional Notes

The time it takes for search engines to re-index favicons is uncertain, and it may vary from a few weeks to several months. This factor contributed to the delay in achieving a stable resolution for all affected customers. Customers are encouraged to maintain patience during this period and to reach out to Zendesk support if they have any further issues or concerns.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

Powered by Zendesk