What's my plan?

Available on Suite Enterprise plans and above

Available on Guide Enterprise plans

The search crawler lets you implement federated search in your help center without developer resources. You can set up multiple crawlers in your help center to crawl and index different content in the same or different websites. See How the search crawler works.

Note: You are responsible for using the Help Center search crawler in compliance with all applicable laws and the terms and conditions of the relevant websites. You should only add Sitemaps where you own the domain associated with such Sitemaps. By using the Help Center search crawler, you confirm that you own the domains for all Sitemaps added to the crawler and that you have the right to crawl such websites.

To set up the search crawler

  1. In Guide, click the Settings icon () in the sidebar, then click Search settings.

  2. Under Crawlers, click Manage.

  3. Click Add Crawler.

  4. In Name this crawler, configure the following:
    • Name - Enter the name that you want to assign to the crawler. This is an internal name that identifies your search crawler on the crawler management list.
    • Owner - Enter the name of the Guide admin user responsible for crawler maintenance and troubleshooting. By default, the crawler owner is the user creating the crawler, however you can change this name to any Guide admin.

      Crawler owners receive email notifications both when the crawler runs successfully and when there are error notifications, such as problems with domain verification, processing the sitemap, or crawling pages. See Troubleshooting the Search Crawler.

  5. In Add the website you want to crawl, verify ownership of your domain by configuring the following:
    • Website URL - Enter the URL of the website that you want to crawl.
    • Domain ownership verification - Click Copy to copy the HTML tag onto your clipboard, then paste the tag into the <head> section in the HTML code of your site's non-authenticated home page. You can do this after you complete the crawler setup and you can always find the verification tag on the edit crawler page. See Managing search crawlers.
      Note: Do not remove the tag once it is in place, as the crawler needs to complete successful domain verification each time it runs.

  6. In Add a sitemap, in Sitemap URL, enter the URL for the sitemap you want the crawler to use when crawling your site. The sitemap must follow the sitemaps XML protocol and contain a list of all pages within the site that you want to crawl.
    The sitemap can be the standard sitemap containing all the pages of the site, or it can be a dedicated sitemap that lists the pages that you want the crawler to crawl. All sitemaps must be hosted on the domain that the crawler is configured to crawl. You can set up multiple crawlers on the same site that each use different sitemaps defining the pages you want the search crawler to crawl.
    Note: The search crawler does not support sitemap indexes. A sitemap is a file that lists the URLs of each page you want to index, and a sitemap index is a file that lists the URLs for individual sitemaps.

  7. In Add filters to help people find this content, configure the source and type filters used to filter search results by your end users. Source refers to the origin of the external content, such as a forum, issue tracker, or learning management system. Type refers to the kind of content, such as blog post, tech note, or bug report.
    • Source - Click the arrow, then select a source from the list or select + Create new source to add a name that describes where this content lives.
    • Type - Click the arrow, then select a type from the list or select + Create new type to add a name that describes what kind of content this is.
    Note: To edit or delete sources and types created during search crawler setup, see Managing search filters.

  8. Click Finish.
    The search crawler is created and pending. Within 24 hours the crawler will verify ownership of the domain and then fetch and parse the specified sitemap. Once the sitemap processing succeeds, the crawler begins to crawl the pages and index its content. If the crawler fails either during domain verification or while processing the sitemap, the crawler owner will receive an email notification with troubleshooting tips to help resolve the issue. The crawler will try again in 24 hours. See Troubleshooting the search crawler.
    Note: Zendesk/External-Content is the user agent for the search crawler. To prevent the crawler from failing due to a firewall blocking requests, whitelist (or allow-list) Zendesk/External-Content.
  9. Set up your help center theme for federated search.

    For external content to show up in search results in your help center search, you must have a theme that supports federated search results. To do this, use the latest version of the Copenhagen theme or replace the old {{help_center_filters}} and {{filters}} helpers with the new {{source_filters}} and {{type_filters}} helpers (see the Help Center templating cookbook).

  10. Select the content that you want to include and exclude in your help center search results. See Including external content in your help center search results.

  11. If desired, configure Knowledge results to include external content in Knowledge searches. See Configuring Knowledge in the context panel.

Powered by Zendesk