Managing search crawlers

What's my plan?

Search crawlers let you implement federated search in your help center without the developer resources required for the Federated Search API. You can set up multiple crawlers in your help center to crawl and index different content in the same or different websites.

This article describes how to review, edit, and delete search crawlers from the search crawler list in your help center search settings.

This article covers the following topics:

• Edit and review search crawlers
• Delete search crawlers

Edit and review search crawlers

You can review and edit your configured search crawlers to troubleshoot current errors, test your site map or domain, change current settings, or download the latest crawl status report csv file.

To review and edit your search crawlers

1. In Guide, click the Settings icon () in the sidebar, then click Search settings.
2. Under Crawlers, click Manage to view a list of your current search crawlers and their status.

   

3. Click the options menu () on the search crawler you want to view, then select Edit.

   

4. Review and edit any of the current search crawler settings. See Setting up the search crawler.
5. If necessary, use the edit crawler page to perform any of the following review and troubleshooting tasks:
   • Click Test site map to have the crawler evaluate the site map and determine whether it is valid. Site maps must follow the site maps XML protocol and contain a list of all pages within the site that you want to crawl.
   • Click Test domain to manually verify domain ownership. The crawler reviews the homepage of the site to determine whether a valid verification tag exists on the page.
   • Download a crawler status report (*.csv) to your computer. This report lists the following for each page that was crawled:
     • Page URL
     • Error description (if present)
     • Crawler name
     • Site map URL
     • Date and time crawler last ran
6. Click Save to save your changes and return to the search crawler list. The search crawler is now pending and will run within 24 hours using the new settings.

Delete search crawlers

You can delete existing search crawlers if you no longer want external content from that source included in your help center search results. Once you delete a search crawler, allow 24 hours for external content to be removed from help center searches.

Note: Once you delete a search crawler, it cannot be restored.

To delete your search crawlers

1. In Guide, click the Settings icon () in the sidebar, then click Search settings.
2. Under Crawlers, click Manage to view a list of your current search crawlers and their status.

   

3. Click the options menu () on the search crawler you want to view, then select Delete.

   

4. Review the delete confirmation message and select the check boxes to indicate you understand the consequences of deleting the crawler. When you are finished, click Delete crawler.