Managing web crawlers

What's my plan?

All Suites

Team, Growth, Professional, Enterprise, or Enterprise Plus

Support with

Guide Professional or Enterprise

Web crawlers let you implement federated search in your help center without the developer resources required for the Federated Search API. You can set up multiple crawlers in your help center to crawl and index different content in the same or different websites.

This article describes how to review, edit, and delete web crawlers from the web crawler list in your help center search settings.

This article covers the following topics:

• Edit and review web crawlers
• Delete web crawlers

Edit and review web crawlers

You can review and edit your configured web crawlers to troubleshoot current errors, test your site map or domain, change current settings, or download the latest crawl status report csv file.

To review and edit your web crawlers

1. In Knowledge admin, click Settings () in the sidebar.
2. Click Search settings.
3. Under Crawlers, click Manage to view a list of your current web crawlers and their status.

   

4. Click the options menu () on the web crawler you want to view, then select Edit.

   

5. Review and edit any of the current web crawler settings. See Setting up the web crawler.
6. If necessary, use the edit crawler page to perform any of the following review and troubleshooting tasks:
   • Click Test site map to have the crawler evaluate the site map and determine whether it is valid. Site maps must follow the site maps XML protocol and contain a list of all pages within the site that you want to crawl.
   • Download a crawler status report (*.csv) to your computer. This report lists the following for each page that was crawled:
     • Page URL
     • Error description (if present)
     • Crawler name
     • Site map URL
     • Date and time crawler last ran
7. Click Save to save your changes and return to the web crawler list. The web crawler is now pending and will run within 24 hours using the new settings.

Delete web crawlers

You can delete existing web crawlers if you no longer want external content from that source included in your help center search results. Once you delete a web crawler, allow 24 hours for external content to be removed from help center searches.

Note: Once you delete a web crawler, it cannot be restored.

To delete your web crawlers

1. In Knowledge admin, click Settings () in the sidebar.
2. Click Search settings.
3. Under Crawlers, click Manage to view a list of your current web crawlers and their status.

   

4. Click the options menu () on the web crawler you want to view, then select Delete.

   

5. Review the delete confirmation message and select the check boxes to indicate you understand the consequences of deleting the crawler. When you are finished, click Delete crawler.