Disallow a directory in robots.txt

已回答
image avatar

Felix

已于 2022年10月24日 发布

Hi,

Is there a way to access robots.txt to be able to disallow a specific directory?

for example /cdn-cgi/

If it's not possible to disallow a specific directory, is it possible to disallow the entire support site?

note:
adding noindex won't solve my issue, as google will still crawl the pages I want to block

Thanks

0

3

3 条评论

日期
    最新
      image avatar

      Tipene Hughes

      Hi John,

      As mentioned by Gabriel above, the robots.txt file is not able to be edited. Unfortunately, it doesn't look like OpenAI provide any other method to prevent their GPTBot from crawling pages, at least as far as I can see in their docs. You could try using meta tags in the document head, but I'm not entirely sure if that will be effective.

      0

      image avatar

      JL

      is there any meta tag we can add to document_head.hbs to prevent chatgpt? We're trying to set the following.

      User-agent: GPTBot
Disallow: /

      0

      image avatar

      Gabriel Manlapig

      Hi Felix,
       
      The robots.txt file is a shared file used by all Zendesk accounts. It is not account-specific and cannot be edited. Currently, the only option to block a page from being indexed is to add a meta tag to your Help Center.

      <meta name="robots" content="noindex" />

      Or, to temporarily exclude a specific page from being indexed by Google, see How do I exclude a Help Center page from being indexed by Google?

      I hope this answers your question.

      0

      登录再写评论。

      找不到所需的内容？

      新建帖子