Recent searches

No recent searches

Disallow a directory in robots.txt

Answered
image avatar

Felix

Posted Oct 24, 2022

Hi,

Is there a way to access robots.txt to be able to disallow a specific directory?

for example /cdn-cgi/

If it's not possible to disallow a specific directory, is it possible to disallow the entire support site?

note:
adding noindex won't solve my issue, as google will still crawl the pages I want to block

Thanks

0

3

3 comments

Date
    image avatar

    Gabriel Manlapig

    Zendesk Customer Care

    Hi Felix,
     
    The robots.txt file is a shared file used by all Zendesk accounts. It is not account-specific and cannot be edited. Currently, the only option to block a page from being indexed is to add a meta tag to your Help Center.

    <meta name="robots" content="noindex" />

    Or, to temporarily exclude a specific page from being indexed by Google, see How do I exclude a Help Center page from being indexed by Google?

    I hope this answers your question.

    0

    image avatar

    JL

    is there any meta tag we can add to document_head.hbs to prevent chatgpt? We're trying to set the following.

    User-agent: GPTBot
Disallow: /

    0

    image avatar

    Tipene Hughes

    Zendesk Developer Advocacy

    Hi John,

    As mentioned by Gabriel above, the robots.txt file is not able to be edited. Unfortunately, it doesn't look like OpenAI provide any other method to prevent their GPTBot from crawling pages, at least as far as I can see in their docs. You could try using meta tags in the document head, but I'm not entirely sure if that will be effective.

    0

    Please sign in to leave a comment.

    Didn't find what you're looking for?

    New post