SEMRUSH crawl returns 403 errors on Zendesk links

12 Comments

  • Eric Nelson
    Zendesk Developer Advocacy

    Hey Roberto Reale, this is actually related to a bug that we're tracking. Do you mind if I move this into a ticket?

    0
  • Thrivous

    Same problem here. How do we fix it?

    1
  • Roberto Reale

    Hello Eric Nelson! Of course, please go ahead and if possible let me know when there is a fix available. We currently have a ton of errors just for that.

    Thanks!

    1
  • Kirsten Flores

    Eric Nelson - Hi there. Has there been any progress on this? I believe we are experiencing the same issue. 

    1
  • Wesley Brooks

    Hi,

     

    Did you get a fix for this?

     

    Thanks

    Wes

    1
  • Sam Nalintakath

    We have the exact issue.

    Our Zendesk HC page also gets 403 error in semrush. But page loads fine when accessed.

    Please let us know if there is a fix .

     

    We got following info from Semrush too.

     

    Thank you for contacting Semrush! 

    We apologize for the delay as we have been experiencing high request volumes. 

    We are getting a 403 status error based on their domain blocking our bots at this time. If our bots are blocked it will cause them to time out. This does not directly mean there is an error with the page, thusly why you may be seeing a 200 code on your end. 

    Additionally information on whitelisting can be found within our Site Audit section of our knowledge base: 
    https://www.semrush.com/kb/681-site-audit-troubleshooting. To recap how the domain reads or blocks our bot is why we specifically are retaining a 403 status code at this time. 

    Please let us know if you have any additional questions or requests! 

    2
  • Taisia Auston

    Hello, we are having this exact same issue too. Can I get this created into a ticket as well?

    2
  • Vincent

    Confirming this issue as well. This is not specific to semrush however, our google search console report also reports this which leads to impact on SEO. 

    2
  • Kurt Uhlir

    We're getting the same error. Any progress on a fix? I see at least from this thread that it was first reported on December 1, 2021 and today is May 20, 2022. 

    1
  • Yujian Weng

    Same issue here, any updates on the fix?

    Thanks

    2
  • Ethan Martin

    Hi Eric Nelson, I wanted to also request an update here. This bug is preventing us from properly evaluating our site's health. Thanks in advance!

    1
  • Greg Katechis
    Zendesk Developer Advocacy

    Hi all, I want to jump in here to give an update and some more context for what is happening here. 

    At Zendesk, we use an edge layer in our network infrastructure to ensure that the requests that are being made are not malicious or otherwise potentially damaging. One of the many ways that we handle this is by determining the likelihood that the requesting is coming from a bot. If the determination is made that it's likely to be a bot, one option for us is to present a captcha. If it's an actual user, they can complete the captcha and successfully return the resource. If it isn't or if the captcha fails, we will return a 403.

    With Semrush, what I described here is very likely what is happening. Our devs are almost finished with the documentation that we'll be sharing that will explain what I'll share with you now, but before that's ready to go, I can give you the general overview.

    In a situation like this, there are two paths that can be taken, one from our side and one from the bot provider's side, in this case Semrush. Any actions that we take on our side have the potential to make our overall infrastructure less safe, so we first have to have the bot provider take action. That first step will be for them to submit an application to become a verified bot with Cloudflare and they can do that by following the instructions here. If you provide that link to them, that should be able to pass that to the right people on their side who can fill that out. If their application is approved, the issue should be resolved!

    If that is not successful, there will be additional steps that Semrush will need to take that we will have outlined in that article once it is available. It will be asking them to consider which endpoints they're using when crawling, as well as some recommendations for specific APIs depending on their use-case. Since that's not quite ready yet and the application will be the next step for Semrush, I'll wait for the article to be published that before I go into that with any more detail.

    I know that it's difficult to be told to go from one company to the other, so I apologize for making that your first step. The good news is that if that is successful, not only will this work for all of you, it will also help ensure that Zendesk's network is as safe as it can be.

    As soon as that article is published, I'll make sure to drop an update here. If anyone receives word back from Semrush that I can help with, let me know and I'll be glad to assist!

    0

Please sign in to leave a comment.

Powered by Zendesk