SEMrush bot blocked by cookiebot

Comments

5 comments

  • Hi Pepe,

    Cookiebot handles crawlers like GoogleBot as having given full consent, so you could follow the guide mentioned further down to change the User-Agent of SEMRush:
    https://www.semrush.com/kb/681-site-audit-troubleshooting#useragent 

    Otherwise, you could provide us with the useragent that SEMrush uses, so we can whitelist it in Cookiebot so that it will not block the crawler.

    Regards,
    Spas

    0
    Comment actions Permalink
  • Avatar
    Pepe Ortuno

    Hi Spas, 

    Thanks for your quick response. I've reviewed booth things mentioned in that help page and we have already the following items: 

    • Crawl source: set to sitemap
    • Semrush user agent: Googlebot-mobile  

    But it keeps crashing on crawl. 

    Anything more that we can test? 

    The bot's IP addresses are 46.229.173.67 and 46.229.173.66.

     

     

    0
    Comment actions Permalink
  • Hi Pepe,

    Cookiebot does not block any crawlers. The only thing that may happen when the crawler visits the site, is that some content may not be displayed, because of the used data-src attrbitue to implement prior consent. This exact same thing happens when new users, who have not given consent yet, visit a site running Cookiebot.
    So, actually the Cookiebot is not messing up with the site when SEMRush visists it. In order to treat SEMRushBot as a user who has given full consent, thus displaying all content, we will add SEMRushBot to our "white-list" of crawlers. Then the SEMRushBot should be able to crawl the site without any problems.

     

    Regards,
    Spas

    0
    Comment actions Permalink
  • Hi Pepe,

    The SEMRushBot now has been whitelisted, so everything should work just fine now.

    Regards,
    Spas

    0
    Comment actions Permalink
  • Avatar
    Pepe Ortuno

    Yes! It's working again. 

    Thank you very much Spas!!! 

     

    0
    Comment actions Permalink

Please sign in to leave a comment.