SEMrush bot blocked by cookiebot
Hi, we use Semrush to track site performance and since we installed cookiebot, semrush is not able to crawl our site. It seems it can't pass thorught cookie alert (set to popup alert).
How can we set semrush's bot to crawl our site again?
-
Hi Pepe,
Cookiebot handles crawlers like GoogleBot as having given full consent, so you could follow the guide mentioned further down to change the User-Agent of SEMRush:
https://www.semrush.com/kb/681-site-audit-troubleshooting#useragentOtherwise, you could provide us with the useragent that SEMrush uses, so we can whitelist it in Cookiebot so that it will not block the crawler.
Regards,
Spas0 -
Hi Spas,
Thanks for your quick response. I've reviewed booth things mentioned in that help page and we have already the following items:
- Crawl source: set to sitemap
- Semrush user agent: Googlebot-mobile
But it keeps crashing on crawl.
Anything more that we can test?
The bot's IP addresses are 46.229.173.67 and 46.229.173.66.
0 -
Hi Pepe,
Cookiebot does not block any crawlers. The only thing that may happen when the crawler visits the site, is that some content may not be displayed, because of the used data-src attrbitue to implement prior consent. This exact same thing happens when new users, who have not given consent yet, visit a site running Cookiebot.
So, actually the Cookiebot is not messing up with the site when SEMRush visists it. In order to treat SEMRushBot as a user who has given full consent, thus displaying all content, we will add SEMRushBot to our "white-list" of crawlers. Then the SEMRushBot should be able to crawl the site without any problems.Regards,
Spas0 -
Hi Pepe,
The SEMRushBot now has been whitelisted, so everything should work just fine now.
Regards,
Spas0 -
Yes! It's working again.
Thank you very much Spas!!!
0
Please sign in to leave a comment.
Comments
5 comments