UserAgent - Need to know its CookieBot to avoid Blocking unknown ScannerAnswered
Our sites auto protect from "rogue" web site crawlers who aggressively crawl our sites; expect for those with known IP ranges and known user-agents (and sometimes a combo of the two e.g. Googlebot is allowed when from the correct IP Ranges)
Hence when a Scan of cookies is done; a HTTP Status Code 429 is returned some of the time as we throttle the access as too many pages have been requested in a short time - i.e. we know its a bot and purposefully slow it down.
I can see CookieBot is using Microsoft as hosting platform; i.e. examples from
The issue is that the User-agent supplied i.e. "Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.117+Safari/537.36"
Doesn't identify CookieBot.
PLEASE.... Can you
a) Add to the User-Agent the phrase "CookieBot"
b) Allow us per web site/account in Cookiebot to define the user-agent that is used
Either way we then known "who" the scanner is and then can "un-throttle it"
I've posted this under Bugs and Errors - as the lack of User Agent is causing in correct scan results
Right now, the Cookiebot is using a specific IP addresses to scan the websites, which you could find here:
So, you can use those public IPs to check whether the Cookiebot is scanning your website or not.
In the future it will have an opportunity to add User Agent that is used, unfortunately for now you can use only the public IPs that are mentioned at the link above.
Best regards, Spas0
Is there an update on the support for identification through user agent string like suggest where "cookiebot" is added to the one used.?
Yes. Since the last update, you could now identify Cookiebot through the user agent string and it will contain "Cookiebot".
Please sign in to leave a comment.