UserAgent - Need to know its CookieBot to avoid Blocking unknown Scanner

Answered

Stuart Jenkins

• June 24, 2018 07:03

Hi

Our sites auto protect from "rogue" web site crawlers who aggressively crawl our sites; expect for those with known IP ranges and known user-agents (and sometimes a combo of the two e.g. Googlebot is allowed when from the correct IP Ranges)

Hence when a Scan of  cookies is done; a HTTP Status Code 429 is returned some of the time as we throttle the access as too many pages have been requested in a short time - i.e. we know its a bot and purposefully slow it down. 

I can see CookieBot is using Microsoft as hosting platform; i.e. examples from 

https://www.whois.com/whois/13.74.44.241

https://www.whois.com/whois/40.91.211.73

The issue is that the User-agent supplied i.e. "Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.117+Safari/537.36"

Doesn't identify CookieBot. 

PLEASE.... Can you

1. Either 

a) Add to the User-Agent the phrase "CookieBot"

or (better)

b) Allow us per web site/account in Cookiebot to define the user-agent that is used 

Either way we then known "who" the scanner is and then can "un-throttle it"

I've posted this under Bugs and Errors - as the lack of User Agent is causing in correct scan results

Thanks

 

 

 

 

 

 

0

• 

  Spas – vConnect – Community Supporter

  • June 25, 2018 07:47

  Hi Stuart,
  

  Right now, the Cookiebot is using a specific IP addresses to scan the websites, which you could find here:
  https://support.cookiebot.com/hc/en-us/articles/360003824153-Whitelist-what-IP-addresses-do-you-scan-from-

  So, you can use those public IPs to check whether the Cookiebot is scanning your website or not.
  In the future it will have an opportunity to add User Agent that is used, unfortunately for now you can use only the public IPs that are mentioned at the link above.
  

  Best regards, Spas

  0
• 

  Tom Jensen

  • May 25, 2020 08:01

  Hi

  Is there an update on the support for identification through user agent string like suggest where "cookiebot" is added to the one used.?

   

  Br.

  Tom

  0
• 

  Spas – vConnect – Community Supporter

  • June 03, 2020 07:12

  Hi Tom, 

  Yes. Since the last update, you could now identify Cookiebot through the user agent string and it will contain "Cookiebot".

  Regards,
  Spas

  0

Please sign in to leave a comment.