Skip to main content

Cookiebot Crawler

Answered

Comments

11 comments

  • Martin - vConnect - Community Supporter

    Hey Eric,

    In regards to your regex, it seems fine and indeed it does return the IP's of cookiebot as a result, but can't tell exactly how you have implemented this.

    You can check this article about unusual traffic on your website: https://support.cookiebot.com/hc/en-us/articles/360005083674-I-see-unusual-spikes-in-the-traffic-on-my-website-is-it-caused-by-your-scanner-   and how to prevent it.

    Regards,
    Martin

     

    0
  • Eric Weisbrot

    Hello,

     

    Thanks for the response.

     

    Yes, those are the instructions I followed for filtering your IP's from Google Analytics. Please see the screenshot attached of the filter I have setup using the regex mentioned above in this ticket. Does this look right to you?

    0
  • Eric Weisbrot

    Hello,

    I'm just following up on my last post. I'd like to get the GA filter working so the crawler doesn't continue to mess up our web data next time Cookiebot crawls our site.

    Thank you.

    0
  • Martin - vConnect - Community Supporter

    Hi Eric,

    If the website is blocking all of Cookiebot public IP-numbers, the scanner will use a rotating set of IP-numbers for the crawl. These IP numbers are changing all the time and it is therefor not possible to filter on in Analytics.

    Regards,
    Martin

    0
  • Eric Weisbrot

    Hey Martin,

    Thanks for the response.

    Just to ensure I'm understanding you, are you saying that there is no way to create a GA filter and filter out the Cookiebot crawler traffic from our analytics reports? The article you provided suggests exactly that here: https://support.cookiebot.com/hc/en-us/articles/360005083674-I-see-unusual-spikes-in-the-traffic-on-my-website-is-it-caused-by-your-scanner-%C2%A0%C2%A0

    The other alternative mentions filtering out the provider of "microsoft corporation", but wouldn't doing so filter out a large portion of actual users from our analytics reports?

     

    0
  • Martin - vConnect - Community Supporter

    Hi Eric,

    The GA filter seems fine and you can use it to filter and whitelist the public Cookiebot IP's.

    The problem in this case is most likely that you have blocked in someway these public IP's somewhere and Cookiebot uses rotating IP's in order to scan the site. These IP's are constantly changing, so you won't be able to catch them and filter them.

    If you are using some kind of CDN or DDOS protection then you must whitelist the Cookiebot IP's. Most likely you will need to contact your hosting provider and ask them in that case.

    Regards,
    Martin

    0
  • Eric Weisbrot

    Thanks for the info.

     

    We use the CDN Cloud Flare. If we white list the Cookiebot IP's there, will this fix the issue of the crawler messing up our web data? Please let me know which IP's to white list. 

    Thank you. 

    0
  • Martin - vConnect - Community Supporter

    Hi Eric,

    The Cookiebot IP's you can find out here: https://support.cookiebot.com/hc/en-us/articles/360003824153-Whitelist-what-IP-addresses-do-you-scan-from-

    In theory this should fix your issue, yes. If not than something else is blocking these IP's and you will need to do further investigation on it.

    Regards,
    Martin

    0
  • Eric Weisbrot

    Hey Martin,

    Just ensure we're on the same page, why would whitelisting the crawler IP's fix the issue of the crawler causing our analytics data to be inaccurate?

    0
  • Martin - vConnect - Community Supporter

    Hi Eric,

    You need to whitelist the IP's of Cookiebot in order to certainly know that Cookiebot will use these IP's to scan your site. In that way you can filter these IP's, so they don't mess up your GA data of your website. If they are not whitelisted, Cookiebot will use rotating IP's, you won't be able to filter them and the data will still not be accurate.

    Regards,
    Martin

    0
  • Eric Weisbrot

    That makes sense. I'll do that now. 

    0

Please sign in to leave a comment.