Does anyone have any experience of slowing down or blocking site scrapers? We've a problem with competitors tracking our price changes far too quickly. It looks like they are using cloud services and some specific market strategy companies such as Semrush. We don't want to make the site awkward to use by using by introducing capcha's etc.
-
-
Cloudflare with some customization can be used for this. Semrush I tend to recommend blocking outright since they're almost never legit, they don't abide by robots directives, and they'll hammer a site endlessly. There are a few other problematic web hosts where this type of traffic originates from, so putting a javascript challenge on them tends to also knock out a bunch of it. Next, you can use the rate limiting rules; Cloudflare does charge for them, but it's not a lot, five cents per 10,000 matching rate limited requests; i.e. you're not paying until something triggers the rule and is rate limited. Unlike the intrusive captcha, which can also be used, their javascript challenge only appears as a momentary screen to a legit shopper and then they're able to browse the site normally for 30 minutes (default duration), and you only need to deploy the challenge against certain targets. -
Thanks David, I'll get on it now...Comment
-
Happy to help once you've got the account active; you can invite me to it, or any of our staff in support, and we can put our best practice rules in place, plus tweak for your specific needs.Comment
-
Thanks David, that'll be very helpful. I've a chat scheduled with them tomorrow afternoon.Comment
Comment