Cloudflare lets customers block AI bots, scrapers and crawlers with a single click
It seems the dislike for machine learning runs deep. In a blog post, Cloudflare has announced that blocking machine learning scrapers is so popular, they decided to just add a feature to the Cloudflare dashboard that will block all machine learning scrapers with a single click.
We hear clearly that customers don't want AI bots visiting their websites, and especially those that do so dishonestly. To help, we've added a brand new one-click to block all AI bots. It's available for all customers, including those on the free tier. To enable it, simply navigate to the Security > Bots section of the Cloudflare dashboard, and click the toggle labeled AI Scrapers and Crawlers.
Cloudflare blog
According to Cloudflare, 85% of their customers block machine learning scrapers from taking content from their websites, and that number definitely does not surprise me. People clearly understand that multibillion dollar megacorporations freely scraping every piece of content on the web for their own further obscene enrichment while giving nothing back - in fact, while charging us for it - is inherently wrong, and as such, they choose to block them from doing so.
Of course, it makes sense for Cloudflare to try and combat junk traffic, so this is one of those cases where the corporate interests of Cloudflare actually line up with the personal interests of its customers, so making blocking machine learning scrapers as easy as possible benefits both parties. I think OSNews, too, makes use of Cloudflare, so I'm definitely going to ask OSNews' owner to hit that button.
Cloudflare further details that a lot of people are blocking crawlers run by companies like Amazon, Google, and OpenAI, but completely miss far more active crawlers like those run by the Chinese company ByteDance, probably because those companies don't dominate the AI" news cycle. Then there's the massive number of machine learning crawlers that just straight-up lie about their intentions, trying to hide the fact they're machine learning bots.
We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection. We will continue to keep watch and add more bot blocks to our AI Scrapers and Crawlers rule and evolve our machine learning models to help keep the Internet a place where content creators can thrive and keep full control over which models their content is used to train or run inference on.
Cloudflare blog
I find this particularly funny because what's happening here is machine learning models being used to block... Machine learning models. Give it a few more years down the trajectory we're currently on, and the internet will just be bots reading content posted by other bots.