Cloudflare and GoDaddy Team Up to Curb AI Bot Brigades

Arthur T Knackerbracket writes:

Pair backs scraper blocking and standards to separate trusted agents from bad bots

Citing the need to adapt to an internet increasingly serving the needs of AI agents without considering the needs of site owners, Cloudflare and GoDaddy are partnering on efforts to control how AIs crawl the web and interact with web content.

The content delivery network and the web host on Tuesday announced that they would help website owners gain better control over their relationship with AI, primarily through GoDaddy integrating Cloudflare's AI Crawl Control utility into its platform. That tool, as the pair explained in a press release, lets owners manage how AI interacts with their websites, allowing, blocking, or requiring payment from crawlers for access.

"By putting tools like AI Crawl Control and open standards into the hands of website owners, we are providing essential underpinnings for a new Internet business model," Cloudflare chief strategy officer Stephanie Cohen said of the move. "We want to ensure that every creator has the tools to verify who is interacting with their site, while giving legitimate AI agents a secure, transparent way to participate."

Cloudflare has been beating the drum over the need to control bots' access to websites and web content, and has rolled out several measures aimed at restricting unauthorized scraping in recent years. In 2025, it rolled out an AI that it said would trap and waste the time of unauthorized AI scrapers by feeding them endless garbage, and it has previously pushed to require AI bots to pay for access to websites.

Charging bots was one of Cloudflare's ideas to help protect website operators, who the CDN has noted are losing tons of money earned via web traffic since many search visitors are now instead fed the answers they're looking for by an AI, like Google's AI Overviews. Therefore, visitors are often less likely to click through to the original source.

The pair have skin in this game, clearly, as without website owners making money, they're unlikely to get paid themselves.

Read more of this story at SoylentNews.