Cloudflare CEO Warns AI Crawlers and Summaries Are Eroding the Internet's Business Model

Arthur T Knackerbracket has processed the following story:

Cloudflare CEO Matthew Prince recently reiterated his warning that generative AI crawlers and summaries threaten the foundations of the internet's business model. To protect publishers from a flood of artificial AI traffic that offers virtually no authentic site visits in return, the company is devising methods to combat AI scrapers.

Speaking at an Axios event in Cannes last week, Prince explained that search engines and chatbots using generative AI to summarize web content have significantly reduced the number of human visitors to many websites. Even compared to six months ago, the problem has worsened considerably.

Traditionally, for every six times Google crawled a website, one person might visit and potentially view ads. In contrast, the rate was about 250 to 1 with OpenAI's crawlers and 6,000 to 1 with Anthropic. Today, Cloudflare's CEO estimates that Google's crawl-to-visitor rate has declined to 18 to 1, OpenAI's has worsened to 1,500 to 1, and Anthropic's is approximately 60,000 to 1.

This decline is likely because chatbots and search engines now save users the effort of visiting websites. Chatbots can retrieve and summarize information without the user needing to leave the chat interface, and AI overviews from major search engines now offer users answers before they click on search results.

When Google first unveiled AI overviews, it claimed that the technology would boost traffic to the original sources of the summarized content. Similarly, large language models such as ChatGPT have recently started citing sources in their responses to help direct traffic back to content creators.

However, Prince claims that users, by and large, aren't clicking on the footnotes. Instead, many many are accepting the AI's responses at face value, as trust in the technology has grown over the past six months. Aside from starving websites of traffic and revenue, the trend is potentially dangerous due to AI's known tendency to generate inaccurate or misleading information.

Read more of this story at SoylentNews.