Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare just set a new deadline for the AI ​​industry to separate the web crawlers used for traditional search purposes, such as Google Search, from the web crawlers used for AI agents and training. Starting September 15, 2026, Cloudflare’s default settings will block mixed-use crawlers from all pages that host ads, the company announced Wednesday.

This means that the crawlers that combine search, agent usage, and training are blocked from crawling these sites by default unless the site owner adjusts the settings otherwise. These changes to the default settings will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says.

This move could impact how AI model providers access web content for training purposes and to support their agent services.

Cloudflare points out that most website owners want their content to be discoverable through searches and often through AI services, but they want protection from giving away their intellectual property for free.

Cloudflare specifically mentions that the “world’s largest search engine” (obviously a Google reference!) has access to about “2x more information” than other AI companies, as the search giant makes it difficult for customers to stay discoverable without being used for AI.

Google has pushed back against this generalization in the past, noting that it offers a bot called Google Extended which allows site owners to opt out of having their content used for training and AI products and services such as Gemini Apps and Vertex API. Using them does not affect a site’s inclusion in Google Search. However, the tech giant’s flagship Googlebot does crawling for Search, including AI features such as AI summaries and AI mode.

“With the majority of traffic on the internet being non-human, we must go further and act faster to create a sustainable ecosystem,” said Matthew Prince, co-founder and CEO of Cloudflare, in announcing the news, referring to the recent milestone where bots surpassed human traffic online for the first time. That shift would not happen until next year.

“Cloudflare’s new tools and partnerships give website owners greater visibility and commercial opportunities and benefit AI companies that have bots with a clear and transparent intent. We hope our proposed standard changes encourage mixed-use crawlers to separate search from agent usage and training,” Prince said.

While Cloudflare offers a number of products to help users launch their own AI systemsthe company has also released a suite of tools to give publishers more control over their content in the AI ​​era. In recent years, Cloudflare has launched tools to combat AI bots, including a marketplace where websites can charge AI bots for scraping, called Pay Per Crawl.

The latter is now also evolving to ‘Pay Per Use’, the company said, allowing publishers to charge AI companies when their content creates value, and not just when it is retrieved.

The change could also help preserve publisher bandwidth and computing resources for AI model providers, as Cloudflare’s data suggests that more than 50% of crawl traffic from AI crawlers is spent refetching unaltered pages.

To put this into practice, Cloudflare will initially work with two partners, Ceramic.ai and You.com. When a publisher signs up, they get paid when their content appears in Ceramic’s AI search results or when You.com gains access to some of its premium content.

Other AI companies can adapt this model to the way they work, Cloudflare says.