Cloudflare launches a marketplace that lets websites charge AI bots for scraping

Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced on Tuesday the launch of a new marketplace that again proposes the relationship between website -owners and AI companies -ideally the publishers more control over their content.

In the past year, Cloudflare has launched tools for publishers to tackle the unbridled rise of AI-Crawlers, including a solution with one click to block all AI bots, as well as a dashboard to see how AI-Crawlers visit their site. In an interview from 2024, Cloudflare CEO Matthew Prince told WAN that these products laid a foundation for a new type of marketplace in which publishers could spread their content under AI companies and could be compensated for it.

Now Cloudflare brings that market to life.

It is called Pay by Crawl and Cloudflare launches the “experiment” in Privé -Bèta on Tuesday. Website -owners in the experiment can choose to have AI -Crawlers scrape their site at a fixed speed on an individual basis -a micro requirement for every “crawl”. As an alternative, website owners can choose to have AI -Crawlers scrape their site for free or to block it completely. Cloudflare claims that its Tools Website -owners will show whether Crawlers scrape their site for AI training data, to appear in AI search actions or for other purposes.

On a scale, Cloudflare’s Marketplace is a big idea that publishers could offer a potential business model for the AI ​​era – and it also places Cloudflare in the middle of all this. The launch of the Marketplace comes at a time when news publishers experience existential questions about how they can reach readers, while Google Search Traffic fades and AI chatbots are increasing in popularity.

There is no clear answer to how news publishers will survive in the AI ​​era. Some, such as the New York Times, have filed lawsuits against technology companies for training their AI models on news articles without permission. In the meantime, other publishers have that Multi -year deals made their content permit For AI modelt training and to make their content appear in Ai Chatbot reactions.

Nevertheless, only large publishers have concluded AI license deals, and it is still unclear whether they offer meaningful sources of income. Cloudflare wants to create a more sustainable system where publishers can set prices on their own conditions.

The company also announced on Tuesday that new websites with Cloudflare will now block all AI crawlers as standard. Site -owners will have to give certain AI Crawlers permission to access their site – a change Cloudflare says that each new domain will give ‘the standard of control’.

Several large publishers, including Conde Nast, Time, The Associated Press, the Atlantic Ocean, Adweek and Fortune, have registered with Cloudflare as standard to block AI-Crawlers as standard to support the broader goal of the company of a “permission-based approach to crawls”.

The business model that many of these publishers trusted for decades is slowly becoming unreliable. Historically, online publishers have allowed Google to scrape their sites in exchange for references in Google Search, which translated into traffic to their sites and ultimately advertisements.

However, new data from Cloudflare suggest that publishers may get a poorer deal in the AI ​​era than in the Google era. While Some websites call chatgpt as a large traffic sourceThat doesn’t seem to be the case.

In June, Cloudflare says it discovered that Google’s Crawler scraped his websites 14 times for every reference it gave them. In the meantime, OpenAi’s Crawler websites scraped 17,000 times for each reference, while anthropic scraped websites 73,000 times for each reference.

In the meantime, OpenAi and Google AI agents who are designed to visit websites on behalf of users are building, collect information and deliver it directly to users. A future in which these tools are mainstream has huge implications for publishers who are dependent on readers who visit their sites.

Cloudflare notes that the ‘true potential’ of wages per crawl can arise in an ‘agent’ future.

“What if an agentic Paywall could work fully programmatically on the EDGE network? Imagine that you ask your favorite deep research program to help you synthesize the latest research of cancer or a legal assignment, or to just help you find the best restaurant in Soho – and then gives that agent a budget in a blog to the best and most relevant content.

To participate in the experimental marketplace of Cloudflare, AI companies and publishers must both be set up with CloudFlare accounts. In their accounts, both parties can set rates against which they want to buy and sell a “crawl” of the content of the publisher. Cloudflare acts as an intermediary in these transactions, charges the AI ​​company and distributes the income to the publisher.

Spokesperson Ripley Park of CloudFlare says WAN that no stablecoins or cryptocurrency is currently involved in paying by cowl, although many have suggested Digital currency would be perfect for something like this.

The Cloudflare market feels like a daring vision for the future that many publishers and AI companies require to board. Yet there are no guarantee that publishers will get a good deal, and convincing AI companies to participate can be difficult, since they are currently scraping for free content.

Nevertheless, Cloudflare seems to be one of the few companies in a position to make a marketplace like this happen.