Some people are defending Perplexity after Cloudflare ‘named and shamed’ it

When CloudFlare AI search engine forbade from secretly scraping websites on Monday, while ignoring the specific methods of a site to block it, this was not a clear case of an AI-webcrawler who became wild.

Many people came to the defense of Pertlexity. They argued that perplexity access to sites in violation of the wishes of the website owner, although controversial, is acceptable. And this is a controversy that will certainly grow as AI agents flood the internet: does an agent who have access to a website on behalf of his user should be treated as a bot? Or like a person who submits the same request?

Cloudflare is known for offering anti-bot crawling and other web security services to millions of websites. In essence, the CloudFlare test case included setting up a new website with a new domain that had never crawled through a bone, setting up a robots. Txt file that specifically blocked the well -known AI -circling bots of PerTlexity and then asked perplexity to the content of the website. And confusion answered the question.

Cloudflare researchers discovered that the AI search engine used “a generic browser that was intended to stimulate Google Chrome on macOS” when his webcrawler himself was blocked. Cloudflare CEO Matthew Prince posted The research into X, writing: “Some so -called ‘renowned’ AI companies behave more like North Korean hackers. Time to name them, ashamed and blocking hard.”

But many people did not agree with Prince’s assessment that this was really bad behavior. Those who are confused on sites like x And Hacker News Be aware that what Cloudflare seemed to document, the AI was accessing a specific public website when the user asked about that specific website.

“If I request a website as a person, I would have to be shown the content,” a person on Hacker News wrote and added: “Why would the LLM be the website on my behalf in a different legal category like my Firefox -Webbrowser?”

A Pertlexity spokesperson previously denied WAN that De Bots were the company and called Cloudflare’s Blogpost a sales pitch for Cloudflare. Than on Tuesday, bewilderment a blog published In his defense (and in general attacked by Cloudflare), the behavior that it was occasionally was an external service that it occasionally uses.

But the core of the post from Pertlexity made a similar profession as its online defenders.

“The difference between automated crawls and the user controlled is not only technical-it is about who gets access to information on the open web,” said the mail. “This controversy reveals that the systems of Cloudflare are fundamentally insufficient to distinguish between legitimate AI assistants and actual threats.”

The accusations of Pertlexity are also not exactly fair. An argument that Prince and CloudFlare used to call on the methods of PerTlexity was that OpenAi does not behave in the same way.

“OpenAi is an example of a leading AI company that follows these best practices,” Cloudflare wrote. “They respect robots.txt and try not to avoid a robots.txt guideline or a network level block. And Chatgpt -Agent signs HTTP requests using the newly proposed Open Standard Web Bot -Auth.”

Web bot auth is a cloud-flare-supported standard developed by the Internet Engineering Task Force that hopes to create a cryptographic method for identifying AI agent web requests.

The debate comes when bone activity reforms the internet. As WAN has previously reported, bots that want to scrape huge amounts of content to train AI models have become a threat, especially for smaller sites.

For the first time in the history of the internet, BOT activity is currently surpassed onlineWith AI traffic items for more than 50%, according to the Bot Bot report from Imperva that was released last month. The majority of that activity comes from LLMS. But the report also discovered that malignant bots now make up 37% of all internet traffic. That is activity that includes everything, from persistent scraping to unauthorized login attempts.

Until LLMS, the internet generally accepted that websites could and should block the most bone activity, given how often it was malignant by using Captchas and other services (such as Cloudflare). Websites also had a clear incentive to work with specific good actors, such as Googlebot, to supervise what not to index via robots.txt. Google indexed the internet, which sent traffic to sites.

Now LLMS is eating an increasing amount of that traffic. Gartner predicts That search engine volume will fall by 25% by 2026. At the moment people tend to click on LLMS website coupling on the point that they are most valuable for the website, that is when they are ready to carry out a transaction.

But if people accept agents such as the technical industry predicts that they will – to arrange our journey, book our dinner reservations and shop for us – would websites harm their business interests by blocking them? The debate on X has perfectly recorded the dilemma:

“I want to be confusing to visit public content on my behalf when I give it a request/task!” written one person In response to Cloudflare calls perplexity.

‘What if the site owners don’t want it? They just want you [to] Visit the house directly, see their things ” another arguedindicate that the site owner who has made the content wants the traffic and the potential advertisement income, not to let Pertlexity take it.

“That is why I can’t really see ‘agent browsing’ working – much more difficult problem than people think. Most website owners will simply block,” a third predicted.