Cloudflare Threatens to Cut Google Off From Their Publishers in Searches Due to AI Scraping

One of our big complaints about AI is that they steal our copyrighted content. On top of that, the intensity of site scraping is so high that even Cloudflare (and we pay pretty hefty monthly fees for their industrial-grade product), which has the best bot detection and blocking on the Internet, is having difficulty in this arms race. Readers may have noticed that from time to time, we have had to put up captchas to stop this theft. If we don’t, the degradation of performance results in time outs for admins and writers. That means we can’t put up new posts or free comments caught in moderation. Apparently Cloudflare is as frustrated as we are. From Adweek in Once Unimaginable, Publishers Are Preparing to Opt Out of Google Search.: The nuclear option is gaining traction as web traffic collapses and Google refuses to negotiate with content creators For decades, publishers have done everything in their power, from the legal to the not-explicitly illegal, to rank as highly in Google Search as possible… Now though, a handful of influential players in the digital media ecosystem have begun moving in the opposite direction, laying the groundwork for what was once unthinkable: removing themselves from Google Search. Last week, the content delivery network Cloudflare, which hosts roughly one-fifth of the websites in the world, gave Google an ultimatum. Beginning Sept. 15, all new websites signing up for Cloudflare, as well as all the customers on its free tier, will have the default settings in their bot management protocol set to block “multi-purpose crawlers” on any webpage that has ads. This means that any crawler that scrapes for both search indexing and AI training will be turned away at the door, unless the site owner decides otherwise. “We’ve been clear about what we want,” said Cloudflare chief strategy officer Stephanie Cohen. “We want a technical solution that allows you to be discoverable without having to give your content away for free.” While a handful of crawlers fit this description—Apple and Bing, among others—the primary, unnamed target of this action is Google, which infamously uses one crawler to both index sites and train its AI models…. o be fair, Google recently introduced an option called Google Extended, which nominally allows publishers to opt out of AI training without disappearing from Search. But publishers are wary that the program will penalize their search visibility, according to executives at two media companies. What this article does not mention is that Google has been hostile to independent publishers at least as early as 2014. We have been repeatedly downgraded in searches, as have sites like The Intercept and Truthout. We don’t see Google as at all helpful in finding new readers, but it would be a shame if they could not longer find articles. However, Kagi is vastly better, particularly on site-specific searches, so we continue to recommend it. Remember, if something is free, you are the product, and with Kagi you also get privacy. They do not retain search histories. Confirming our dim view of Google’s predation: Google pays Reddit millions for content access. Your website's content is an asset too. Don't let every AI crawler scrape it for free. Control who can access your content while keeping Google Search working. Cloudflare AI Crawl Control + Pay Per Crawl are just the beginning. pic.twitter.com/WRxXzEx6KD — Cuong Thach (@cuongthach_) July 10, 2026 And some powerhouses are effectively joining Cloudflare. Again from Adweek: USA Today Inc., which encompasses not just USA Today but a nationwide network of news sites, is weighing its options on the matter, according to CEO Mike Reed. The company, like many others, has responded to declines in search traffic by bolstering audience from other sources, like newsletters, social media, and events. Its traffic has remained relatively stable in recent years, hitting its goal of 1 billion pageviews every month for the last three years, according to Reed. Still, its monetization strategy going forward as it relates to AI will come from licensing agreements, which the company has struck with Meta, Microsoft, and Amazon, among others. Google, unlike its hyperscaler peers or pure-play AI firms like OpenAI and Anthropic, has not struck any licensing deals with any publishers. As a result, USA Today Inc. is prepared to delist from Google in the next six to twelve months, according to Reed. Likewise, the creator network Beehiiv announced in a recent partnership with Cloudflare that its network of creators now has the ability to block the Google crawler. “I wouldn’t call it a big decision because we’re blocking other crawlers,” Reed said. “For those with licensing agreements, they get our content. For those without, we block them.” While USA Today Inc., Beehiiv, and Cloudflare are the first major players to take this step, executives at every major media company have a model for what it would look like if they blocked the Google Bot, according to one executive who wished to remain anonymous because of business engagements with Google. It’s worth noting what these AI scrapers ought to be paying is considerable. For ECONNED, your humble blogger is due $3,000 in a class action settlement with Anthropic. How much do you imagine all of these AI thieves should be paying for the vastly larger body of work at Naked Capitalism? So again, AI is evil, it is destroying the business of original content provider to generate regularly erroneous and colorless derivatives. And it is not good for critical thinking or retention. But informational junk food is proving to be popular. Didn’t see this coming, but good on them! And if Google is downranking your site, why allow their bots in the door? There are other, less scammy search services. That really doesn’t help when the majority of people still googlesearch, particularly insidious for Android OS. Cloudflare chief strategy officer Stephanie Cohen: “We want a technical solution that allows you to be discoverable without having to give your content away for free.” This is why I have no patience with people who go on and on (and on) about the dangers of copyright and how information (an animate object) just wants to be free. The alternative is just what Yves Smith is describing here — “scraping” is a polite word for copying and stealing. How do you think that writers make money (a pittance in most cases) off their work? Through copyright. Cloudfare will benefit from copyright law, too, because copyright law is clear, well established, and has plenty of court cases to back it up. And, no, “fair use” doesn’t allow you to copy large chunks of text from someone’s essay and download photos off someone’s site just because You Be You. > How much do you imagine all of these AI thieves should be paying for the vastly larger body of work at Naked Capitalism? A lot more than $3000. That’s nothing to the likes of Google. When you combine volume, relevance, quality and the variety of perspectives and checks in comments… Idk. More than a master’s degree per user, lets say. Edit to add: Props to Cloudflare for caring and doing something. At an employer I know, they’re hogwild for AI. They haven’t yet entered the “find out” period of cost per token, despite being a huge software infrastructure player in the enterprise market. Everyone is expected to, and is being pushed constantly, to use AI as much as possible for any task, in the mercurial quest of finding some purpose for it. In upskill training material, it explains in one segment that asking for JSON/structured output is worthwhile, because it enables a company to ensure content adheres to internal content styling requirements. Hello. Non-deterministic. It outputs “JSON like” slop. No guarantees that it is valid, so how does this accomplish anything? I hate this timeline. I want to opt out.