"When Cloudflare CEO Matthew Prince first started getting calls from distressed publishers about the threat of AI crawlers scraping their content, his knee-jerk reaction was to roll his eyes. And honestly, a little eye-rolling is fair - Cloudflare's day job is fending off botnets and nation-state cyberattacks, not debating how Google and other AI companies crawl publisher sites. That means any AI-focused crawling the company tracks represents a narrow slice of the overall traffic and data the cloud-edge company processes."
"But when publishers continued to press him, he pulled the data - and that's when the scale and urgency of the problem snapped into focus for him. He was shocked. He could see that the business model of digital publishing was dissolving. "Ten years ago, for every two pages that the Google [search] bot slurped down, copied, they would send you one visitor. Fast forward to today, it's 18 pages scraped in return for one visitor. What's changed? [Google] AI Overviews," he says."
Cloudflare data revealed a dramatic rise in pages crawled per visitor returned to publishers. Google now scrapes about 18 pages for each visitor via AI Overviews. New AI entrants scrape far more: OpenAI around 1,500 pages per visitor and Anthropic about 40,000 per visitor. The growing imbalance reduces referral traffic and ad revenue, eroding the business model of digital publishing. Cloudflare primarily focuses on defending against botnets and nation-state cyberattacks, but the scale of AI crawling prompted regulatory engagement, including calls for separating AI and search crawlers. Cloudflare relocated its headquarters to Lisbon after Brexit.
Read at Digiday
Unable to calculate read time
Collection
[
|
...
]