"Prince's goal: to prove why Google must completely separate its AI and search crawlers - now a major point of friction for publishers. Google technically separates its search crawler (Googlebot) and its AI crawler (Google-Extended), but in practice, they overlap. Even if a publisher blocks Google-Extended, their content can still show up in AI Overviews, because those are tied to Google Search."
"Prince, ever the middleman, between publishers (and other content owners) that rely on Cloudflare's network for security, performance, and access control, and AI companies, which depend on it to deliver requests, crawl content, and serve models efficiently. That puts Cloudflare at the chokepoint where access rules, provenance, and traffic flows get enforced. If a publisher wants to block or throttle AI crawlers, or if an AI company needs to prove it's respecting provenance standards, Cloudflare is the infrastructure layer that can make that happen."
Matthew Prince met with the U.K. Competition Markets Authority after the CMA granted Google Strategic Market Status, allowing probes beyond search into AI features such as AI Overviews, AI Mode, Discover, Top Stories, and the News tab. Prince seeks separation of Google's AI and search crawlers because Googlebot and Google-Extended overlap in practice, enabling content blocked from AI crawlers to still appear via Search-tied AI features. Google warned that heavy-handed measures could slow AI-era product launches, raise costs, and cited a claimed £118 billion contribution to the U.K. economy in 2023. Cloudflare occupies an infrastructure chokepoint that can enforce paywalls, provenance, access controls, and rate-limits at scale.
Read at Digiday
Unable to calculate read time
Collection
[
|
...
]