Cloudflare has announced it will block AI web scrapers by default for websites it manages, requiring operators to permit indexing if desired. This significant move affects about 20% of all internet traffic and may create a new marketplace for bot access via its Pay Per Crawl program. This shift comes amid growing concerns from media companies about AI chatbots negatively impacting website visits and increasing legal actions over copyright issues. The response from media companies includes blocking AI crawlers as they navigate the evolving landscape influenced by AI technologies.
Cloudflare's new policy will require site operators to allow AI bots to index their content, or risk having them blocked by default, significantly impacting web traffic.
With around 20% of internet traffic passing through Cloudflare, their decision to block AI scrapers by default could change the landscape of how media and AI interact.
Collection
[
|
...
]