Cloudflare is now blocking AI crawlers by default, allowing content creators to block unauthorized access to their content. This shift from opt-out to opt-in for AI crawling is significant, as about 20% of web pages utilize Cloudflare's services. The new Pay Per Crawl model enables content creators to establish their payment rates for AI services seeking to crawl their content. Publishers have full control over which AI crawlers can access their content, providing flexibility and the potential for compensation.
Cloudflare has become the first Internet infrastructure provider to block AI crawlers by default, requiring permission or compensation to access content.
Cloudflare's Pay Per Crawl initiative allows content creators to set rates and get paid by AI services that wish to crawl their content.
With Cloudflare's new system, publishers can block all AI crawlers or allow specific ones, controlling how their content is accessed.
Customer adoption of Pay Per Crawl includes many notable publishers like ADWEEK, BuzzFeed, and Fortune, indicating significant industry interest.
Collection
[
|
...
]