The Internet's traditional model of generating traffic through search engine indexing is breaking down due to AI crawlers that collect content without redirecting users to the original websites. This change undermines content creators, denying them revenue and audience engagement. To address this issue, Cloudflare is piloting a payment mechanism called pay per crawl, allowing website owners to control AI access to their content and potentially charge for it. This framework is currently in private beta, with plans for future enhancements, including dynamic pricing.
For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and interesting information. That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source - depriving content creators of revenue, and the satisfaction of knowing someone is reading their content.
Cloudflare is testing a new mechanism payment mechanism, pay per crawl, that enables website owners to decide whether they will permit AI crawlers to access their content, and if that access will be free or they will charge for it. The technology, now in private beta, integrates with existing web infrastructure to create a framework to enable site owners to require payment, and tell the crawler the price via an HTTP '402 payment required' response code.
The site owner can currently set a single price for the site or choose to let certain crawlers access it at no charge, but Cloudflare expects the feature to evolve over time, perhaps to allow dynamic pricing, or charge different amounts for various types of content.
Collection
[
|
...
]