"the nonprofit Creative Commons has come out in favor of "pay-to-crawl" technology - a system to automate compensation of website content when accessed by machines, like AI webcrawlers. Creative Commons (CC) is best known for spearheading the licensing movement that allows creators to share their works while retaining copyright. In July, the organization announced a plan to provide a legal and technical framework for dataset sharing between companies that control the data and the AI providers that want to train on it."
"Spearheaded by companies like Cloudflare, the idea behind pay-to-crawl would be to charge AI bots every time they scrape a site to collect its content for model training and updates. In the past, websites freely allowed webcrawlers to index their content for inclusion into search engines like Google. They benefited from this arrangement by seeing their sites listed in search results, which drove visitors and clicks. With AI technology, however, the dynamic has shifted."
"This shift has already been devastating for publishers by killing search traffic, and it shows no sign of letting up. A pay-to-crawl system, on the other hand, could help publishers recover from the hit AI has had on their bottom line. Plus, it could work better for smaller web publishers that don't have the pull to negotiate one-off content deals with AI providers."
Creative Commons expresses cautious support for pay-to-crawl technology to automate compensation when machines access website content. Creative Commons is known for leading the licensing movement and earlier proposed a legal and technical framework for dataset sharing between data holders and AI providers. Pay-to-crawl, championed by firms like Cloudflare, would charge AI bots each time they scrape a site for model training or updates. Historically webcrawlers drove search traffic and clicks, but AI chatbots provide direct answers and reduce user clickthroughs. That decline has hurt publishers' search revenue. Pay-to-crawl could help publishers recover and offer a scalable option for smaller publishers.
Read at TechCrunch
Unable to calculate read time
Collection
[
|
...
]