#ai-scraping

#ai-scraping

New data is reinforcing a structural shift in how AI systems access publisher content: AI models are increasingly scraping publisher content, regardless of bot-blocking measures or content licensing deals meant to control usage, improve attribution or drive referral traffic. New research from analytics firms and bot-tracking companies shows AI tools are increasingly crawling publisher sites as inputs for AI-generated summaries and training, while sending back only limited referral traffic.

Free riding isn't just unfair. It's stealing,

An open licensing standard that aims to make AI companies pay for the content they vacuum up across the web is now an official specification. Really Simple Licensing 1.0 - or RSL for short - gives publishers the ability to dictate licensing and compensation rules to the web crawlers that visit their sites. The RSL Collective announced the standard in September with backing from Yahoo, Ziff Davis, and O'Reilly Media.

As AI bot traffic grows, content creators are taking steps to protect their intellectual property from being scraped against their will. The publishing industry has spent the past year battling against the encroachment of AI tech, with companies like The New York Times and Ziff Davis suing AI platforms for scraping their copyrighted content and using it to train large language models.

A good place to start finding answers is the most recent State of the Bots report from AI startup TollBit. For publishers that are feeling the heat of AI, it attaches real numbers to the presence of AI in the media ecosystem and how quickly it's growing. And while the rise of AI bots is a worrisome trend to those in the content business, it may also be an opportunity.

Cloudflare claims that Perplexity conceals its crawling identity to circumvent website restrictions, resulting in concerns over unauthorized content scraping from various sites.