Scrapers Gonna Scrape; No More Fast-Forwarding The Ads, DVR Friends | AdExchanger
Briefly

Scrapers Gonna Scrape; No More Fast-Forwarding The Ads, DVR Friends | AdExchanger
"The policy allows website operators to opt in and out of scraping for specific use cases, including showing up in search results, training AI models and inputting content into AI models. Cloudflare is hoping it can thread the needle between being useful to publishers, not causing publishers to lose search traffic and actually functioning as a block on LLM scrapers."
"Last month, Cloudflare announced its much-heralded Content Signals Policy, which updated millions of websites' robots.txt files so as to restructure the way crawlers access content. As in, restrict LLM web-scraping bots. Google currently bundles its AI crawler with the main search crawler, so nobody can risk its removal. Whereas publishers are relatively powerless on their own, Cloudflare's scale makes it more challenging for Google to simply choose not to scrape the affected sites or to ignore the new protocol altogether."
Cloudflare's Content Signals Policy updated websites' robots.txt to restructure crawler access and restrict LLM web-scraping bots. The policy allows website operators to opt in or out of scraping for use cases including appearing in search results, training AI models, and inputting content into AI models. Google bundles its AI crawler with the main search crawler, making removal risky. Cloudflare's scale makes it harder for Google to ignore the new protocol and reduces publishers' powerlessness. Some consumers who relied on ad-skipping hardware may drop cable yet resist pricier ad-free streaming tiers. Nestlé plans to shift marketing investments toward growth opportunities including product innovation, distribution, packaging, and digital capabilities.
Read at AdExchanger
Unable to calculate read time
[
|
]