#robotstxt

[ follow ]
#web-scraping #perplexity #web-crawlers #sitemap #ubersuggest #google-search-console #seo #ai-licensing
Intellectual property law
fromArs Technica
1 week ago

Pay-per-output? AI firms blindsided by beefed up robots.txt instructions.

RSL enables publishers to declare licensing terms and require compensation from AI crawlers and agents via an automated robots.txt-based protocol.
Artificial intelligence
fromThe Verge
1 week ago

The web has a new system for making AI companies pay up

Really Simple Licensing (RSL) lets web publishers specify licensing and royalty terms in robots.txt and other content to require payment for AI training-data scraping.
Tech industry
fromAdExchanger
3 weeks ago

Amazon Gets Scraped, Too; LinkedIn Loves Video | AdExchanger

AI companies are crawling Amazon for shopping data, LinkedIn is expanding invite-only video revenue-sharing, and platform competition is generating disputes between Google and Fox.
Intellectual property law
fromTheregister
3 weeks ago

Asahi, Nikkei sue Perplexity AI for copyright infringement

Perplexity faces a copyright lawsuit from Japan's Nikkei and Asahi alleging unlawful scraping, robots.txt violations, and seeking injunctions plus ¥2.2 billion damages per firm.
E-Commerce
fromDigiday
3 weeks ago

Amazon quietly blocks AI bots from Meta, Google, Huawei and more

Amazon is blocking AI companies' web crawlers via robots.txt to prevent scraping of its e-commerce data and protect its marketplace and ad business.
E-Commerce
fromDigiday
2 months ago

Shopify has quietly set boundaries for 'buy-for-me' AI bots on merchant sites

Shopify is implementing measures to block agentic AI bots from completing transactions without human review.
Apple
fromSearch Engine Roundtable
4 months ago

Apple Updates Applebot Documentation Explaining Applebot-Extended vs Applebot

Applebot-Extended clarification highlights its role in AI training and implications for web publishers.
[ Load more ]