#ai-web-crawling

[ follow ]
#digital-publishing #web-scraping-metrics #regulatory-action #amazon #openai #robotstxt #e-commerce-advertising
fromDigiday
2 days ago

The accidental guardian: How Cloudflare's Matthew Prince became publishing's unexpected defender

When Cloudflare CEO Matthew Prince first started getting calls from distressed publishers about the threat of AI crawlers scraping their content, his knee-jerk reaction was to roll his eyes. And honestly, a little eye-rolling is fair - Cloudflare's day job is fending off botnets and nation-state cyberattacks, not debating how Google and other AI companies crawl publisher sites. That means any AI-focused crawling the company tracks represents a narrow slice of the overall traffic and data the cloud-edge company processes.
Information security
E-Commerce
fromDigiday
1 month ago

Amazon quietly blocks more of OpenAI's ChatGPT web crawlers from accessing its site

Amazon is blocking OpenAI-related crawlers via robots.txt to protect advertising revenue and steer shoppers to its own AI offerings.
Web development
fromBusiness Insider
3 months ago

Cloudflare goes after Google's AI Overviews with a new license for 20% of the web

Cloudflare's Content Signals Policy lets site owners control AI crawling and creates a license restricting AI scraping, targeting Google's AI Overviews.
Artificial intelligence
fromThe Verge
3 months ago

Encyclopedia Britannica and Merriam-Webster sue AI search company Perplexity

Encyclopaedia Britannica and Merriam-Webster sued Perplexity alleging copyright and trademark infringement, plagiarism, and stealth scraping of their websites.
[ Load more ]