Cloudflare Radar Year in Review 2024: big source of traffic is AI crawlersGlobal internet traffic grew 17.2%, influenced by trends in connectivity and the rise of AI.
Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)Websites often block outdated bots in an attempt to prevent scraping, but in reality, are leaving new AI crawler bots unblocked.
One-Half of Top News Sites Blocked OpenAI's Crawlers in 2023, Study FindsNearly half of top news websites blocked OpenAI's crawlers, 24% blocked Google's AI crawler.Legacy print publications were more likely to block AI crawlers.