The popularity of generative AI has made the demand for content used to train models or run inference on skyrocket, and although some AI companies clearly identify their web scraping bots, not all AI companies are being transparent,
Google reportedly paid $60 million a year to license Reddit's user-generated content, highlighting the high value of original content. Cloudflare allows customers to block specific categories of bots, including AI crawlers, to protect content integrity.
Perplexity has been accused of impersonating legitimate visitors to scrape content, emphasizing the importance of transparency in content sourcing. Cloudflare's feature enables blocking certain AI crawlers that misuse licensed content for training models.
Collection
[
|
...
]