A public feud between Cloudflare and Perplexity highlights significant flaws in bot detection tools managing AI data collection. Cloudflare accused Perplexity of "stealth crawling" to bypass content blocks, while Perplexity refuted these allegations as a "publicity stunt." Despite website owners employing methods to block unwanted data access, issues still arise, revealing inadequacies in safeguarding enterprise content from AI abuse. Analysts suggest this controversy calls for updated standards to effectively distinguish between beneficial AI assistants and malicious scrapers in web interactions.
Cloudflare's investigation began when customers revealed that Perplexity was still accessing their content, even after the site's owners blocked its known crawlers through robots.txt files and firewall rules.
Cloudflare's report indicated that when it blocked Perplexity's declared crawler, Perplexity swiftly switched to using a generic browser to continue accessing the restricted domains.
Collection
[
|
...
]