"A swarm of AI "crawlers" is running rampant on the internet, scouring billions of websites for data to feed algorithms at leading tech companies -- all without permission or payment, upending the online economy. Before the rise of AI chatbots, websites allowed search engines to access their content in return for increased visibility, a system that rewarded them with traffic and advertising revenues."
"But the rapid development of generative AI has allowed tech giants like Google and OpenAI to harvest information for their chatbots with web crawlers, without humans ever needing to visit the original sites. Traditional content producers, such as media outlets, are being outpaced by AI crawlers, which have cut into their online operations and advertising revenues. "Sites that gave bots access to their content used to get readers in exchange," said Kurt Muehmel, head of AI strategy at data management firm Dataiku."
A swarm of AI crawlers scans billions of websites to collect data for algorithms at major tech firms without permission or payment, eroding the internet's economic incentives. Previously, search engines provided visibility that drove human traffic and advertising revenue to sites. Generative AI enables companies to ingest site content for chatbots so humans often never visit original sources. Media outlets and other content producers are losing traffic and advertising income as a result. Cloudflare moved to block AI crawlers from accessing content without payment or permission. Wikipedia reported an eight percent fall in human traffic between 2024 and 2025 due to AI summaries.
Read at US AFPNews
Unable to calculate read time
Collection
[
|
...
]