Bots account for almost a third of web traffic

"Bots, automated scripts that perform tasks across the Internet, come in many forms: those considered non-threatening or "good" (such as API clients, search indexing bots like Googlebot, or health checkers) and those considered malicious or "bad" (like those used for credential stuffing, spam, or scraping content without permission). In fact, around 30% of global web traffic today, according to Cloudflare Radar data, comes from bots, and even exceeds human Internet traffic in some locations."

"A new category, AI crawlers, has emerged in recent years. These bots collect data from across the web to train AI models, improving tools and experiences, but also raising issues around content rights, unauthorized use, and infrastructure overload. We aimed to confirm the growth of both search and AI crawlers, examine specific AI crawlers, and understand broader crawler usage. Every now and then I glance at traffic sources, and AI bots seem increasingly common."

Bots include non-threatening 'good' crawlers such as API clients, search indexing bots like Googlebot, and health checkers, and malicious 'bad' bots used for credential stuffing, spam, or unauthorized scraping. Cloudflare data shows bots generate roughly 30% of global web traffic and can exceed human traffic in some regions. AI crawlers harvest web data to train models, improving tools but raising content-rights, unauthorized-use, and infrastructure-overload concerns. Recent trends indicate growth in both search and AI crawlers and increasing prevalence of AI bot traffic in site analytics.

Read at FlowingData