AI crawlers are hammering sites and nearly taking them offline
Briefly

Edd Coates expressed concern over the surge in web traffic due to AI crawlers, stating that his website performance declined despite having fewer visitors, which seemed counterintuitive.
Kyle Wiens highlighted the alarming situation at iFixit when he noticed a spike in traffic that was 'abnormally high,' caused by a web crawler from Anthropic.
According to the Data Provenance Initiative, approximately one in four tokens from critical web domains have implemented restrictions to combat the influx of activity from AI crawlers.
Wiens noticed that his staff's alerts about high cellphone traffic indicated an issue, leading them to discover that nearly a million queries had hit their site in just over 24 hours.
Read at Fast Company
[
]
[
|
]