Cloudflare has initiated a significant change by blocking AI web crawlers by default, starting July 1. This policy change is a response to the overwhelming number of requests and slowdowns experienced by websites due to aggressive behavior from AI crawlers like OpenAI's GPTBot. The new protocol ensures that any new websites using Cloudflare services will automatically block these AI crawlers unless permission is granted. Additionally, Cloudflare aims to detect and block shadow scrapers through advanced technology, allowing site owners to regain control over their content and access rights.
Cloudflare's policy change blocks AI web crawlers by default starting July 1, addressing excessive and disruptive requests from AI companies accessing content without permission.
Website owners, including those with smaller sites, have reported performance issues due to aggressive crawling by AI bots like OpenAI's GPTBot, resulting in slower site speeds.
The previous situation required website owners to opt out of AI crawling, but now, new customers of Cloudflare will automatically block AI crawlers unless they grant explicit permission.
Cloudflare's new strategy includes monitoring and blocking shadow scrapers through behavioral analysis and machine learning, granting control back to the website owners.
Collection
[
|
...
]