The number of high-ranking media websites using robots.txt to "disallow" OpenAI's GPTBot dramatically increased from its August 2023 launch until that fall, then steadily (but more gradually) rose from November 2023 to April 2024, according to an analysis of 1,000 popular news outlets by Ontario-based AI detection startup Originality AI. At its peak, the high was just over a third of the websites; it has now dropped down closer to a quarter.
When Apple debuted a new AI agent this summer, for example, a slew of top news outlets swiftly opted out of Apple's web scraping using the Robots Exclusion Protocol, or robots.txt, the file that allows webmasters to control bots. There are so many new AI bots on the scene that it can feel like playing whack-a-mole to keep up.
Collection
[
|
...
]