Meta's new web crawler, Meta External Agent, scrapes publicly available data for AI training, similar to OpenAI’s GPTBot, raising concerns about data usage.
The existence of the Meta External Agent was confirmed by web scraping tracking firms, indicating a shift in how Meta collects data for its AI models.
Meta's spokesman confirmed that the company has been operating crawlers for years but updated its guidance for publishers to exclude their domains from scraping.
The practice of scraping data for AI training has faced backlash from content creators, leading to lawsuits over the unauthorized use of intellectual property.
[
Collection
]
[
|
...
]