A good place to start finding answers is the most recent State of the Bots report from AI startup TollBit. For publishers that are feeling the heat of AI, it attaches real numbers to the presence of AI in the media ecosystem and how quickly it's growing. And while the rise of AI bots is a worrisome trend to those in the content business, it may also be an opportunity.
Google has one crawler, which means they use the same crawler for their search, where they still send us traffic, as they do for their AI products, where they steal our content,
The idea behind RSL is brutally simple. Instead of the old file -- which only said, "yes, you can crawl me," or "no, you can't," and which AI companies often ignore -- publishers can now add something new: machine-readable licensing terms. Want an attribution? You can demand it. Want payment every time an AI crawler ingests your work, or even every time it spits out an answer powered by your article?
Google's CEO, Sundar Pichai, said in May that web publishing is not dying. Nick Fox, VP of Search at Google, said in May that the web is thriving. But in a court document filed by Google on late Friday, Google's lawyers wrote, "The fact is that today, the open web is already in rapid decline."
The verdict has felt anti-climactic for media and ad execs who had hoped for sweeping change, especially given the court had already ruled Google's search dominance a monopoly last August. Publishing execs hoped the remedies would separate Google's search engine crawler from its AI experiments - such as AI Overviews and AI Mode - or at least force them to provide more data on how those products are impacting publisher clickthroughs from search. That would've given publishers more control over where their content is showing up, and what it's used for.
Publishers are picking up new vibes from Meta, which they believe signal that the platform may be changing its stance on AI licensing. So far, it's more rhetoric than reality. Nevertheless, if it were to come to fruition, it could reset the dynamic between Meta and publishers, many of whom still feel burned by years of declining referral traffic from the platform.
Last year, Facebook moved away from a revenue-share payout model that paid publishers based on the performance of ads in content, transitioning to a model that rewards creators based on content performance itself. This shift has effectively changed how publishers engage with the platform, focusing more on the content they create for Facebook.
