Media Briefing: From blocking to licensing, publishers inch toward leverage with AI

"It's become publishers' eternal dilemma in the AI era: to block or not to block - weighing the need to protect content against the lure of potential licensing and visibility deals. This debate still dominates discussions at many of the working groups established between publishers and LLMs, like the IAB Tech Lab's last week, among others. There is still no clear one-size-fits-all solution. For publishers, the decision to block or not block AI crawlers isn't just about principle - it's about leverage."

"But there are new levers for publishers to test in the AI era. Moves by CDN vendors like Cloudflare to crack down more aggressively on bots, combined with the emergence of AI marketplaces like Microsoft's, the push toward pay-per-usage, and intensifying competition among LLMs, are slowly shifting the conversation toward standards and licensing. Publishers are still far from holding the upper hand, but compared to a year ago, the outlook no longer looks quite so bleak."

"Until recently, publishers were virtually defenseless against unconsented AI scraping - with robots.txt serving as little more than a polite request most crawlers ignored. And of course, there is the trickiness of Google's AI and search crawler overlapping, making it hard for publishers to opt out of appearing in AI Overviews without risking their search rankings. Now, thanks to CDNs like Cloudflare stepping in to create real friction, publishers are at least able to contemplate how much access to allow."

Publishers face a dilemma between blocking AI crawlers to protect content and allowing access to gain visibility and licensing opportunities. Blocking grants bargaining power but risks invisibility; allowing access preserves discoverability but can give away value for free. New tools and market shifts are creating alternatives, including CDNs imposing friction, AI marketplaces, pay-per-usage pricing, and intensified LLM competition. These developments are nudging conversations toward standards, protocols, and licensing frameworks. Robots.txt offered little protection previously, and overlapping Google AI/search crawlers complicated opt-outs. Growing infrastructure and marketplace options are slowly improving publishers' leverage.

Read at Digiday