IAB Tech Lab tackles the growing AI bot problem | MarTech

A growing volume of AI crawlers and automated agents is pushing publishers and content owners to decide who can access content and under what terms. IAB Tech Lab released guidance on bot and crawler management strategies for public comment through June 26, 2026. The guidance helps publishers, content owners, and ad tech companies manage non-human traffic, including AI systems scraping or accessing content. It complements the CoMP API V1 framework, intended to support communication and permissions between AI systems and publishers. Many organizations lack formal bot and crawler policies despite concerns about content consumption, infrastructure strain, and monetization complications. Blanket bot blocking is described as no longer practical as AI systems become embedded across the web ecosystem. The document outlines multiple approaches and their operational, financial, and strategic implications for business leaders.

"The flood of AI crawlers and automated agents is forcing publishers and content owners to face a challenger: Deciding who gets access to their content, and under what terms. To help address that problem, IAB Tech Lab released new guidance today on bot and crawler management strategies, opening the document for public comment through June 26, 2026."

"The guidance is designed to help publishers, content owners, and ad tech companies think through how they manage non-human traffic, including AI systems scraping or accessing content. It also complements the organization's recently released CoMP API V1, a framework intended to support communication and permissions between AI systems and publishers."

""Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way," Anthony Katsur, CEO of IAB Tech Lab, said in a statement. "This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace.""

"The IAB paper underlines how quickly AI crawling became a business issue across publishing and advertising. Many organizations don't have formal policies on bots and crawlers, despite growing concerns about AI systems consuming content, straining infrastructure, and complicating monetization. According to IAB Tech Lab, the guidance was created because many publishers and content owners were struggling to navigate the trade-offs involved in managing AI access."

Read at MarTech