Microsoft To Publishers: Don't Block The AI Bots | AdExchanger

Publishers and retailers navigating AI-driven search and agentic tools should make their content accessible to bots that generate new traffic. Site owners should update robots.txt to avoid blocking crawlers that reduce visibility to AI engines. Many websites currently block AI bot traffic, which makes content and products difficult for agents to interpret. When agents cannot access or understand site content, discovery, recommendations, and demand decline. Microsoft also emphasizes licensing so publishers receive value for their data. Microsoft’s Publisher Content Marketplace supports licensing agreements between publishers and AI developers, initially focused on Microsoft Copilot and later expanded to other AI firms with Microsoft acting as a clearing house.

"Microsoft has one big piece of advice for publishers and retailers that are cautiously navigating the AI era: Let the bots scrape your sites. That's according to Nikhil Kolar, VP of publisher product at Microsoft AI, who spoke at AdExchanger's Programmatic AI event in Las Vegas last week. Rather than fighting against the rising tide of AI-based search engines and agentic tools, Kolar said publishers and retailers should actually be creating content that can speak directly to these bots (and thus the LLMs generating new forms of traffic)."

"And site owners should also update their robots.txt files to avoid restrictions on crawlers that will diminish the site's visibility to AI engines. Currently, though, four out of five websites block traffic from AI bots and crawlers, according to Kolar. As a result, "the content or products that you have are not legible to agents." Which means, he added, that "your business is closed" and "you get no discovery, no recommendations that you're a part of and no demand.""

"There is a clearly self-serving angle for Microsoft in pitching publishers on the idea that they ought to open up their sites to AI crawling. After all, Microsoft has its own AI models and AI chatbot searches, and it needs publisher data to inform their outputs. But Microsoft also wants publishers to receive fair value for their data, Kolar said. He pointed to Microsoft's Publisher Content Marketplace as a sign of good faith."

"The initiative, which was announced in February, facilitates licensing agreements between publishers and AI developers. The initial focus was for publishers to monetize by licensing their content or data to be used by Microsoft Copilot. But Microsoft has since opened the ecosystem to other AI firms, with Microsoft acting as the clearing house for licensing the publisher data."

Read at AdExchanger