"For 25 years, search engines like and Microsoft's Bing crawled and indexed websites, sending users to relevant pages. This drove traffic, supporting the web's grand bargain: sites let tech firms copy their data for free in exchange for referrals. Ads and subscriptions funded content creation, which in turn improved search results. In today's new AI era, products such as Google's AI Overviews and , OpenAI's , and Perplexity deliver answers directly, often eliminating the need to visit the original source."
"The IETF is working on new standards that would define search engines and generative AI answer engines as different entities. It would also provide a way for sites block AI bot crawlers, while still allowing other bots that feed traditional search engines and send users to the original sources of information."
The Internet Engineering Task Force is developing new web standards to distinguish traditional search engines from generative AI answer engines and to enable sites to block AI bot crawlers while permitting bots that drive referral traffic. For decades, search engines crawled and indexed websites, sending users to original pages and supporting advertising and subscription revenue models. Generative AI products now deliver direct answers using scraped content, reducing referral traffic and threatening revenues that fund content creation. Major technology companies including Google, Microsoft, OpenAI, and Amazon have opposed or sought to narrow the proposed standards, while publishers and content owners are pushing them forward in a process that could conclude by the end of 2025.
Read at Business Insider
Unable to calculate read time
Collection
[
|
...
]