#robotstxt tag - Briefly

2 days ago

OpenAI Updates Its ChatGPT Crawler OAI-SearchBot

It looks like ChatGPT User (the user action bot) will no longer comply to robots.txt rules (Open changed the wording from "the following robots.txt tags", referring to all 3 user agents to "OAI SearchBot and GPTBot robots.txt tags") OAISearchBot is no longer used to feed the navigational links in ChatGPT answers (blocking this bot does not mean your will not appear in the links)

Artificial intelligence

fromTheregister

2 days ago

Publishers say no to AI scrapers, block bots at server level

Millions of websites are blocking AI crawler bots via robots.txt to prevent training-data scraping and reduce non-human server traffic.

#amazon

fromZDNET

1 week ago

E-Commerce

Amazon puts ChatGPT on the naughty list, blocking shopping access - what we know

2 weeks ago

E-Commerce

Amazon quietly blocks more of OpenAI's ChatGPT web crawlers from accessing its site

E-Commerce

Amazon quietly blocks AI bots from Meta, Google, Huawei and more

fromZDNET

1 week ago

E-Commerce

Amazon puts ChatGPT on the naughty list, blocking shopping access - what we know

2 weeks ago

E-Commerce

Amazon quietly blocks more of OpenAI's ChatGPT web crawlers from accessing its site

E-Commerce

Amazon quietly blocks AI bots from Meta, Google, Huawei and more

more#amazon

Artificial intelligence

fromFast Company

3 weeks ago

Misinformation sites have an open-door policy for AI scrapers

Reputable news websites increasingly use robots.txt to block AI crawlers, while misinformation sites rarely restrict such crawling.

Online marketing

Google On Case Sensitivity For URLs

Be consistent with URL casing because path, filename, and query parameters are case-sensitive and affect canonicalization, duplication, and robots.txt.

#cloudflare

fromAdExchanger

Artificial intelligence

Scrapers Gonna Scrape; No More Fast-Forwarding The Ads, DVR Friends | AdExchanger

fromArs Technica

Tech industry

Inside the web infrastructure revolt over Google's AI Overviews

2 months ago

Artificial intelligence

Cloudflare updates robots.txt for the AI era - but publishers still want more bite against bots

fromAdExchanger

Artificial intelligence

Scrapers Gonna Scrape; No More Fast-Forwarding The Ads, DVR Friends | AdExchanger

fromArs Technica

Tech industry

Inside the web infrastructure revolt over Google's AI Overviews

2 months ago

Artificial intelligence

Cloudflare updates robots.txt for the AI era - but publishers still want more bite against bots

more#cloudflare

Privacy technologies

fromSitePoint Forums | Web Development & Design Community

2 months ago

Cloudflare Tries To Give Sites A Way To Block Google AI Overviews

Cloudflare's Content Signals Policy adds robots.txt directives allowing site owners to block AI Overviews use of their content while keeping standard Google Search inclusion.

Online marketing

Intellectual property law

Website sitemap is not detecting by ubersugget

Ubersuggest can fail to detect live sitemaps due to access, format, robots.txt blocks, firewall/password protection, or lack of Google Search Console submission.

fromArs Technica

Pay-per-output? AI firms blindsided by beefed up robots.txt instructions.

RSL enables publishers to declare licensing terms and require compensation from AI crawlers and agents via an automated robots.txt-based protocol.

Artificial intelligence

fromThe Verge

The web has a new system for making AI companies pay up

Really Simple Licensing (RSL) lets web publishers specify licensing and royalty terms in robots.txt and other content to require payment for AI training-data scraping.

Tech industry

fromAdExchanger

Intellectual property law

Amazon Gets Scraped, Too; LinkedIn Loves Video | AdExchanger

AI companies are crawling Amazon for shopping data, LinkedIn is expanding invite-only video revenue-sharing, and platform competition is generating disputes between Google and Fox.

fromTheregister

Asahi, Nikkei sue Perplexity AI for copyright infringement

Perplexity faces a copyright lawsuit from Japan's Nikkei and Asahi alleging unlawful scraping, robots.txt violations, and seeking injunctions plus ¥2.2 billion damages per firm.

Web development

Good Web Crawler Attributes

Good web crawlers support HTTP/2, identify via user-agent, respect robots.txt, follow caching and redirects, back off on slow servers, and expose crawl details.

Information security

fromTheregister

4 months ago

Perplexity AI crawlers accused of stealth data scraping

Perplexity AI search startup is allegedly disguising its content-scraping bots to ignore website restrictions.

E-Commerce

4 months ago

Shopify has quietly set boundaries for 'buy-for-me' AI bots on merchant sites

Shopify is implementing measures to block agentic AI bots from completing transactions without human review.

Apple