"It looks like ChatGPT User (the user action bot) will no longer comply to robots.txt rules (Open changed the wording from "the following robots.txt tags", referring to all 3 user agents to "OAI SearchBot and GPTBot robots.txt tags") OAISearchBot is no longer used to feed the navigational links in ChatGPT answers (blocking this bot does not mean your will not appear in the links)"
"ChatGPT User is now also said to be used for Custom GPT requests and GPT Actions. OAISearchBot and GPTBot share information with each other ("If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicate crawling.") -> this confirms what we have been seeing lately in the log files: for some prompts we see 2 or all 3 OAI bots visiting the site."
OpenAI updated descriptions for OAI-SearchBot and ChatGPT User, removing claims that crawlers feed navigational links and are used to train generative foundation models. The robots.txt wording now specifies OAI SearchBot and GPTBot tags, which alters stated robots.txt coverage for ChatGPT User. OAISearchBot is no longer described as supplying navigational links, and blocking OAISearchBot does not guarantee exclusion from link results. ChatGPT User is identified as handling Custom GPT requests and GPT Actions. OAISearchBot and GPTBot may reuse crawl results to avoid duplicate crawling, and logs show multiple OAI bots visiting sites for some prompts. Crawling generally exceeds referral traffic from OpenAI.
Read at Search Engine Roundtable
Unable to calculate read time
Collection
[
|
...
]