"Myriam Jessier asked on Bluesky, "what are the good attributes? One should look into when picking a crawler to check things on a site for SEO and gen AI search?" Martin Splitt from Google replied with this list of attributes: support http/2 declare identity in the user agent respect robots.txt backoff if the server slows follow caching directives* reasonable retry mechanisms follow redirects handle errors gracefully*"
"It covers the recommended best practices including: Crawlers must support and respect the Robots Exclusion Protocol. Crawlers must be easily identifiable through their user agent string. Crawlers must not interfere with the regular operation of a site. Crawlers must support caching directives. Crawlers must expose the IP ranges they are crawling from in a standardized format. Crawlers must expose a page that explains how the crawled data is used and how it can be blocked."
Recommended crawler attributes include HTTP/2 support, explicit user-agent identification, adherence to robots.txt, respectful backoff behavior when servers slow, caching-directive compliance, reasonable retry mechanisms, redirect handling, and graceful error handling. Additional best practices require crawlers to avoid interfering with normal site operations, to disclose the IP ranges used for crawling in a standardized format, and to provide a page explaining how crawled data is used and how crawling can be blocked. Emphasis on clear identification and predictable, non-disruptive behavior reduces operational impact and simplifies site operators’ ability to manage crawler access.
Read at Search Engine Roundtable
Unable to calculate read time
Collection
[
|
...
]