In a recent episode of the Search Off the Record podcast, Gary Illyes spoke about the evolution of search engine crawlers. He mentioned that while Googlebot currently doesn't support HTTP3, it plans to in the future due to its efficiency. Crawlers have adapted with pre and post HTTP headers, an old robots.txt protocol, and new tactics to combat spammers. Additionally, different companies have various policies, affecting crawler behavior. Illyes emphasized that despite changes in technology, the fundamental way crawlers operate remains consistent.
Behave, yes. How they crawl, there's probably not that much to change. Nowadays you have h2/h3. I mean, we don't support h3 at the moment, but eventually, why wouldn't we?
And then how different companies set policies for their crawlers, that of course differs greatly. Some publishers are complaining that crawler X or crawler B...was doing something that they would have considered not nice.
Collection
[
|
...
]