fromSearch Engine Roundtable
4 weeks agoOpenAI Scaling Up Crawling & Bots
OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Vercel noticed OpenAI adding a lot of new IP ranges for its bots and crawlers. Ryan Siddle from Merj wrote on LinkedIn, "OpenAI scaling up their infrastructure ahead of Thanksgiving & Black Friday with a lot of /28 blocks." He added later in the comments, "That's just across OpenAI User for new IPs. It doesn't include what they already had. We've seen quite a significant ramp up over the past 1-2 months."