OpenAI cofounder says scaling compute is not enough to advance AI: 'It's back to the age of research again'

"The wisdom goes that the more compute you have or the more training data you have, the smarter your AI tool will be. Sutskever said in the interview that, for around the past half-decade, this "recipe" has produced impactful results. It's also efficient for companies because the method provides a simple and "very low-risk way" of investing resources compared to pouring money into research that could lead nowhere."

"Is the belief really: 'Oh, it's so big, but if you had 100x more, everything would be so different?' It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don't think that's true," Sutskever said. "So it's back to the age of research again, just with big computers."

AI companies prioritized scaling compute and accumulating vast training data to improve models like LLMs and image generators. For roughly the past half-decade, that recipe produced impactful results and provided a low-risk path for investment compared with speculative research. Data availability is finite and many organizations already possess massive compute resources, reducing marginal returns from further scaling. The premise that multiplying scale by orders of magnitude will automatically transform capabilities is questioned. A renewed emphasis on foundational research is necessary, while compute remains essential for experimentation and can still serve as a significant differentiator.

Read at Business Insider