The end of AI scaling may not be nigh: Here’s what’s next

How AI systems achieve superhuman performance in increasingly complex tasks, the industry is grappling with whether larger models are even possible—or whether innovation must take a different path.

The general approach to developing a large language model (LLM) is that bigger is better and performance increases with more data and more computing power. However, recent media discussions have focused on how LLMs are approaching their limits. “Is AI hitting a wall?” The Verge asked, doc Reuters reported that “OpenAI and others are looking for a new path to smarter AI as current methods hit limitations.”

The concern is that scaling, which has driven progress for years, may not extend to the next generation of models. The reporting suggests that the development of frontier models such as GPT-5, which push the current boundaries of artificial intelligence, may face challenges due to performance degradation during pre-training. The Information reported on these challenges at OpenAI and Bloomberg covered similar news in Google and Antropiko.

This issue has led to concerns that these systems may be subject to the law of diminishing returns — where each additional unit of input yields progressively less gain. As LLMs grow, the cost of obtaining high-quality training data and scaling the infrastructure grows exponentially, reducing the return on performance improvements in new models. This challenge is further compounded by the limited availability of new high-quality data, as much of the available information is already embedded in existing training datasets.

This does not mean the end performance boost for AI. This simply means that further engineering through innovation in model architecture, optimization techniques, and data usage is required to sustain progress.

Learning from Moore’s Law

A similar pattern of diminishing returns emerged in the semiconductor industry. For decades, the industry benefited from Moore’s Law, which predicted that the number of transistors would double every 18 to 24 months, leading to dramatic improvements in performance through smaller and more efficient designs. And this eventually hit diminishing returns, starting somewhere between 2005 and 2007 because of Dennard Scaling — the principle that shrinkable transistors also reduce power consumption — reaching their limits prompted the predictions the death of Moore’s Law.

I had an up close look at this issue when I worked with AMD from 2012-2022. This problem did not mean that semiconductors — and thus computer processors — stopped making performance improvements from generation to generation. This meant that the improvements came more from chiplet design, high-bandwidth memory, optical switches, more cache and accelerated computer architecture, rather than transistor reduction.

New ways of progress

Similar phenomena are already observed with current LLM. Multimodal AI models such as GPT-4o, Claude 3.5, and Gemini 1.5 have proven the power of integrating text and image understanding, enabling advances in complex tasks such as video analysis and contextual image captioning. More tuning of the algorithms for both training and inference will lead to further performance improvements. Agent technologies, which enable LLMs to perform tasks independently and seamlessly coordinate with other systems, will soon expand their practical application significantly.

Future model discoveries may arise from one or more hybrid AI architecture designs that combine symbolic reasoning with neural networks. The O1 reasoning model from OpenAI already shows the potential for model integration and performance extension. While it is just emerging from its early stages of development, quantum computing promises to accelerate AI training and inference by solving current computing bottlenecks.

The perceived scaling wall is unlikely to end future gains, as the AI ​​research community has consistently proven its ingenuity in overcoming challenges and unlocking new capabilities and performance advances.

In fact, not everyone agrees that the wall even exists. OpenAI CEO Sam Altman was succinct in his views: “There is no wall.

Speaking of “Diary of a general manager” podcast, former Google CEO and co-author Genesis Eric Schmitt essentially agreed with Altman, saying he doesn’t believe there is a growing wall — at least not in the next five years. “In five years, you’ll have two or three more turns of these LLMs.” “Every one of these knobs looks like it’s a factor of two, factor of three, factor of four capabilities, so let’s just say that turning the knob on all of these systems becomes 50 or 100 times more powerful,” he said.

Leading AI innovators remain optimistic about the pace of progress, as well as the potential for new methodologies. This optimism is evident in a recent conversation on “Lenny’s Podcast” with OpenAI’s CPO Kevin Weil and Anthropic CPO Mike Krieger.

In this discussion, Krieger described what OpenAI and Anthropic are working on today “feels like magic,” but admitted that in just 12 months, “we’re going to look back and say, can you believe we used that garbage? … So fast [AI development] it’s moving.”

It’s true — it feels like magic, as I recently experienced when using OpenAI Advanced voice mode. The conversation with ‘Juniper’ was completely natural and seamless, showing how AI is evolving to understand and respond with emotion and nuance in real-time conversations.

Krieger also talks about the recent o1 model, calling it “a new way to increase intelligence, and we feel like we’re just at the very beginning.” He added: “Models will get smarter at an accelerated rate.”

These anticipated advances suggest that while traditional scaling approaches may or may not face diminishing returns in the near future, the field of artificial intelligence is poised for continued breakthroughs through new methodologies and creative engineering.

Does scaling even matter?

While scaling challenges dominate much of the current discourse around LLM, recent studies suggest that current models are already capable of delivering outstanding results, raising the provocative question of whether scaling up even matters.

A a recent study predicts that ChatGPT will help doctors make diagnoses when complicated patient cases arise. Conducted with an early version of the GPT-4, the study compared the diagnostic abilities of ChatGPT to those of physicians with and without AI assistance. The surprising outcome revealed that ChatGPT alone significantly outperformed both groups, including AI-assisted doctors. There are several reasons for this, from doctors not understanding how to best use a bot to their belief that their knowledge, experience and intuition are inherently superior.

This is not the first study to show bots outperforming professionals. VentureBeat reported on a study earlier this year that showed LLMs can perform financial statement analysis with accuracy that rivals — and even surpasses — that of professional analysts. Also using GPT-4, another objective was to predict future earnings growth. The GPT-4 achieved 60% accuracy in predicting the direction of future earnings, significantly higher than the 53-57% range in human analysts’ forecasts.

It is significant that both of these examples are based on models that are already outdated. These results highlight that, even without new breakthroughs in scaling, existing LLMs are already capable of outperforming experts in complex tasks, challenging assumptions about the necessity of further scaling to achieve effective results.

Scaling, skill, or both

These examples show that current LLMs are already very capable, but scaling alone may not be the only path for future innovation. But with greater scaling possible and other new techniques promising to improve performance, Schmidt’s optimism reflects the rapid pace of AI progress, suggesting that in just five years the models could evolve into polymaths, seamlessly answering complex questions in multiple fields.

Whether through scaling, skill, or entirely new methodologies, the next frontier of AI promises to transform not only the technology itself, but also its role in our lives. The challenge ahead is to ensure that progress remains accountable, equitable and impactful for all.

Gary Grossman is EVP of the Technology Practice at Edelman and global leader of the Edelman AI Center of Excellence.