Rising to the TOPS: How will NPUs and Windows AI grow in 2025?

2024 is a big year for on-device artificial intelligence in consumer electronics. Both Microsoft and Apple have made adjustments to their respective operating systems. Microsoft Launched the “Copilot+ PC” brand for artificial intelligence notebook computers Apple publishes Apple Intelligence.

These early examples provided mixed results. Certain features, such as instant translation and on-device speech-to-text, may be useful. Others, like Microsoft’s Windows Recall, Haven’t proven myself yet.

All this hype about artificial intelligence has important implications for the new year. 2025 looks set to be the year that mainstream developers try to add on-device AI to Windows apps, which means you’ll want to pay closer attention to the AI ​​performance of modern Windows laptops before buying one.

I spoke with two experts in AI research and testing to explore their thoughts on how AI on Windows devices will evolve in 2025.

NPU is about to see huge gains

If you’re curious about the AI ​​performance of Windows laptops, you might end up comparing the promised “TOPS” of each laptop model. TOPS (“tera operations per second”) is a measure of the NPU’s ability to perform matrix multiplications for AI tasks on the device. (Learn more about What is NPU and why it is so important for AI.)

In 2024, the TOPS performance of Windows laptops will be greatly improved. In order to qualify for Microsoft’s “Copilot PC+” brand, Windows laptops must have an NPU performance of at least 40 TOPS. For reference, Qualcomm’s first Copilot+ PC performed at about 45 TOPS, which is four times the performance of Intel’s “Meteor Lake” Core Ultra 7 165H, which had an NPU performance of only 11 TOPS.

“I think Qualcomm really woke everyone up,” said Carl FreundFounder and chief analyst of Cambrian Artificial Intelligence Research Company. Freund noted that AMD and Intel quickly responded with their own chips, delivering similar boosts.

By the end of 2024, consumers looking for high-end Windows laptops such as Microsoft Surface, Asus ProArt or Dell XPS can expect NPU performance to be approximately three to four times higher than similar high-end laptops on the market. . But will this trend continue into 2025?

Ryan Shrout, President, Performance Testing Labs Signal 65think it is ok. “I wouldn’t be surprised if we see a double again, I wouldn’t be surprised if we see a triple again.” However, he expects those final gains to get a greater weight by the end of next year. “My guess is that by the end of 2025, maybe even 2026, we will see the most significant NPU improvements.”

TOPS may not be able to stay on top for long

On-device AI performance could be significantly improved by two to three times. Freund and Shrout caution, however, that it’s best not to place too much faith in the TOPS performance quoted by chip manufacturers.

“TOPS really stands for ‘Too Highly Overcommitted Performance Statistics,'” Freund said. “It doesn’t have much value.”

Shrout agreed, comparing TOPS to the TFLOPS numbers AMD and Nvidia often quote when marketing their GPUs. These numbers indicate the maximum possible computing speed of a GPU, but surprisingly they tell little about actual performance.

Right now, real-world AI performance is a bit uncertain, in part because Windows has yet to integrate a single API to take advantage of the NPU’s AI capabilities. This is a problem for owners of Copilot+ laptops that lack Qualcomm chips inside.

Although AMD and Intel both have chips that qualify for the Copilot+ brand, Qualcomm has so far enjoyed priority. Qualcomm machines are among the first to receive support for Windows Recall and several popular applications such as Blender and Affinity Photo. Recently announced to only work with Qualcomm Snapdragon X hardware.

However, that should change in 2025 as Microsoft ramps up support for its low-level machine learning API (DirectML) and the Windows Copilot runtime, including multiple task-specific AI APIs (some of which are yet to be released) ). At present, it is obvious that The Copilot+ computer leaves a lot to be desired And there is still a lot of room for growth in the future.

“I think Microsoft will solve this problem in 2025,” Shrout said. “Once application developers use DirectML, just like they used DirectX, the problem will be solved. And I don’t think it will be a problem long-term.” Shrout compared it to the early days of PC 3D, where competing API, but eventually consolidation coalesced around the leaders, with Microsoft DirectX becoming the most popular choice.

Proving the case for Windows AI

Better NPUs and unified APIs that make it easier for Windows application developers to actually use the full power of the NPU are important advances, but they don’t necessarily guarantee that on-device AI will become commonplace.

That’s because developers still have the option to turn to companies like OpenAI and Anthropic, which make their artificial intelligence models and services available to any device with Internet access. Their AI models are still more powerful than on-device AI models, able to do more and produce results faster.

However, those The cloud-hosted AI model has a major drawback By 2025, price will become even more important.

“The fact that we can have small language models running continuously on the NPU in the background to monitor what’s going on is something that’s not possible in the cloud, or at least would be more expensive from an infrastructure perspective,” Shrout said.

OpenAI’s recent release of ChatGPT Pro, a new premium version for advanced users, seems to illustrate this point. ChatGPT Pro offers unlimited access to the company’s new o1 model and priority access to the Sora video generator, but it’s priced at $200 per month. The price per token that application developers pay to provide o1 to users is equally high.

On the other hand, users and developers using the NPU built into their Windows laptops can basically use it anytime for free. It can be said that this will be the last brick on the road to device-side artificial intelligence. Developers and users will have the tools and motivation to cut costs by relying on NPUs in Windows laptops wherever possible.

It remains to be seen how quickly the shift to on-device AI will occur, and to what extent it will spread through the Windows software ecosystem, but 2025 will likely be a huge turning point.

“I think Qualcomm was right five years ago when it said artificial intelligence would move to devices. At first, I was skeptical. But now I’m a believer,” Freund said.

Further reading: Free AI tools that run locally on your PC