AWS HyperPod Task Governance keeps GPUs from idling

Cost remains a major issue with enterprise AI use, and it’s a challenge AWS is grappling with.

exist AWS: Reinventing 2024 At today’s conference, the cloud giant announced HyperPod Task Governance, a sophisticated solution that targets one of the most costly inefficiencies in enterprise AI operations: underutilization of GPU resources.

AWS said that HyperPod task management can increase the usage of AI accelerators, help enterprises optimize AI costs, and potentially save significant costs.

“This innovation helps you maximize computer resource utilization by automatically prioritizing and managing these Gen AI tasks, reducing costs by up to 40%,” said Swami Sivasubramanian, AWS Vice President of Artificial Intelligence and Data.

End GPU idle time

As organizations rapidly expand their artificial intelligence initiatives, many are discovering a costly paradox. Despite significant investments in GPU infrastructure to support a variety of AI workloads, including training, fine-tuning, and inference, these expensive computing resources often sit idle.

Business leaders report surprisingly low utilization of their AI projects even as teams compete for computing resources. As it turns out, this is actually a challenge facing AWS itself.

“We ran into this type of problem a little over a year ago when we were scaling internally, and we built a system that took into account the consumption needs of these accelerators,” Sivasubramanian told VentureBeat. “I spoke with many of our customers, CIOs, and CEOs We talked to them and they said that’s exactly what we want; we want it to be part of Sagemaker and that’s what we’re rolling out.

Swami said that after the system was deployed, AWS’s AI accelerator utilization rate soared to more than 90%

How HyperPod task management works

SageMaker Hyperpod technology debuts in re:invent 2023 conference.

SageMaker HyperPod is designed to handle the complexity of training large models with billions or tens of billions of parameters, which requires managing large clusters of machine learning accelerators.

HyperPod task management adds a new layer of control to SageMaker Hyperpod by introducing intelligent resource allocation across different AI workloads.

The system recognizes that different AI tasks have different demand patterns throughout the day. For example, inference workloads often peak during the hours when the application is using the most work, while training and experiments can be scheduled during off-peak hours.

The system provides enterprises with instant insights into project utilization, team resource consumption and computing needs. It enables organizations to effectively balance the load on their GPU resources across different teams and projects, ensuring expensive AI infrastructure never sits idle.

AWS wants to make sure businesses aren’t wasting money

In his keynote address, Sivasubramanian emphasized the critical importance of AI cost management.

For example, he said, if an organization is assigned 1,000 deployed AI accelerators, not all of them will be available for 24 hours. During the day, they are heavily used for inference, but at night, when inference demand is likely to be very low, a large portion of these expensive resources sit idle.

“We live in a world where computing resources are limited and expensive, and it can be difficult to maximize utilization and allocate resources efficiently, which is often done with spreadsheets and calendars,” he said. “Now, without a strategy for resource allocation method, you will not only miss opportunities but also waste money.”