NVIDIA AI Podcast

Powering the AI Inference Wave with EPRI's Ben Sooter - Ep. 292

32 min

This podcast explores the critical intersection of AI data centers and energy grids, focusing on the immense energy demands of AI inference. Ben Sooter from EPRI details how micro data centers, strategically located near underutilized substations, can efficiently power the **80% of a model's lifetime energy consumption** attributed to inference. This distributed approach leverages existing infrastructure, offers faster deployment, and provides crucial flexibility for the energy grid.

Summarized by Podsumo

🎙️ Ask about this episode

✨ Key Takeaways

1

AI Inference Dominates Energy Use

A significant insight reveals that 80% of an AI model's total compute and power consumption occurs during inference (when models are actively used), compared to only 20% for training.
2

Micro Data Centers for Inference

Smaller, geographically distributed data centers are proposed to handle inference loads, placing compute closer to end-users for reduced latency and improved performance in real-time AI applications.
3

Leveraging Existing Grid Infrastructure

EPRI's strategy involves utilizing underutilized electrical substations to host these micro data centers, tapping into existing excess capacity (typically 3-10 MW per site, scalable to 20-25 MW projects through distribution).
4

Benefits for Grid and Users

This approach offers a 'win for everyone' by leveraging sunk-cost infrastructure, reducing societal costs, accelerating deployment timelines, and providing grid flexibility through integration with energy storage and dynamic load management.
5

Impact of AI Agents on Load Patterns

The emergence of autonomous AI agents running tasks, potentially at off-peak hours, is expected to create new and evolving load patterns for inference, necessitating adaptive energy solutions.

💬 Notable Quotes

"If you look at like the lifetime of a model... only about 20% of its like compute capacity and thus its power consumption is in the training site. 80% of it is in the inference."
"Having the huge mega data centers that are centrally lily-occated don't necessarily make sense for the inference data centers because they are more consumer centric and user centric."
"If we can get extra capacity if we can get extra usage out of existing assets then that's sort of a win for everyone. If a societal cost, if we're not having to put new steel on the ground then that's helping keep rates lower and things like that."