Transcript
INTRO
Jamie Nolan: Hi everyone, good morning. I'm Jamie Nolan, here with my co-host Jigar Shah. Thank you all for being here bright and early. I know we had an event last night, so it was a little bit of a late night, but here we are to see all of you beautiful people.
Today we're talking about the physical infrastructure behind something that can feel almost invisible: AI. Every answer, image and calculation happens somewhere, and that somewhere needs electricity, cooling, and a place to be built.
Joining us are Josh Parker, head of sustainability at NVIDIA, and Jim Gao, co-founder and CEO of Phaidra. Thank you so much for joining us.
The description of this session makes a provocative claim: AI is creating an energy problem, and AI is also the only tool precise enough to solve it. We're going to test that claim and ask what solving the problem would look like for everyone outside the data center.
CRISIS OR OPPORTUNITY
Jamie Nolan: So, Josh, I want to start with you. When people hear "data center energy crisis," they could be hearing several different things: not enough electricity, not enough room on the local grid, higher bills, or a bigger environmental footprint. Which problem do you think is the most urgent, and which can your technology solve?
Josh Parker: It's hard to limit myself to one answer there. I think the premise of the question is valid, but I would quickly pivot to characterize it not as a crisis but as an opportunity.
If you think about economic development, starting from first principles, as our CEO likes to talk about: is load growth a good thing generally for the United States? And is it good for the consumer, for the ratepayer? I think the answer to that, at least in the long term, is yes. Load growth is associated with economic development. If we want more productivity, if we want more economic development, new energy load is a good thing. And again, I think it's good for ratepayers, because it spreads the costs of infrastructure across a wider rate base. It also gives us an opportunity to upgrade the grid.
And if you think about what type of load growth you would choose, one that would be good for the clean transition and good for ratepayers at the same time, I think having companies like the hyperscalers leading the charge is a really good thing. The hyperscalers, so think of Google and Microsoft and AWS, have and continue to have some of the most ambitious clean energy and climate commitments out there. Just last year, they represented 75% of the clean energy corporate contracts in the United States and almost 50% globally. So these are the biggest consumers of renewable energy, the biggest consumers of clean tech, and they're investing, along with NVIDIA, in next-gen clean energy companies like Commonwealth Fusion. We're working on fusion, and we're getting closer. Advanced fission, geothermal, Redwood Materials, which is recycling EV batteries to support solar and wind backup. Really across the board.
This is challenging, because ChatGPT took the world by storm and we still have supply chain constraints. It's challenging to build out the infrastructure that we need, especially because we haven't upgraded our grid in a long time. So there are some expenses associated with that. But where we get at the end of this upgrade cycle is going to be better for ratepayers, it's going to be better for economic productivity, and it's going to accelerate the clean transition.
TOKENS PER WATT
Jamie Nolan: Great. Jim, where would you put the emphasis?
Jim Gao: I see it as all different symptoms of the same underlying problem, which is efficiency. NVIDIA has this concept of tokens-per-watt efficiency, and for me, that is the holy grail. Focusing on efficiency is good for sustainability, but it is also good for business economics. If you trace the root cause of the demand for power, the lack of grid space, the need for more and more tokens: if we can substantially improve tokens-per-watt efficiency, if we can generate more intelligence for every watt, that is undeniably a good thing for business and for sustainability overall. So for me it really boils down to this: we have to focus on massively improving the amount of tokens we can generate for every unit of energy input.
INSIDE THE AI FACTORY
Jamie Nolan: Jim, Phaidra uses AI to manage data center operations, including cooling. So take us inside one of those buildings. What is your system watching? What decisions does it make, and what changes when you turn it on?
Jim Gao: Absolutely. Let me ask a question: how many folks have been in a data center before? All right, this is more than I thought there would be. This is pretty cool. I'm surprised.
I think the most important thing to mention is that data centers used to be perceived as buildings, like commercial buildings. But now data centers are really factories. Jensen calls data centers AI factories all the time. Well, what is a factory? A factory is an industrial process where electrons come in, ones and zeros come in, and intelligence in the form of tokens comes out.
As with any type of factory, there is a lot of data associated with it. A typical data center will have tens of thousands to hundreds of thousands of sensor trends inside it. As a former data center engineer and operator, I can tell you from firsthand experience that is a lot of data and a lot of complexity for human brains to manage alone. So yes, I do think AI is necessary to drive massive efficiency improvements, because a human operating solo can only absorb so much data, so much complexity, before it overwhelms even the smartest human. AI has a really good opportunity to find the hidden correlations across hundreds of thousands of sensor trends and dramatically improve the energy efficiency of these massive AI factories.
Jamie Nolan: Can you give us a real example, with a before and after? Are you reducing the electricity used for cooling, or the whole building's electricity use? Make it real for us.
Jim Gao: Absolutely. This is a very fitting example since we're here in New York City. One of our largest customers is CoreWeave, which is also a really close partner of NVIDIA. The CoreWeave folks worked with us to deploy an AI agent that manages the liquid cooling systems for their NVIDIA chips. These are very heat-intensive systems, so you've got to extract the heat.
This liquid cooling agent reduces the thermal spikiness of the chips. When large training jobs kick off, you get spikes in power consumption, and that leads to spikes in the thermal profile of the data center as well. That's a bad thing, because large spikes don't just throttle your GPUs, which means you're reducing the token output. They also mean that people subcool the system. If you don't know when the spikes are coming or how large they're going to be, you're going to run the data center as cool as possible. And this isn't a CoreWeave thing, this is an entire industry thing.
What we've been doing with NVIDIA and the CoreWeave folks is developing AI agents that can predict when the spikes are going to happen, and then we eliminate the spike, like a really smart MPC controller, before it even happens. The effect is that you get higher token output, but you can also increase the temperature at which the AI factory operates. And that has a massive impact on energy consumption. The single largest source of energy consumption after the IT power itself is cooling.
AI'S SHARE OF GLOBAL EMISSIONS
Josh Parker: We should be talking about the footprint of AI. I welcome that discussion, and we're trying to provide more and more transparency about the emissions associated with AI and the energy consumption.
We're here at Climate Week because we care about climate and we're focused on climate change. It's a global problem. If you look at the numbers published by the IEA and the World Economic Forum and others, more and more independent analysts are concluding that AI can drive net emissions reductions, because it drives efficiencies in other, much larger sectors. AI probably accounted for around 0.1% of global emissions last year. You wouldn't know that by looking at the headlines, because people see the infrastructure going up and they have questions about the emissions. But it's dramatically less than 1% of global emissions.
So should we be focused on the footprint and trying to keep it as small as possible, through efficiency and by using clean energy to power the data centers? Absolutely. That's what we're trying to do. But AI has the potential to drive net emissions reductions. We're seeing that, and like I said, more and more people are agreeing with that. At an event last night, Secretary John Kerry said he fully believes that. And John Kerry, of course, cares about the environment. This is something he is very sincere about, and I'm very sincere about it as well. If we use AI as a tool to help us solve sustainability challenges, it will help us solve climate change.
DO WE NEED GIGAWATT DATA CENTERS
Jigar Shah: We do have this challenge where there isn't clarity. Part of the challenge I see right now is that you've got a bunch of hyperscalers saying, "We want to build seven gigawatts in the middle of Pecos, Texas," which they're never going to build, and Amazon knows it, but whatever. And on this side, you guys have been really aggressive at working with Crusoe on their 20 megawatt format, working with Prologis on their five megawatt format, working with American Tower and Crown Castle on the 200 kilowatt format. So at some point, what responsibility does the AI community have to say inference can be sub-20 megawatts? That we don't need one gigawatt data centers the size of Cincinnati going up everywhere, which by definition are hard to integrate into the grid?
Josh Parker: That's a great question, Jigar. I think we are still in a period of semi-chaotic innovation right now. And that's good, because we're trying all sorts of ideas. We're throwing everything at the wall to see what sticks: what ends up being efficient, what ends up being practical, what ends up being sustainable, what ends up being pro-community. It feels like a long time since ChatGPT first came out, but in the world of AI, things happen very rapidly. The innovation is still very, very new, and we're still trying to figure out what makes the most sense. So I think right now we should all give ourselves a little bit of grace to say we're trying all the things. Let's try all of them.
And I like how you mentioned SPAN. That's one of the examples where we're going as small as we can. You take a GPU and you put it on the side of a house. We're partnering with SPAN and with Pulte Homes. Very, very small footprint. Part of the idea is that you've got distributed computing, and it's more efficient at using energy locally when it's available. So you can arbitrage energy consumption and spread out the inference across a much larger area. That might be a huge solution that we end up pursuing, because it makes a lot of economic and energy and sustainability sense.
So right now we're trying everything. I think that's the right approach. Where we end up, I don't know, but I think it will be a mix. Sometimes it'll make sense to have a big data center that's running big training runs or doing massive inference, and in a lot of cases I think we'll have something much closer to home.
Jim Gao: I strongly agree with that. I think a lot of folks don't recognize that there is no single silver bullet. We have to come together with a lot of different solutions. It really is a proverbial big tent of ideas. You've got the clean energy sources that Josh was just talking about, with the hyperscalers driving 75% of all renewable energy contracts in the US. You've got grid-scale battery energy storage, and data centers are now one of the primary drivers of grid-scale battery storage. You've got folks like Phaidra who are dead focused on energy efficiency and maximizing tokens per watt. You've got other folks focused on grid flexibility, ensuring these assets can flex with the grid. You've got Redwood Materials using recycled car batteries to drive more sustainability from batteries that would have been thrown away anyway. This is the entire industry coming together to come up with solutions.
MEASURING EFFICIENCY WHEN DEMAND KEEPS GROWING
Jamie Nolan: Okay, Josh. If you make AI cheaper and more efficient, people may use more of it. So how should we judge progress if the energy needed for each task falls but total electricity demand keeps rising?
Josh Parker: That's the really difficult thing: measuring the improvements we're making over time in efficiency per workload. That's what really matters. If you're doing the same workload, how much energy does it consume, generation over generation? The challenge is, again, because the innovation is so rapid, those workloads are changing as rapidly as the infrastructure is. I think we're all probably using more agents than we were a year ago. We're doing much more complex tasks. And if you try to look at it at the token level, tokens aren't identical from generation to generation. A token on the newest models is not the same as a token on previous models.
So the best I think we can hope for is to look at the workload level and try to come up with the same type of workload, and it's really hard to control for variables. You say, okay, I ran this workload a year ago, so it was on older infrastructure, using older networking, using a different model. And then you run it again today, and all of those variables have changed: newer infrastructure, newer networking, newer model. You don't know exactly where the efficiencies are coming from when you're looking at that level, but I think that's the only way to really compare gen over gen.
We do look at throughput per megawatt hour, tokens per second per megawatt hour, as a proxy. If you isolate it to the infrastructure, that helps us assess, at least in that realm, what the efficiency gains are. That has its own limitations, but it's kind of the best we can do for now.
WHEN THE CHIPS CAN FLEX TOO
Jigar Shah: Jim, one of the things I'm curious about is that NVIDIA, Google, Emerald AI and others all came together on demand flexibility. And the way they came together was not just to talk about it, but to hire a bunch of regulatory lawyers to go into the Federal Energy Regulatory Commission, PJM, et cetera, to get all these demand flexibility rules finalized. They were already finalized in Texas. As a company that sells into this market, how do you think about all these rules, how they help your business, and whether you're capable of doing it?
Jim Gao: Absolutely. Look, at the end of the day, it's a good thing that folks recognize the need for flexibility. Speaking as a longtime data center engineer: historically, IT loads were considered static, because you had mostly CPU-based cloud compute. Your IT load was fairly predictable over time. What was not static, what could flex, was the cooling. Cooling is usually considered the sacrificial load. It's the second largest component of data center energy consumption, usually around 30% of the total. So historically, the cooling load has been a lot more flexible.
What's changed over the last few years is that we have a new type of compute architecture, GPUs, that's dominant today, and the IT workloads themselves are also very dynamic. NVIDIA itself is starting to introduce a lot of new tools, like what NVIDIA calls DSX Max-Q, which enables these GPUs to operate at different power budgets without sacrificing the underlying workloads. So now you've got cooling-side power flexibility, which has always existed, but increasingly the IT side is becoming flexible as well. That opens up a huge aperture for tools like Max-Q to treat these assets as flexible grid assets.
HALF A PERCENT, 100 GIGAWATTS
Jigar Shah: Josh, there are 36 governors running for election this fall. Many of them have put AI on the ballot, and I think a lot of what they're talking about is grid flexibility. Texas passed Senate Bill 6 a few years ago, then they have CLR, they've got all these rules that basically reward you for flexibility. So when people call you, how much flexibility are you saying you can achieve, with Phaidra but also with all the other things you're seeing? I know you guys have pilots with Emerald AI and others. How much are you willing to say to the governors: this is how much we know is possible, and this is where we're trying to get to?
Josh Parker: It really depends on the type of data center you're building and what it's going to be used for. Because of what we do, we generally provide the infrastructure to the data center operators that are building them out, so we're not typically servicing customers on a data center ourselves. There are some workloads that you don't want to be flexible. If that data center is providing infrastructure to a hospital or some other important workload, you don't want that one to be flexible. You want the other ones to be flexible. But there are a lot of workloads that can be preempted, that can be postponed, that can be shifted to another location temporarily, or where you can temporarily pull from local battery storage instead of from the grid to help manage the grid's peak hours. The Tyler Norris paper from Duke University suggested that 1% flexibility in data centers could unlock 100 gigawatts.
Jigar Shah: Half a percent.
Josh Parker: Half a percent, thanks. We can soak up more of that power, convert it to intelligence when it's available, and then turn the data centers back up when the grid is below peak.
And this is consistent with what we were saying earlier about the need for a variety of implementations of AI. Some that are small, some that are big. We also need some that are tailored to non-preemptible workloads, and some that are tailored to be very, very flexible. But I always have optimism that when market incentives are aligned with sustainable outcomes, we're going to get there eventually. We've got some transaction costs, we've got some logistics to work through. Organizations that have truly important non-preemptible workloads may end up paying a little bit more for those workloads, and it may take them a little bit longer to get online. But if companies can be flexible, they'll have economic incentives to do that, because it makes sense for the energy developers too.
ARE THE INCENTIVES ALIGNED
Jigar Shah: But I'd push back on you. I don't think the market incentives are aligned with sustainability incentives. Let me give you some raw numbers. Say you have a five megawatt data center. At $25 million a megawatt of capex, on the low end, that's $125 million. The one to two hours of battery backup they're going to put in, to be able to run that thing off grid, is $2 million. Getting that to 12 hours of battery backup is $6 million. It literally doesn't matter. But they're not going to 12 hours, because the market incentive is to get a one-year payback on the $125 million. The market incentive is not to figure out how to integrate Phaidra and 12 hours of battery storage and all that stuff.
So in some ways, I think a lot of what voters are saying is that this stuff just needs to be mandated onto the infrastructure. And you and I both know the people they're mandating are not the data centers. They're mandating the utilities, who the data centers don't want to piss off. The data centers are like, "We'll do whatever the utilities tell us for speed to power." And the utilities are saying, "I don't want to put in 12 hours at the data center, because I want to build that new natural gas peaker plant." So I wonder how much responsibility this new group has to fight utilities, because each individual data center doesn't want to stick its head up and pick a fight with the people giving them speed to power.
Josh Parker: I wouldn't say that every sustainable solution is going to be economical. That's certainly not the case, and there's going to be an optimum solution. But being more efficient with energy and being more flexible are things that support cost reduction, they support the highest use of capital, and they support resource conservation. Those economic incentives, to keep energy use as limited as possible and to prioritize the workloads that need to be prioritized, are aligned. We still have work to do in figuring out how to coordinate with utilities, and to make sure the incentives are there for them to participate and deploy the upgrades we need to build this out. But in the medium term, I'm optimistic that we're going to get there. I think policymakers care about this. There's certainly a lot of economic opportunity available if we pursue this properly. So I think we'll get there, but it's a challenge in the near term.
Jigar Shah: Jim, I'm interested in your thoughts. How do you think about this? Because I love me a good ratepayer protection pledge. But at some point, don't these data centers just have to be mandated to use a technology like yours to make their cooling more efficient? Is it really just an opt-in thing that's going to work?
Jim Gao: It's a good question. Speaking as a company that is literally selling into the industry, as you pointed out, I personally think the incentives are already aligned. I agree with Josh. This may be a crude way of saying it, but nobody I know wants to pay NVIDIA a billion dollars for more compute. If you could get twice as much... But that's the thing. If you could get the same amount of intelligence for massively less compute and massively less power, of course you would do it. Nobody wants to build new infrastructure if they can avoid it. But the demand is there. We're all using ChatGPT, and it's creating so much value for us today.
The point I'm trying to make is that efficiency is good for sustainability, and it's also good for business. I think the incentives are already there. But like Josh said, we're literally three or four years into the largest infrastructure buildout humanity has ever seen. We are figuring a lot of stuff out as we go along, and there are a lot of solutions out there. And we have to remember this is mission-critical infrastructure. I get why folks aren't trying out 20 different new solutions all at once. Maybe they're trying five, because these are mission-critical assets that can never go down. There's an adoption curve the industry has to go through as well. There's only so much innovation you can absorb all at once.
CAN A DATA CENTER LOWER YOUR BILL
Jamie Nolan: I want to make sure we get to community benefits, because I think this is a really important topic right now. A new Virginia poll commissioned by Deploy Action and the Virginia Distributed Solar Alliance found that 77% of voters oppose data center development in their local area. As we've seen today, there are lots of very strong opinions about this. But 76% support developers funding improvements that lower nearby residents' electricity bills by at least 20%. So what would it take for a data center to lower its neighbors' power bills, and what responsibility does the industry have to help make that happen?
Josh Parker: This is a challenging area, because to a lot of us it feels counterintuitive, the impact that new data centers can have on ratepayers' utility costs. Lawrence Berkeley came out with a study a couple of months ago looking at the connection between large new loads like data centers and increases in utility bills for ratepayers, and they found no strong correlation between the two. If anything, sometimes the load growth tended to reduce overall rates for consumers compared to what they would have been otherwise.
Unfortunately, we're in a period where, because we've had stagnant electricity demand in the United States, we've gotten out of the habit of investing in our grid infrastructure and making the upgrades we need to have a modern infrastructure and support expansion and load growth. A lot of the upgrades we need for a resilient, intelligent and efficient grid are ones we haven't paid for yet. So we're in a period of some inflation, and there are some extra price increases that ratepayers are seeing because of the delayed upgrades and the new load growth.
With the ratepayer protection pledge, I take the signers of that at their word. The companies that we work with are very committed to protecting ratepayers from any impact from data centers.
Jigar Shah: I think you mean the data center people. I would definitely not take the utilities at their word.
Josh Parker: Yeah, I'm talking about...
Jigar Shah: I mean, they deliberately try to take the prepayments that the data center companies are providing and slip them in as revenue so they can raise rates on ratepayers. Entergy Arkansas just raised rates by $5 a month instead of using the $700 million that Google provided them as an offset to the infrastructure, because it served their shareholders. It kind of sucks.
Josh Parker: You know more about utilities than I do. But the desire from the partners we work with directly to shield ratepayers from increases is sincere. It's real, they're willing to do it, and we're trying to support it to make sure that doesn't happen. It's hard to see, though. Rates are going up across the country. My rates are going up for my house. Is that tied to data center growth? We're doing everything we can to make sure that doesn't happen. But again, because we're in this period of delayed investment in our energy infrastructure, it's hard to see a path forward where we have a resilient grid that supports economic development without those changes.
But with the innovations we're looking at, AI helping to manage the grid more smartly, lower costs of renewables than we've ever seen before, better battery storage than we've ever seen before, now is a fantastic time for us to be upgrading the grid and creating the infrastructure we need for virtual power plants. In time, hopefully very soon, that will lead to lower electricity costs for all of us, consumers as well as organizations, so that we can have the benefits of energy abundance.
PREDICTIONS FOR NEXT CLIMATE WEEK
Jamie Nolan: Wonderful. We only have a few minutes left, and we love predictions on this show. So Jim, when we're back at Climate Week next year, what's one measurable change you expect to see that would show these solutions are working beyond individual demonstrations?
Jim Gao: I think the most important thing to note is that we're well past the demonstration stage now. As of today, Phaidra has over a gigawatt of AI agents helping our industry partners manage their data centers to massively reduce energy consumption. When I come back next year, I hope to be at over five gigawatts deployed of live data center capacity, and I hope to tell you all about how much CO2 and energy we're saving.
Jamie Nolan: Love it. Josh, any predictions? Where will we be this time next year?
Josh Parker: We just crossed a significant benchmark at NVIDIA, transitioning our entire enterprise stack over to not only direct-to-chip liquid cooling, but also to this new technology we have called S45, which refers to the 45 degrees Celsius intake temperature for the cooling system. That's 113 degrees Fahrenheit. So the coolant we use to cool our chips is 113 degrees Fahrenheit.
The rollout of that technology, I think, is going to transform the conversation, not only about water, because these systems are exceptionally water efficient and could eliminate the need for water consumption for cooling in a lot of geographies, but also about energy, because they're dramatically more energy efficient. That's a 10 degree Celsius differential between the industry norm at 35 degrees and where we are at 45 degrees Celsius. So I think more and more people are going to recognize that these are very different types of data centers than we've had in the past. It's much, much easier to make the case for sustainable data centers when you have that type of innovation. I think we're going to hear more about it as people realize these innovations are driving sustainable outcomes.
Jamie Nolan: Fantastic. We'll look forward to that. Josh Parker and Jim Gao, thank you so much for joining us.