OpenAI’s Compute Chief: We Can’t Build Fast Enough | Sachin Katti

Thank you. Great to be here.

It definitely feels like one of the largest things humanity has ever built, effectively. Definitely bigger than many of the things that I've heard of. I'm not old enough to have experienced the highway buildout, but it feels exactly like what it sounds like. I'm in the belly of the beast, so to speak. Every day, we are making decisions on compute that historically, from my previous role, for example, at Intel, we'd probably take months to make, given the magnitude of those decisions.

But the demand is so insatiable, and it is growing so rapidly that we have to move very quickly. So it's an intense time, but it's probably the most exciting thing an engineer would want to be part of.

Directionally, that sounds about right.

And it's probably continuing to grow, right? And a lot of build happening. So a lot of that is also going to translate to compute usage from people like us in a year or two.

Yeah, I mean, I think OpenAI always has had a fundamental belief that compute is at the foundation of everything, right? Compute is the foundation for intelligence. And the way we keep continuing to scale intelligence and distribute intelligence is by having compute. And so that has never been different. That's always been the belief. I think what's becoming clear is, to build the kind of compute we need, and at this scale, we have to not just rely on getting compute from our partners; we increasingly have to take a much more active role in building and getting that compute that we need.

So it does absolutely feel like a new muscle that we are building in the company.

I think the biggest probably is the scale, right? So we are essentially building large supercomputers as we think about AI. And as we build intelligence and deliver intelligence, and models become more capable, we use it for more and more complex tasks. We need bigger and bigger computers, effectively. And so I think the best way to visualize data centers is giant factories, right, that are turning electrons into tokens. It's a popular phrase nowadays, but it actually has a lot of ring of truth to it.

So how do we take power, how do we take those electrons and actually use them to power chips that effectively are delivering intelligence? And the way I visualize it is large football fields, liquid-cooled because these chips run really hot. The temperatures on these chips are very, very high. And so you cool them with liquids; you can't cool them with air. So a lot of liquid-cooled, basically refrigerators, effectively, that are sitting alongside the building.

Both.

Right, so you need to cool the data halls, but you also need to cool the chips individually, because it's not going to be enough to do one or the other. And you also have to cool the things that connect chips, right? And so that's why you need cooling pretty much everywhere nowadays. Even the cables, the transformers that distribute the power, become too hot, so they also need to be cooled.

I think liquid cooling has been around for some time but has never been deployed at this scale. And so the innovation is more around how to make it reliable, how to make it cheaper, more scalable, right? So there's a lot of innovation around that. There's also a lot of new innovation, new kinds of liquids, new kinds of materials that can absorb heat better, because anything that can improve the efficiency of heat transfer is very important for data centers.

So we can then run the chips hotter, right? And there is a direct correlation between running a chip hotter and how powerful the compute is. So the hotter the chip, the more memory bandwidth you get, the more FLOPs you get. And so there's a strong payoff. If you can cool well, that also means you can produce more intelligence.

I think in the early days, we all connected to the grid, and we still all would want to connect to the grid. At this point, we are beginning to hit, and we are investing in generation infrastructure for the grid, transmission infrastructure for the grid. So whenever we build a data center anywhere, we make it a hard commitment that we are not taking power away from the grid. In fact, we are investing in the grid to generate new power so that we can consume it for data centers.

You have a grid somewhere. It has a certain generation and distribution capability, a certain number of megawatts. Obviously, a data center shows up. If there was spare capacity, then of course the data center can use it. But if there isn't spare capacity, then we have to add new gas or solar or hydro generation infrastructure to the grid. So we are investing and funding that build-out, and then you have to build transmission lines, invest in transformers, substations to distribute that power.

So wherever we are building data centers, we are funding the development of all of that infrastructure. And so that's one of the things that we do want to emphasize, which is this is infrastructure that would otherwise not have been funded if not for these data centers. And one of the side benefits of this big data center build-out is the grid infrastructure of America, and the whole world for that matter, is getting upgraded very quickly. And so that's the power piece. So whenever we can do that, we do that and we consume power from the grid, but we are good citizens of the grid because we are improving the infrastructure for everyone.

Not just for data centers, but also for households. In some places, we are beginning to hit the limits of how much grid power we can build and consume. And so, everyone's looking at behind-the-meter. We are also doing some behind-the-meter generation, where we would have on-site power generation and distribution capability that does not come from the grid, where, in fact, the data center becomes effectively self-sufficient in terms of power.

Today it's gas turbines, especially in the U.S., because that's the most dense, transportable form of energy, and also the one that is quite widely available in the U.S. But there, we are bottlenecked by the supply chain.

It can't come soon enough. I think it is the densest form of energy we can all produce and consume, and it's also clean. So I think definitely would be a good source of massive, scalable energy for data centers. Obviously, outside of France, the rest of the world has a lot of catching up to do in building this infrastructure, but I think it's going to play a very important role.

As we begin to serve a pretty big fraction of the world's population, AI usage is exploding. Inference is obviously becoming a big fraction of our workload, and it's consuming a lot of compute. And one of the other realizations is because we know what the workload exactly is, what the model we want to run is, we can co-design the hardware to be super efficient in delivering those models. And so the strategic thesis we have at Jalapeño is: how do we take advantage of knowing what the end workload is, what the model itself is, and design chips that are very efficient in serving those models?

So it really allows us to drive an efficiency advantage, drive more tokens per watt. So the key metric that Jalapeño is optimizing is maximizing the number of tokens you can produce per watt. And because the world is constrained by power today, the more tokens you can produce for the same amount of power, the better it is for everyone. So we look at it as a very critical ingredient in scaling how we deliver intelligence to the world. Great.

No, inference is big, perhaps even the majority of compute. And I think we don't like to make a distinction between training and inference because a lot of training is now inference. So when we train a new model, we are generating synthetic data, for example. That's inference. When we train a new model, we are doing post-training.

And that's inference. When you train a model, you're doing test-time compute. That's all inference. So when we say training, a lot of the compute actually is inference, even in that phase of the work. So inference is a fundamental building block.

We have deep conviction in scaling, and history has borne us out. So effectively, our revenue, for example, has tracked the compute. We tripled compute, and we tripled revenue. And we believe that continues to be true. Demand far outstrips compute supply today. So anything we can bring online, we consume immediately. So there's no compute that is going to waste, for us at least. So I think that conviction has not changed whatsoever. And if anything, we are seeing that scaling laws on research and training continue to hold, and potentially the pace at which we are doing research is accelerating because of AI itself.

So, AI is doing a lot of AI research now. And so, one of the subtle implications of that is previously our researchers used to run experiments, and they needed compute to run experiments, but the number of experiments they could run was limited by the number of human researchers they had, which is a scarce resource in the world. There's not a lot of people who can do AI research. Now, if AI itself can do AI research, the number of experiments we can run explodes.

And therefore, the amount of compute you need for research also explodes. So, we don't see a world where we will have unused compute for the foreseeable future.

When I was referring to surprises, my worry is more on the downside of we are not able to actually build all the compute we want.

It's the other way for us, right? Because that is consistently the case. Any time we have thought we have enough compute, we can slow down, it has always negatively surprised us, like, oh shit, we should not have slowed down, right? And so our biggest worry is that still. And at the scale at which we are planning to get compute and build compute, the physical world does not move that fast, right? Physical supply chains, factories don't move that fast, cannot add capacity that fast.

So for us, the surprise is more in that direction than the other direction.

I think anytime there's new technology which is as revolutionary as this technology is, there is always disruption that's going to happen. But inevitably, we have learned this over history that this always leads to better outcomes for society, right? And so, how do we draw a line from where we are today to that outcome, right, and explain to the world why this is the trajectory we all need to be on? I mean, it's our responsibility to do that. On the communities front, there's a little bit of a local-versus-global issue, right?

On the communities, I think data centers are, even today, a net positive to every community because we are building these data centers in rural areas of America, for example, right, where there's nothing else that is being built on this scale. So, we show up in rural Texas, we build a data center, that produces new property tax receipts for the community, that funds schools, that funds hospitals. We show up and we invest in new grid infrastructure, which otherwise would never happen because there's no demand.

So there's a modernized grid that that area can enjoy. Basically, we produce jobs.

And so I think one of the things we are investing a lot in is explaining the local benefits every time we build a data center somewhere and making sure that it is well understood, the kind of upside that this has. And data centers, once they're built, are essentially very clean citizens, right? They don't produce any gases or toxic chemicals or anything like that, right? They're self-contained. They just produce intelligence.

These are liquid-cooled, and the liquid is recycled. So, the water consumption of a data center is shockingly small relative to household water consumption. So, I think as you put it, it's been debunked. It's a misperception that data centers consume a lot of water. If anything, they consume so little water for what they do, and all of that water is recycled. So we don't net consume new water. Once we get to a particular point, the water just gets recycled as we use liquid cooling.

It's a closed loop. Yeah, it's a closed loop. It's a closed loop.

So many factors. So one is, of course, land, like plentiful land. Number two would be permitting. Like, can we build these things? And we want to build these things such that they are not affecting any neighborhoods, right? So land that is somewhat removed is the ideal candidate. Of course, access to power, right? So a strong grid, strong gas availability, all of those are important factors. And then four is labor, right? So how quickly can you build these things?

So availability of labor, construction labor, qualified electricians, plumbers, all this has to be available. So all of those factors go into every single site selection decision. And I know obviously Texas has been popular because it fits a lot of these criteria, but it's not the only state. I mean, we have data centers all around the country. Okay, great.

Yeah, I think of it as my role and our team's role, rather, as how do we bring compute online at industrial scale, right? That's effectively what we're doing. And that's the entire lifecycle. So, how do we find the ingredients that go into compute? Land, power, shells, chips. How do we finance them, right? Because these are massive dollars. And so, how do we make sure that we finance the grid infrastructure? How do you finance the construction of the compute shells?

How do you finance the chips? Then it's about how do you operationalize all of this? So how do you actually make sure these things happen on time, they stay up? How do you operationalize all of this infrastructure? So it's that entire lifecycle. And then, of course, how do you actually use the compute? So a big part of my role is capacity allocation inside OpenAI. So it is always a scarce resource.

I am not very popular. There is always someone who is unhappy with whatever decisions you make. But yeah, our team provides the input to make the capacity allocation decisions. So we surface what are the different choice points and what are the what-if questions, different allocation choices that we have. So capacity planning, and then, of course, using that to forecast how much capacity we need where, because it's not just more compute, it's also where, what kind, what shape, what chip, what workload you want to run there.

So this team figures out what should be the forecasting and planning, and that closes the loop.

So that informs where do I go find the next chunk of land and power and chips to put into.

It's a portfolio approach, right? So we are never going to be in a world where we outsource everything or build everything ourselves, right? It's always going to be a mix because that's the reasonable thing to do, right? So you don't want to put your eggs all in one basket. So we will have hyperscalers probably providing a big chunk of our compute, a majority of our compute. We will have new clouds as parts of our portfolio. We will be partnering with design-build firms that can build the compute that we need.

And of course, we may build some of it ourselves. And so we are always going to have a portfolio approach because, at the scale which we need, we will need to tap into all sources of compute. We can't just rely on one particular mechanism.

A bit of all of the above. But yes, at my heart, I'm an academic. So I've been a professor at Stanford since 2010. Recently—

I was faculty in computer science and electrical engineering.

Yeah, my area of research was networking. So I did build networks, both mobile wireless networks other than cellular and data center networks, actually. But three or four years ago, while I was at Stanford, I did a couple of startups. The last startup got acquired by VMware. And that's how I got to know Pat Gelsinger, Intel's CEO. And that's how I ended up at Intel. Most recently, before coming to OpenAI, I was Intel's CTO. And so I've kind of seen all the different things: academia, startups, corporate at Intel, and then, of course, a mix of all of the above at OpenAI because we have a research lab, a startup, and a fast-growing company all mixed into one at OpenAI.

Yes.

It's actually what I just said. That makes this so unique, and it's hard to find anywhere, right? Because you always have to choose. But having a world-class research environment coupled with the hardest technical problems—like, we are building the largest compute in the world—and so there are a lot of new problems that we need to solve, but also a fast-growing business. I like problems that sit at the intersection of business, technology, and strategy. And so this is, like, a very unique time in history and a unique role, which is very attractive, obviously.

We have compute from effectively many sources. So Microsoft, obviously, is a big partner, important partner. We also have compute, as we have announced, from AWS and Google. So we have compute from all of the hyperscalers, effectively. We also have compute from CoreWeave, for example, so a neocloud. And then, of course, compute that chip partners are supplying now, like NVIDIA itself directly providing compute for us. So I think that's the mix roughly today. As we go forward, obviously, we'll be building on all of these relationships, but also looking at more options where we design the compute, the data center itself, ourselves, or potentially even build a data center ourselves.

So, all of those are ways of scaling the amount of compute that we have. So, I think coming back to my earlier answer, the answer always will probably be trite, but it's all of the above.

Yeah. I mean, we look at Stargate as our compute strategy, and it is varying degrees of us designing or building the compute ourselves. For example, with Oracle, close partnership, we help them design, we help them with how to operate AI compute, which is, in fact, a very new kind of compute for us. We work with SB Energy, which is public. We basically have co-designed the warm shell with them, and they're executing on that warm shell, and we will be kind of figuring out how to operate our chips in these data centers ourselves, even the new chips.

So, Stargate to us is that umbrella strategy across all of these different things. And think of it as an evolution that we'll continuously be on, because it's never going to be that tomorrow we wake up and do only one kind of way of building compute.

I think Stargate to us is a continuous way of learning how to scale compute, and we'll be adding on more and more capabilities.

Yes.

We obviously have a big partnership with Oracle. That's the Abilene data center. That's where, for example, we are training our newest models. So, very excited about that. That's a very big GB200 Blackwell cluster for our needs.

It's up and running. It's been used for training the last two models, more actually. So we're super thrilled about that, and you're seeing the results, right? You're seeing how quickly the models are becoming more capable. It's because of these kinds of compute.

Yes. So Oracle is building a number of data centers, all of which are public, across Michigan and Texas and other places. So these are coming online in the next couple of years as they get built, and we put whatever chips at that time are the latest for training. But these are really meant to be very big clusters that allow us to do both training, but also product inference-kind-of compute.

Oracle is the cloud.

And we're consuming compute from Oracle.

Yeah.

Interesting.

I mean, with all of this compute that we have today, we have amazing partners who actually are handling that for us. So Microsoft, Google, Amazon, Oracle—we are the offtake. So we are the tenants, as you put it. So we commit to consuming that compute, to buying that compute, whether it's on-prem.

Partners are building.

Correct.

Yes, it was incredibly quick. Nine months is very, very fast, probably the fastest I've seen in my career. I think several reasons. One is it's a team, it's a strong team. Many of the team have designed TPU chips at Google in the past, so a very well-experienced team. We have a great partner in Broadcom that has a very strong track record of delivering XPU ASICs. So I think that strong partnership with Broadcom in making this happen. Third, I think perhaps an OpenAI-unique point.

In most chip companies, or when you design chips, you don't know what you're designing it for because you're a vendor. The customer who eventually runs the workload is someone different. There's a unique advantage here of us knowing what the future models might look like, and therefore being able to short-circuit a lot of the decisions you need to make, design decisions you need to make on the chip side. So that's super helpful. And finally, increasingly, AI itself helping design and optimize the chip.

That is usually the one that takes the longest time, because you're basically limited by how much human time there is to process all this data and run the experiments. And we can do a lot of those iterations much faster using AI. Yeah.

Yes. I think that world is not very far. I mean, AI right now is assisting in chip design, but we do believe that the world of recursion is not that far, where AI will design the systems it needs to train and run the next generation of AI.

Including chips.

It's a new networking protocol, routing technology, if you will, to scale these really large cluster fabrics. So imagine you have 100,000 GPUs. They need to be connected together. And when you're doing large training runs, they're constantly communicating with each other because these models are so large that the processing of the models is happening over the entire 100,000-GPU cluster, for example. And you can imagine the number of links and switches and NIC cards that need to be there to connect all of these chips together.

At this scale, failures are common, right? Happens all the time. You can't really even enumerate all the ways things could fail. So the strategy behind MRC is: how do you design algorithms and protocols that can gracefully mask all these failures and make sure that the training workload does not get impacted? The network is an abstracted system that the training job does not have to worry about. It's always going to be there. It's always going to find a path, even if a link fails.

So it was all about reliability. It was all about availability. So how do we design protocols that can tame the complexity of such a big cluster and make sure that we don't get stopped because of failures, which are very common in the system? So MRC is like a multipath spraying protocol where you can spray packets over multiple paths. So between any two chips, there are many, many routes to get there, kind of like between any two points in the city, there are many routes.

So instead of picking just one route, we will send traffic across all of them. And whichever one succeeds, you take that, right? And so that way, even if any one of them fails, it's not a showstopper. So that's kind of the basic intuition behind it. It's obviously a lot more sophisticated than that, but doing this at scale, at that speed, is hard. And so that's why it's quite innovative.

I think there are bottlenecks everywhere, to be honest, in the supply chain. So I don't think there's any one, right? We have bottlenecks in the data center building itself around permitting, around availability of gas turbines, transformers. Those industries historically have not added much capacity over the last decade or so.

And they've suddenly experienced a demand shock.

And it takes years before you can add capacity to produce more turbines and transformers. So they're trying to play catch-up.

Absolutely. Yeah, absolutely.

No, I think there is definitely a shortage of electricians, plumbers, all kinds of trades, you name it. So anything we can do to train more folks to be able to do those, they are very well-paying jobs that a lot of us, all of the hyperscalers, all of the labs, would actively hire you for if you had the qualifications. So I think that is definitely a bottleneck. I think it'll become an increasing bottleneck because we are all trying to build more and more, and we have a limited number of these capabilities.

Yeah, so guaranteed capacity is guaranteed tokens, right? So we are effectively saying we will guarantee you a certain dollar's worth of tokens of intelligence. And it makes sense, right? In a world where compute is in short supply, tokens are always going to be at a premium, and there's a shortage of tokens that we can produce given the limited compute that we have. And as this becomes a fundamental input to the enterprise, enterprises are going to need intelligence, more and more of it, to run, right?

And so this is a way for enterprises to gain assurance that the tokens of intelligence that they need will be there for them, and so they don't have to take business risk, right? And so I think it's good business hygiene. If you have a critical supply resource, and for every enterprise, intelligence is kind of the most important supply item, if you will, it makes little sense not to make sure you secure that supply. And so that's the demand we are trying to fulfill here.

I know it's a new concept, like, what does it mean to have guaranteed capacity for intelligence? But I think that's what intelligence is becoming. It is becoming a supply unit for every digital enterprise.

For the geek in me and the engineer in me, it's definitely exciting. It's one of those things that can be super cool to see, a constellation of satellites producing compute. I actually think it will become feasible, right? As engineering problems go, they can be solved with enough time and investment. Whether it is needed, I think there is room for orbital compute. I don't think it's going to serve all the compute needs, but it's definitely going to be a component in the arsenal.

I think what we are waiting for is: when does the economics of launching satellites change, and when does the economics of the hardware change? Because we need to get to a point where it's cheap to launch the hardware, and if something fails, it's cheap to throw it away, because you can't go up and fix it, unlike on the ground. And so I think that inflection point hopefully happens soon, and at that time it becomes viable.

Thank you. It's been great to have this chat. Cool.