Inside Nemotron & NVIDIA’s AI Lab | Bryan Catanzaro

Well, it's really exciting to see all of the energy going into open technologies for AI because we know that open technologies make it possible for people to innovate. The internet is such a great example of that. We actually did have closed internets. I don't know if you remember things like America Online and Prodigy back in the day. And they were great. And the open internet has also been amazing, right? So many different companies have been able to figure out how to transform their work thanks to an open technology.

The application of the internet to retail is very different from the application of the internet to healthcare or manufacturing, but all of them have been totally transformed by the internet. AI, I believe, is also a very transformational technology and also a technology that needs to be applied in very diverse ways. And because of that, I believe that open technologies for AI are really fundamental. And it's very exciting to see continued investment and development of open technologies for AI from so many different organizations around the world.

And I hope that continues.

Well, I feel like this question is maybe a tempting question because it's fun to set up kind of competition, but I actually feel like the whole AI community is moving very fast. And if you look, for example, at the progress in AI, whether it's closed or open, just over the past three months, it's been incredible. And so if you're in a field that's moving really, really fast, I think that's more important than any particular gaps that might exist between different models, because the most important thing is: how is AI developing as a field?

I think there's a number of things that are pushing open technologies for AI forward. One is just the demand. There's so many organizations that want to customize AI and want to integrate it deeply into their work in a way that really requires open technologies for AI. And so I think the demand is certainly there. I think also it's just the best way to develop technology. And we've seen this for many decades, that technologies developed in the open move quicker because we can all learn from each other.

And in an era where we're undergoing the most exciting thing to happen in technology in our lifetimes with the development and deployment of AI, what else do computer scientists want to work on other than making AI awesome? And if working together as a community is the best way to do that, then that's also a driver that pushes the community towards openly developing technology.

In my mind, there's no question that when the technology community decides to make huge investments in the most transformational technology of our time, there's going to be rapid progress, and also that that technology is not going to be controlled by a small group of people, because that's just not the way that the industry works. We do our best work, we have the most impact with our work when we're able to each think about it in our own way and apply it in our own way.

So, I love the closed AI APIs, whether from Anthropic or other people. I think they're amazing. I'm really, really impressed with the work that those labs are doing. But they're not the only labs in the world. There's lots of labs around the world, and lots of people have a good idea. It's not the case that there's only a few labs that have the monopoly on all good ideas. That's just not true. That's not how humanity operates.

There's a lot of bright people on this planet. And the community, of course, cares deeply about this technology. It's obviously so transformational, has such profound impacts on so many things that, of course, many people want to be involved in that. And so I think over time, we're going to see that community-oriented approaches to developing and deploying AI are going to continue to strengthen and be widely adopted, because that's really the history of how we build things as a human species.

Perhaps unusually, I actually did work at a Chinese company for about two and a half years. I worked at Baidu. I worked in the Silicon Valley AI Lab along with Andrew Ng, as well as Dario Amodei. We all worked for a Chinese company and saw how smart, hardworking, creative, inventive our colleagues were at the rest of Baidu. And that experience has stuck with me. I think it's absolutely false to say that the achievements of some other country are all being created by a sort of copycat mentality.

It's just not true. Now, do we all learn from each other in the technology community? Of course. Of course we learn from each other. But I would say it's been a really good thing for the world that the Chinese AI community has been so open with what they've been building. I think it's enabled a tremendous number of companies to build things that they couldn't have done without that community. And I think it's also spurred technological progress throughout the AI ecosystem.

So, I'm really grateful for the contributions that our colleagues in China have made over the years. And I would love to encourage a spirit of openness amongst AI labs around the world outside of China as well. I was really excited when OpenAI released the GPT-OSS models a while back. And then, of course, Google's been doing great work with Gemma. Absolutely thrilling to see that. And we're pushing Nemotron along here at NVIDIA as well.

So I think there's a chance for the rest of the world to catch up to China, in the sense that we can understand the benefits of working together as a community to build technologies for AI in a way that I think China has, frankly, been leading. Great.

Every company is built around a secret. This is a secret that has to do with not just their intellectual property, but also their platform, which has to do with how they interact with problems and customers. How do they think about solutions to what their customers need? And it is always the case that the value of AI is greater when it can be more tightly connected with those secrets, because AI depends critically on data. So the more valuable the data that goes in, the more valuable the solution becomes.

Now, every company, when it's thinking about how to deploy AI, has to think through what are the implications for the core secrets of our company. And there's a lot of circumstances where, due to trade secrets, trying to think through the business model, or even regulatory requirements, there's data that you really have to treat very carefully by law. And it is much better to do that when you are able to think that through and implement it yourself, thinking about the integration of AI, the way that AI interacts with customers, the guardrails that are put in place.

Every company has a specific understanding of its customers and therefore what the customer needs. And the amazing thing about open technologies for AI is that they allow customization, right? So companies can think this through, they can build things that really matter for them. And I started out this conversation talking about the internet and about how the deployment of the internet has been done in very different ways for very different industries. And there's a lot of desire to do that as we see AI change the way that we work and play throughout the entire economy.

This is really spurring a lot of demand for open technologies for AI.

So I started work at NVIDIA in 2008. At the time, I was a graduate student trying to figure out parallel computing for artificial intelligence. And I thought NVIDIA had a chance of changing the way computers work with AI.

Oh, it was very chaotic. Back then, people thought I was crazy. And I remember going to ICML in 2008. I published my first paper training models on the GPU, and people asked me why I was there. People said, "This is not a good paper for ICML. We just do fancy math here." And I was like, "Well, but I think computing actually matters a lot for AI. If we could train bigger models that had more capacity to learn, we could probably solve more problems."

And they kind of nodded their heads in the booth, like, "Well, we're not really sure why you're here."

Oh, right. Yeah, there's also that, right? Which we continue to run into, that idea. Actually, a GPU is whatever NVIDIA says it is. We make them. So a GPU is a thing that we make in order to accelerate the world's most important computations, which in 1995 was graphics, and for a long time now it's been AI. So anyway, I started at NVIDIA. I was in the research group doing strange things about trying to make compilers, libraries for AI on the GPU.

That led to the creation of first Copperhead, which was a Python-embedded language that compiled to the GPU, which I think foreshadowed a lot of things in TensorFlow and PyTorch. And then that led to the creation of cuDNN, which was NVIDIA's first product for deep learning on the GPU. And I really enjoyed working on that, but I was always wanting to see more firsthand about the applications of AI. And at NVIDIA, I was mostly working on libraries and compilers for AI.

So I thought, when Andrew Ng asked me to go build the Silicon Valley AI Lab with him at Baidu, I thought, oh, this is a great opportunity, because even back then, Baidu was very advanced in its application of AI to its core business. And so that was a fantastic opportunity for me. The Baidu Silicon Valley AI Lab was an amazing place, full of brilliant people that were working really hard.

Dario was brilliant from the beginning. I remember I interviewed him. I was on the panel, and at the time he had been working in bioinformatics, so he hadn't been working on deep learning or the things that we call AI these days. But it was very clear that he learned extremely quickly, and also that he thought extremely deeply. The thing I admire most about Dario is the strength of his conviction. I've been working in this field for a long time, and I've also believed that AI is going to transform the world.

But I don't think that I believed in it as completely as Dario did. And perhaps that was because my academic training during my PhD was full of a lot of caution. I don't know if you remember, but AI was old and bad in 2005.

That people did with computers. They started doing it in 1945, right? And so there had been so many grandiose promises that failed to deliver over the years. And so I came to AI with a lot of caution. In fact, back then we used to call it machine learning, which was basically a dodge. Like, we just didn't want people to know that we were working on AI because then they would be like, oh, we've heard about that. Whatever works, right?

So I came to AI with a little bit of this academic caution, like, oh, we should hedge a little bit. I don't know if now's the time. And Dario, his strength of conviction and his understanding of the moment, of how the technology was developing—this time it was actually going to work—and then the implications of that on how the technology should be developed, what kind of institutions to build.

I think he's done a spectacular job. And so, yeah, working with him, it was always a fun experience.

Yeah, so 10 years ago, actually, in 2016, Jensen called me up and said, "Hey, would you like to come back and build an applied research lab?" And I thought that would be a fantastic opportunity. I've always loved NVIDIA. I've loved the way the company works, the convictions the company holds. NVIDIA is a very unique company. It follows through over long time periods, and I've seen that with CUDA. I've seen it with our deep learning technologies.

I've seen it with our ray tracing graphics technologies, our AI for graphics, over and over again. NVIDIA is not afraid to put in five or 10 years' worth of research in order to change the world. And working at a company that has that strength of conviction and the ability to follow through is kind of an ideal thing for me. I just really love the support that the company gives its researchers to invent the future.

And so I thought I'd come back. The first project that I worked on actually became DLSS, which some of your audience may know about. But DLSS is our real-time AI for graphics, and it makes a small GPU run like a big GPU. It's about 10 times more efficient because rather than computing the color of every pixel for every frame, we use AI to infer the color. And these days, 23 out of every 24 pixels is being generated by our AI model when you're using DLSS to play games.

And gamers love it. It's become the standard way of playing games because it's just so much more responsive and it's more beautiful. Our AI, we train it offline on huge datasets, and it's able to render graphics in real time more beautifully than traditional methods do. We recently actually announced DLSS 5, which is a fully generative version of DLSS, and I am so excited about it. It represents the culmination of 10 years' worth of research on how to make real-time graphics much more beautiful.

And so that's part of the journey here for me, was real-time AI for graphics. But then at the same time, we also started a language modeling project. And this was back in 2017, before Transformers were big and before language modeling started taking over the world. But I just had this intuition, maybe built on some of the things that I had seen while working at Baidu.

I just had this intuition that working with text and understanding text was going to lead to better reasoning, which was going to lead to better application of AI in all sorts of domains. And so we started this project called Megatron. Megatron stands for the biggest, baddest transformer. That's why we named it that. And it was really a systems project to show the world how to train the largest transformer models on NVIDIA's hardware. Back at the time, some of your audience may or may not remember this, but there were claims made that the only way to train big transformer models was on the TPU, because after all, the transformer had been invented at Google.

And so we looked at the transformer paper. We thought, "Wow, this has amazing potential." We tried it out on our own language modeling tasks, and it worked so much better than the RNNs that we had been using before. And also, we saw immediately that there was an enormous systems opportunity to co-optimize the GPU, the networking, all of the compilers and software that would enable people to scale transformer-based language models really dramatically.

And we thought, this is something that could really have an impact. So we started the Megatron project, which then led to, I think, basically helping the whole industry figure out how to train extremely large LLMs. And also led to the foundations of today's Nemotron project, where NVIDIA trains its own LLMs for its own purposes. So that's kind of the history.

Nemotron has two jobs. The first job is to help us understand how to build the systems of the future. NVIDIA is an accelerated computing company, and that means thinking through the world's most important computational challenges from first principles and designing systems, which includes a lot of software, in order to make it possible for people to invent and deploy things that never could have been done with standard computing. But in order to do that, NVIDIA has to deeply understand everything about how AI works.

That's how we co-design all of the systems and software for our main product line. So the first job of Nemotron is to make sure that NVIDIA continues to exist, so that we can continue delivering meaningful acceleration in an era where Moore's Law has died. And the acceleration that we get these days comes through specialization. But again, specialization comes through understanding. So that's Nemotron's first job, is to help NVIDIA understand how to build its core products. Nemotron's second job is to support the ecosystem.

One of the most valuable things that NVIDIA has built over the years is all of the people around the world who build and deploy amazing AI using NVIDIA's technologies. And we think that it's necessary for open technology for AI to continue to exist from NVIDIA to help support that. Nemotron's not trying to be the only open technology for AI. We love all technology for AI for the very straightforward reason that whenever AI is further developed and further deployed, it's an opportunity for our business.

So we're very explicitly trying to develop our ecosystem because that's good business for us, but we're not trying to be the only provider of technologies for this ecosystem. We love seeing other companies contribute as well. The most important thing for Nemotron's second job is just making sure that it continues to be possible for companies of all shapes and sizes to build and deploy their own AI.

It's been dead for years.

Well, you just look at the progress in semiconductor manufacturing. The original statement of Moore's Law was economic, right? It was about, we can afford to put twice as many transistors on the same chip every 24 months, whatever the time period is. And these days, that is absolutely not the case, and it hasn't been for probably five or 10 years, right? Now, we are still scaling our systems through a number of ways. One is just applying a lot more silicon to it, right?

Transistors are continuing to get smaller and more efficient, although at a slower pace, but they're also getting quite a bit more expensive at the same time. So, in an era where Moore's Law was alive, the best way to make the system of the future was to take the system of the present and then just shrink it and maybe double it at the same time, right? But in an era where we've been living for a while now, where you don't get economic benefits from taking your existing design and shrinking it, you really have to be more clever about how you use every part of the system.

That's an era where accelerated computing is much more valuable than ever, because the work of thinking through the problem from first principles and co-designing absolutely everything from transistors to algorithms and applications in order to reduce waste and deliver meaningful acceleration, that's more valuable than ever.

Yes, yes, yes. The numbering is somewhat lost to time. I almost feel like we're in The Lord of the Rings, and there's some ancient relics that we're digging up out of an old mine. This is a long time ago. What we originally called Nemotron 1 was actually a project that we did with Microsoft. We jointly trained a 530-billion-parameter model.

I believe that was released in 2021. And so this is GPT-3 era, and that's what, at the time, we called Megatron-Turing NLG. Turing was what Microsoft was calling their language model efforts at the time. But that, in retrospect, we called Nemotron 1. Then along the way we built a few more. We got up to Nemotron 3, and then Llama came along, and we were really excited about that.

We were very happy that Meta was supporting the open AI technology space. And so we started taking our language model technology and adding it to Llama models, which then resulted in Llama Nemotron 1. And that was the first reasoning model built on Llama. We were really proud of that.

Might have been '24, I believe. I can't remember, somewhere around there. And then we continued to develop that. Last week, so the numbers kind of started over again. We released a Nemotron 2, I believe it was last year. And then we quickly followed that up with Nemotron 3 because we needed to put MoE support in. Nemotron 2 didn't have MoE support, and that made it kind of uncompetitive against other models like GPT-OSS-20B, which was just so fast because of MoE.

And so we were like, okay, we've got to put the MoE in. So that became Nemotron 3. Now we're in a slightly difficult state because we're working on Nemotron 4, right? But we already released a Nemotron 4, which was in 2024. We released a 340B model called Nemotron 4. And so I'm not exactly sure how we're going to solve this marketing problem. I didn't create this marketing problem, so I'll do my best to make it clear that Nemotron 4, whenever we release that, is different from the 2024 Nemotron 4.

But in any case, we've been working on this for a long time. I think more important to us than any particular generation is just the sustained commitment that NVIDIA has to developing these models. We've been doing it for a while. I think our models have gotten dramatically more useful in the past year, which is a reflection of two things primarily. One is that the whole company has come together. So there are many different teams around NVIDIA that now understand how important this is to NVIDIA's future.

And so there are dramatically more people and better ideas that are going into Nemotron. And then number two, along with that, we've been able to scale the compute resources that go into it. Obviously, it's very important to have good computing infrastructure to build AI. We've recently increased our investment substantially because we believe that this is really, really key to our future. Fascinating. But, just to continue the thought, I think it's really important that everybody knows that we've been doing this for a long time.

We are increasing our investments substantially. And NVIDIA is a company that follows through. We followed through over 10-plus years with CUDA, and we're doing that with Nemotron now.

So Nemotron exists to help support the ecosystem, and we were thinking, well, this is a different kind of AI project than other projects around the industry, right? Because we're not actually trying to dominate in any way. We're just trying to support. We're not trying to control the way that AI is being integrated into all these companies. We're just trying to make sure there's good AI. But we thought, well, maybe if we worked with people while we develop it, then it's going to be more useful for them.

It'll be easier to integrate because we will consider what they need from the beginning. And Nemotron has always been collaborative. I was telling you that, a long time ago, our first big model that we trained, we did with Microsoft, right? It was a joint effort where NVIDIA and Microsoft researchers worked side by side to build that. And that ended up, I think, helping both NVIDIA and Microsoft. I think we both learned a lot from that experience.

And so because Nemotron is not trying to compete with other companies, but rather support, because we're going to be putting it out there openly anyway, why not collaborate before the thing is built? Rather than Nemotron being a project that NVIDIA does all on its own and then posts on the internet and says, "Hey, why don't you try this? We think it might be good," why don't we make sure that it's good for the partners that are interested by working with them before Nemotron is even created and incorporating any sort of feedback, evaluations, environments, benchmarks, or any other kinds of technology that other people want to bring.

It turns out that the entire ecosystem—there are a lot of companies that really want open models to succeed, and so they have a self-interest. They have their own vested self-interest in making sure that open technologies are excellent. And so why not work with them and let them contribute however they'd like to making Nemotron better? So that's the idea of the Nemotron Coalition. It is not an exclusive coalition. We're not trying to be the only model out there. All the companies that we work with are free to continue doing the work however it makes sense to them.

And yet, these companies want to work with us because they want to make sure that open technologies for AI keep developing quickly and that they have a chance to influence how that happens.

Nano is a 30 billion total, 3 billion active parameter model. Super is 120 and 12. And Ultra is 550 and 55. They're designed really to fit small, medium, and large deployment scenarios. Nano can be really capable for things that don't require nearly as much knowledge or reasoning. But obviously, for the most capable model, you go for Ultra. Super, in a lot of ways, is our most popular model because it represents a great balance between cost and intelligence.

So we kind of like having this small, medium, and large approach to building a family just because our customers seem to respond to that pretty well. But the most important thing from NVIDIA's point of view that people are doing with LLMs is agents, right? Building agentic workflows. Having an agent working on your behalf, solving problems for you night and day, is such an exciting way of approaching the problems that we have to solve.

And it's our dream to make Nemotron amazing for that purpose. That's our goal.

That's right. Nemotron has always been a speed-first approach to building models because NVIDIA is an accelerated computing company. As I was saying, we're trying to think through what is the problem here computationally from first principles. And the Nemotron 3 family has a lot of things in it that we're really proud of. For example, Nemotron Ultra and Super were pre-trained using 4-bit arithmetic. We pre-trained those in MVFP4, which is a non-trivial thing to do. To invent the algorithm so that your model can converge to an excellent result using such coarse arithmetic required a lot of invention.

Really proud of that.

Actually, there was a fantastic post I saw on Hacker News yesterday where somebody would let you upload a picture, and then it would basically posterize it, basically reduce the colors to fit different number formats, including NVFP4 and MXFP8 and some of the other formats that are out there. And so you could kind of swipe around and look what it does to the colors of a picture. And it's really quite dramatic. Four bits is not a lot of bits, right?

That's only 16 values. Now, of course, these are all what are called block-scaled formats. So groups of numbers also come with an 8-bit scaling factor. And the specifics of this can get rather complicated, so maybe they're not quite as important. But the reason why we want to do this is because, first of all, we have dramatically higher throughput for these formats in our GPUs, specifically on Blackwell Ultra. And secondly, we know that it's going to save an enormous amount of energy.

One way to think about the computational problem of AI is that we are going to be running at the limit, whatever the limit is. It could be an economic limit, like we only have so many billion dollars to buy servers with. It could be a power limit. We only have so many gigawatts that we can afford to train a model with. Whatever the limit is, we're going to be running at that limit. Every organization is. Because why? Because the value of intelligence is so high that people are gonna invest because they know that they're gonna get return.

The value of intelligence is enormous. So if you accept as the truth that we're gonna be running at the limit, then what that means is that the way to get more intelligence is to be more efficient. We can't get more intelligence by applying more force if we're already at the limit. We have to be more thoughtful about how we use what we have. And 4-bit number formats are dramatically cheaper to move around. They take up less space in memory.

They take up less picojoules when you move them from memory, in front, or even on the chip, around the chip, much less energy when you compute on them. And so that's really driving the investment in 4-bit formats. And I think these days 4-bit formats for deployment are very well established. It's pretty straightforward these days to make a good quantized 4-bit checkpoint that you can deploy and that gets you a lot of inference cost and speed advantages.

But using 4-bit formats for pre-training, that's quite a bit more challenging because you have this numeric solver that's optimizing the weights. And it can be quite sensitive. So if you don't treat the numbers right, your model can diverge. And instead of actually getting a model done through pre-training, you end up with basically just that run diverged, which is always scary. So it took a lot of invention for us to be able to pre-train Nemotron in 4-bit.

We're really proud of that.

Yeah, we published a paper in 2024 that showed that you actually get a smarter model by combining state-space models with Transformers. And we actually did a sweep of how much of the model should be full attention and how much of it should be a state-space model in order to get the lowest perplexity, basically the best language model that you could get. And we found that you actually want it to be mostly a state-space model with a little bit of attention.

And the intuition behind that is that the state-space models seem to be better at this intuitive, impressionistic understanding of a sequence because they're summarizing the entire sequence into a constant space. That's how they work, right? So instead of having the ability to look at the entire sequence randomly, they summarize everything at every step into a constant cache or a little scratchpad that they're working on.

And that constraint seems to actually make them smarter at some tasks that involve global understanding. On the other hand, the advantage of full attention is that it can pick out very specific bits of information and look at those exactly. It doesn't lose anything. There's no lossy compression going on. You can actually see the whole thing. And so we found that using both of these together was actually better than using either one on their own.

And that is independent of the speed benefit. That is just the model is smarter. And since we published that, I think a lot of other labs have also found this to be true. A lot of models these days are being built with hybrid SSM approaches. For example, Qwen has done that. Kimi is using what they call Kimi Linear Attention these days. So it's become, I think, quite widely adopted to use some sort of state-space model in conjunction with full attention for the base architecture.

Now it also has some speed benefits because the amount of memory that you need to hold that state space cache is actually constant with respect to your sequence length, which then means that generally you can fit much higher batches on the GPU when you're training and doing inference, because the memory requirement is lower. And it keeps the GPU fuller and busier, and therefore provides some pretty important efficiency benefits as well.

Mixture of experts is a form of sparsity. The idea is, well, you want to train a model on the entire internet. You want it to remember absolutely everything about the history of everything. But when you're answering a particular question, does it seem reasonable that it needs to actually think about the entire universe in order to answer that question? Actually, no, it seems like it's quite sparse, right? It seems like we're using a language model to explore a very tiny space of ideas in order to answer a question or solve a problem.

We want the model to be able to draw from the entire universe. We want to train it so that it understands everything that it possibly can. But when it's actually running, it doesn't really need to see all of that information. There's been a variety of approaches to sparsity that try to take advantage of this property, but mixture of experts has been the most successful. And the way that it works is that the neural network has what's called a router that is learned, that is going to decide to send activations to a subset of the experts for every token that's flowing through every layer of the model.

It's going to be making choices about which fraction of the model is going to actually get to interact with this token as we try to understand it, build up representations of the problem, and then generate the next token that we're going to output.

That's right. Yeah. Or you could think about it as a library. Like, if you go into a library to do research, you don't read all of the books in the library. Your first job is to figure out which books do you need to look at in order to find the answer to your question? And so that's kind of the idea behind MoEs. Now, MoEs have fascinating implications for the systems that we build. So with Blackwell, for example, NVIDIA went all in on MoEs.

That's why we built NVL72, which allows up to 72 of our GPUs to read and write each other's memory at very high speeds, very low latency. Now, why is that important? It's because as you put a token through the stack of layers, at every layer you have a router that's routing that token somewhere else. Why don't you partition your experts so that not every expert is sitting on every GPU, but you have a subset of the experts assigned to each GPU, and then you're routing the tokens between the GPUs very dynamically as you push the token through the network.

Now, this is impossible to predict in advance, where the tokens need to go, because it's very specific to that particular token for that particular model. And so that's why we built NVL72. And that's why Blackwell is so amazing for inference for today's AI models, is because we thought deeply about mixture of experts when we were building it. And this speaks to Nemotron's first job. If we hadn't been working on understanding AI, we wouldn't have been able to build Blackwell properly.

And that has translated directly into increased deployment of Blackwell, which we're very excited about.

Latent MoE is a specific innovation that we have in the Nemotron 3 family. And what it does is actually reduces the amount of communication that has to be sent through NVLink during MoE computations by basically down-projecting it. So every token produces a vector, and the idea is like we're going to take that vector and learn a way to compress it and then send that compressed thing through the network, and then we're going to uncompress it at the other end. And as a result, we save on network bandwidth, and we also get four times the number of experts for the same inference cost.

So you could think about it as like our library of books got four times bigger, and we get to read four times more books at the same inference cost because of this particular innovation.

Yeah, I believe MoEs have been the default in frontier AI for a long time. They're just a really good combination of inference cost and intelligence.

But they have drawbacks as well. They take a lot more memory. If you have a very small amount of memory, a dense model is going to be smarter. And they tend to work best either if you're running at batch size one, so you're running basically a single job, or you're running a huge data center with, like, infinite queries coming in. In the middle, they can be a little bit tricky.

The longer the context length, the more challenging problems we can solve with a language model. That allows us to do things like append all sorts of information to a query, which could be a codebase, it could be instructions. In the long term, I'm hoping that I have my own personal LLM that's able to read all of my emails and help me answer questions about that. The more information that we can attach to a particular query, the more useful the model can be.

Now, it can get more and more expensive to reason over large amounts of input data. And so that's one of the reasons why there's usually a limit on how big the context length can be. But with Nemotron 3, we tried to push it as far as we could go. We think a million tokens is a lot of tokens, and you can do a lot of things with that.

100%. Compaction is a thing, if you're using an agentic workflow, you deal with all the time. And compaction tends to work pretty well because language models are pretty good at identifying the most relevant things and summarizing. And you're basically trying to summarize your context when you compact it. So compaction is not a bad approach. I think having models that can just natively reason about larger amounts of data is just inherently more useful.

So of course we want to push the boundary on that as well.

If you're running at a low batch size, which is when you are trying to get the most interactivity, if you're in a data center, so you want the model to respond as quickly as possible, and it's okay for it to be more expensive—your cost per token might be higher—but you want the result as quickly as possible. Or if you're running locally, so you might be running at batch size one just because you're the only person using it. It turns out that the GPU has extra execution capabilities that are just lying there unused.

The bulk of the work when you're running in these scenarios is actually fetching the weights from memory, and then you push the token past those weights, and then you fetch more weights from memory. But it turns out if you push two tokens or even five tokens through those same weights, it would cost basically the same amount of time because the expensive thing is not doing the math to push the token through the weights. The expensive thing is just reading all of those weights from memory, all those parameters that have to come in.

And so the idea with multi-token prediction is to take advantage of this by having the model predict multiple tokens at once. Let's say that the model predicts five tokens. We know the first token is correct. The next four tokens may or may not be correct. So then what we do is, on the next pass, we take those four tokens and we stick them into the model, and then run it through. And at the end, we check. The model then predicts another set of tokens, right?

Then we check, were the extra tokens we predicted last time correct? If so, then we just accept them, and then we get like a 4x speedup. And if they were incorrect, then we only accept the ones that were correct and then proceed from there. So the benefit of this is it doesn't degrade accuracy at all because you're using the model to double-check, right? So all this speculation is going to get checked during the next token that you run through the model.

So it doesn't degrade your accuracy at all to turn on multi-token prediction, but it can give you a speedup, and it's probabilistic depending on the acceptance rate of your predictor. So if your predictor is more accurate, the acceptance rate goes higher, you get a higher speedup. So with our recent Nemotron models, we're pretty proud of our acceptance rates, but we're always trying to make them better, always trying to improve that acceptance rate.

This is a really good example of accelerated computing. With multi-token prediction, the speed that you get is a function of the accuracy of your model. The more accurate your model is, the faster the inference is, the cheaper the inference is, the more accurate it is. That's not usually how it works, but in this case, that's how it works. And what that implies is that if we're trying, as NVIDIA, as a company, to provide meaningful acceleration to the world's most important computational workloads, this has to be an important part of how we think about it.

If there's a 3x cost reduction or speed improvement for inference, which is the most important computational workload of 2026, if that's on the table and it depends on the accuracy of the multi-token prediction network, then that's something that NVIDIA needs to understand very deeply because it's going to affect our business directly.

So with Nemotron 3 Ultra, we did post-training using something called multi-domain on-policy distillation. And what that entails is that we have many different aspects of the model we want to improve. For example, science understanding is different from math theorem proving, which is different from coding, which is different from agent-harness interactions, right? With Nemotron 3, I think we had about 10 or 15 of these teachers. And so the idea is that you take these teacher models and you push them as far as you can go on some specific domain.

So you just don't worry about making it good at everything; just make it really, really smart at this one domain. Then you have a collection of these models, and you want to create one model that learns to be good at everything. And we do that using a specific reinforcement learning technique that a lot of labs these days use called MOPD. And the good thing about this is that because the teachers are supervising, they can give really dense rewards to the student model.

Basically, every token is getting supervised, and so the student can learn really quickly and then become almost as good as all of the teachers at all of the things. So one benefit of this is that it really helps the team work together better. If you don't have a technique like this and you have, let's say, 500 people working to try to make a model better, and one team's like, "Well, I'm trying to make it better at this thing," and then another team's like, "I'm trying to make it better at that thing," there can be a tug of war where it's like, "Well, who wins?"

And if you have to make a choice like, "Oh, I'm going to choose to prioritize this one over that one," then you make the other team feel like their work doesn't matter. It's just really hard. This is one of the challenges of building AI in 2026, is that you have to figure out how to get the people to work together, even though you're only building one thing at the end of the day. And so this particular technology has been really instrumental in helping more people work together to make Nemotron stronger.

Exactly.

Yes.

It's not an easy question to answer because it is quite complex. But I would say we rely on a number of things. One is that we do purchase data from companies that are building datasets that you can purchase. And to the extent that we have the rights to redistribute or to open up that data, we do as part of our Nemotron data effort. With Nemotron, we are trying to be maximally open with the data that we release because our goal is to support the ecosystem, right?

Our goal is not to be the only model out there, and we love it when we hear of other models around the industry that are using our datasets to make their AI stronger. Because that means we're succeeding at our job to keep the ecosystem thriving and growing. Now, we also are big believers in synthetic data generation. We use an enormous amount of compute running language models on our own systems to create synthetic data that then helps our models be better at solving problems in specific domains.

And we release a lot of that data as well. Now, it's of course not very straightforward to do this. Like, AI is always garbage in, garbage out. So you have to work really hard to make sure that any synthetic data that you create is actually adding value, that's actually helping the model generalize and solve problems more intelligently. But those are the primary ways that we go about building our datasets.

Coding is really special because it's a very intellectual exercise that created a lot of economic value, which then meant that we had an enormous amount of tokens that we could learn from, as well as tooling that allows us to verify whether our models are actually solving problems. So coding is always going to have a special place in our heart, and something that I think AI is going to continue to get much better at because we have this special relationship with it.

With regards to other domains, I think what I'm excited about has to do with significantly more diverse environments for AI to learn in during reinforcement learning. I believe that reinforcement learning is such a general form of teaching an AI how to solve problems. We're just getting started at figuring out how to apply that. And I think as our environments get more sophisticated, the AI then learns more understanding of the problems that it's trying to solve, as well as the implications of the actions that it can take. Then it becomes much better at actually solving those problems.

When I look at the environments that we're using today, they're still fairly simple, all things considered. And I think that's going to become significantly more complex and diverse over the next few years.

Well, NVIDIA is not structured according to an org chart. We have one, but it's not actually the best way of understanding how we work. My team, for example, is not part of the official NVIDIA Research team. My team is actually part of the organization that builds the GPU.

And my team is not the only team building Nemotron. There's probably 10 teams around the company that have significant involvement in building Nemotron in different parts of the company, in enterprise software, in our AI software division. The part of NVIDIA that actually designs the GPU is also significantly involved in building Nemotron. So there's so many different teams that have to work together. We always like to say that the mission is the boss rather than the organization. But what that implies is that people have to figure out how to work together, which is challenging in the sense that humans are naturally tribal creatures.

And it's not natural for us to be friendly with people we don't know very well, or trust coworkers that we haven't had success working with in the past. Actually, the name Nemotron reflects that. We had the NeMo team, which was building software for AI, and the Megatron team, which was primarily focused on systems research for building large language models. And we decided to work together and then start calling our projects Nemotron, reflecting the collaboration between these teams.

Since then, Nemotron has dramatically expanded. There's so many more teams that are part of the effort. And it's really important that we have structured it in this open way inside of NVIDIA. We are inviting volunteers from around the company to come help build NVIDIA's AI. We think it's very important to the future of the company. And as that vision continues to develop, more and more people want to join. That's fantastic. We're really excited about that.

And it means that we then have to figure out how to organize the work so that everybody has a chance to contribute and feel heard and feel like their ideas are fairly evaluated on the path towards impact. We have a formal process for doing that. We have an internal website where people share ideas, and then those ideas are assigned to one of 25 different leads that are over various parts of building Nemotron. They interact with those ideas. Some of those ideas get further developed.

Some of those ideas get deferred until the next time we go around building a new model. But we're trying to build Nemotron in an open and inclusive way so that we can really come together as a company to build it. I think organizations that figure out how to collaborate to build AI succeed. Organizations that struggle with control over who owns the AI tend to waste a lot of effort. And so NVIDIA's success and Nemotron's success, I think, is directly proportional to our ability to collaborate, something that I care deeply about.

It's a really complicated question, and it's obviously a difficult problem for everyone in the industry to figure out how to allocate their compute. Inside Nemotron, we have a budget for Nemotron, and inside Nemotron we allocate compute based on what we think the needs of the project are. We have a hierarchy, so we have a set of programs, and inside each program, we have a set of projects, and each of them puts forward their requests.

And then we have a two-week cycle where we review requests and review the budget, and then we make decisions in kind of a hierarchical way. And then compute gets decided that way. Now, having said that, this is something that I think we can still do better at. It's hard when we're making decisions about compute allocation because every researcher is convinced that their idea could change the world if it just got 1,000 times more GPUs attached to it, right?

And they might be right. It might actually be true. And yet we're running at the limit. We don't have 1,000x more GPUs for every idea that we have. We have to operate within the limits that we have. And so it is a challenging process. We try to incorporate as many people's perspectives into that as possible so that it's as much as possible a shared sense of understanding, maybe not agreement. So there may be times when one project feels like it really deserved more GPUs because the impact of that would have been so high, but it didn't get it.

We hope in that circumstance that they have an understanding of why some other project did get more GPUs and why that was considered more of a priority during this particular allocation round for the company, so that people can at least understand that there's a reason for the allocations that we have. Having said that, this process is always improving. There's always more work to be done to make this more transparent and more fair.

And then, of course, my number one is just to get more GPUs so that we can also fund more things, because I would like to do that too.

My belief is that research needs to be bootstrapped. Research is a chicken-and-egg problem. So it is always the case that every researcher believes, if I just had a lot more resources, my idea would change the world. Actually, it's important that researchers feel that way. Because if you didn't feel that way, you wouldn't have the conviction that's required to go do something crazy and new, right? So you have to believe. And so, of course, you start with that belief.

But then how do you translate that belief into something that other people can understand, right, that other people are willing to invest in? This is what I call the chicken-and-egg problem, right? Because once your research idea is obviously good and impactful, it's easy to get resources. But how do you get it to be obviously good and impactful without those resources, right? So the way you solve chicken-and-egg problems is by bootstrapping. This is an iterative problem-solving approach where you do something small, you get some sort of signal that this is a good idea, and you tell people about that.

And then you ask for just a little bit more. And if people saw, like, oh yeah, that experiment turned out pretty well, that's pretty intriguing, we should probably do a little bit more there, then you're on track, right? And over time, iterate a lot, iterate quickly, iterate many times, you can bootstrap to finding significant resources for your idea and also usually attracting more people to come along with it on the way because they have a chance to see that this idea is going to change the world, and then they want to be part of it.

Well, Jensen has lots of good ideas too. And so the company is very responsive to his ideas. And that's important as well. But Jensen very explicitly says all the time, this is a company of volunteers. Each of us is here because we choose to. We could be doing something else with our lives, but we choose to be here. And so we tend to make decisions, especially for early-stage research, in a very bottom-up way.

Because it's sort of an invitation, like, bring your best ideas, let's figure out what are all of our best ideas, and then we'll take a step from there. Now, do we sometimes have top-down ideas that are important for the company strategy? Of course. NVFP4 pre-training is one of those. So we decided, as leadership of the company, we're going to really invest in FP4 hardware. Now it's time to go invent some optimization algorithms that succeed in using it.

And so we told the team, we didn't say to the team, you have to work on NVFP4 pre-training. What we said is, there's an opportunity, we're making a big investment. And if we can figure this out, it will be significant for our company. And then we let the people who are interested in that work on that. And as a result, we succeeded. So it is a balance of bottom-up and top-down. But it always has this bootstrapping feeling. Even with something like NVFP4, where there's a significant strategic top-down component, the actual technical solution, which is very intricate and complex and has a lot of moving parts, came from the researchers themselves.

And that's my belief, is that research always comes from the researchers themselves. You can't tell research exactly how to go solve a problem because then it wouldn't be research; it would be engineering. But in a world of AI where the most important problems we have to solve all have this research component, there needs to be freedom for researchers to innovate if we're going to make progress.

Yeah, one thing that's very unusual about NVIDIA is the tenure of its leadership. Jensen Huang has been running the company for 33 years, but he's not alone. There are a lot of other very senior leaders in the company who have been there for three decades or longer, including my boss. And these people remember what it feels like to work at a very small NVIDIA. And they know what it feels like to work at a very large NVIDIA. They have a shared sense of ownership for the company.

NVIDIA is a place we often say no one fails alone. And the point of that, that's just a statement of fact, right? You work at a company, it's one company, you all succeed together, you all fail together. You work in accelerated computing. Accelerated computing is the composition of thousands of technologies. If any of them fail to deliver acceleration, the value is destroyed. It doesn't matter whether the chip is great if the compiler sucks. At the end of the day, the thing that you're selling is time and capability to researchers that are trying to build the future of AI.

And if they don't get that, it doesn't matter whether it was the transistor or the math unit or the compiler or the library or the networking or anything else along the way that failed to live up to its expectations. The whole thing in composition fails. The whole value is destroyed. And so we have a deep understanding of that culturally at NVIDIA, and it is something that motivates the way that we work together.

True.

Well, I think that intelligence is just so incredibly multifaceted. I always think about this question: if a company were to be looking for its next CEO, would it find the next CEO by looking for somebody who won the International Math Olympiad? Probably not, right? Even though it's incredible for people—I could never even compete in any way at the International Math Olympiad. Those people are amazing, right? They have just incredible brilliance. That's not the right kind of brilliance to run a company.

If we look, for example, at other aspects of our culture that are really important—for example, musicians—what kind of intelligence does it take to become a hit musician? Don't assume that it's all luck. It's not. These people are working hard, and they're very smart in ways that I might not understand with my PhD. I might not have that kind of intelligence. And so when I think about intelligence, I think it's just so multifaceted and so contextual. It really depends on the situation.

It's not just about raw intelligence. Raw intelligence is kind of like the horsepower of an engine, but an engine running without wheels doesn't go anywhere, right? So intelligence, the impact of intelligence has a lot to do with the context that the intelligence is put in, the harness, the platform. And so when I think about that, I think the singularity, although it's an attractive idea, is really a wrongheaded idea because it doesn't really take into account these other factors.

So I believe that artificial intelligence is going to continue to develop at a rapid pace. It's going to unlock significant capabilities for people in every aspect of our world economy, people doing every kind of work. I'm very excited about the opportunities that it's going to bring. I am also a little bit concerned with how we're going to manage the transition. So I do think that transitions are hard for humans in general. We're conservative generally. And there is going to be a lot of change.

This is a profound change in the way that we think and the way that we work, the way that we learn. Ultimately, I have faith in our ability as humans to figure it out. We've done it in the past. This is who we are. We build tools. We build external organs that help us solve problems. We have an external stomach, we call it a kitchen. It creates enormous value for us. We can eat things that we couldn't eat without a kitchen, right?

Now, we're creating an external brain. The implications of the external stomach were pretty profound for us as a species. They led to agriculture, which led to organized societies, the way our cities are built. So, we think about what are the implications of an external brain? Pretty profound. Nobody actually really knows. But what I do believe in is the power of humanity to solve problems and to learn and to incorporate new technologies in ways that benefit us. I also believe that the problems we face as a planet all require more intelligence, every single one of them, whether that's inequality or climate change or any of the other structural problems that I think are very worrisome that we face.

The solutions to those are going to require invention and intelligence. And what that means for me is that the only kinds of tools that we can really create moving forward are going to be AI, because the problems that we face are all about intelligence. And regardless of the technological approach to solving those problems, the solutions will always be called AI. And so that makes me hopeful for the future, but also somewhat respectful of the challenge that it is going to bring to us as we try to figure out how to live in a new way with this new external brain.

But I believe in our ability to learn and to change. And I think ultimately, this is going to make our lives better.

I'm always worried about the way that the public thinks about technology and interacts with it. It matters a lot. And it is definitely the case that societies that want technological advancement have more technological advancement than societies that don't want change. So I think it is actually important to think about it. One thing that's interesting about AI is that I believe it tends to be much more accepted when it is part of everyday life. And then, at that point, people stop thinking about it as AI.

It's just, oh, this is the tool that I use. Like, do you care whether it's AI that's helping you route your car when you ask the map application to help you drive somewhere? I mean, there is actually sophisticated AI that's going into that. But you're not really thinking about that, right? You're just using a tool. And so I feel like people's acceptance of AI comes with experience, right? The more experience we have working with it, the more we learn how to work with it productively, I think the more comfortable we become with it.

Safety is on everybody's minds right now. Watching the DeepSeek release and the way that the government interacted with that, I think, is a consequence of concerns about safety, about these models. They get stronger and stronger, and then they could be misused. And there's different approaches to thinking about safety and trying to define safety. I have maybe a slightly unorthodox opinion about this, which is that I think open technologies are generally safer because there's more sunlight.

When more people are thinking about the safety of a technology and evaluating it and then contributing to making it safer, I think that's inherently safer than having a small group of people being in charge of safety for everyone else. I also think with artificial intelligence, because it is really about ideas, it's really about exploring ideas in different ways, that diversity is more safe than monoculture. And what that means is that there's going to be different beliefs. Diversity isn't just about the easy stuff.

Diversity is about the hard stuff. Like, when people have deeply felt disagreements, they really, really totally disagree with each other. Making it possible for people to explore their ideas in a diverse way, I think, is more safe than trying to create a walled garden where certain ideas are considered safe and certain ideas are considered unsafe. And this is controversial in today's AI environment, which I think is interesting because we've had hundreds of years of tradition that speak directly to this.

In the United States, for example, we have laws about freedom of conscience and freedom of speech. And it's not because we didn't consider for thousands of years, would it have been safer if we didn't have those, right? We tried that. We tried actually having a monoculture about, like, these ideas are safe to talk about, these ideas are safe to believe. And we found that to be much less safe than a pluralism where we officially don't take a position about what ideas are safe.

We actually found that it is much safer as a society to support diversity than it is to try to keep everybody safe top-down. And so I believe that open technologies for AI are inherently the safest way of building AI.

Thanks for inviting me.