OpenAI's Yann Dubois: Why AI Progress Suddenly Feels Real

Hi, Matt. Thanks for having me.

Yeah, the last few months have been pretty wild. Internally, we also really feel it. And I think anyone who's coding basically is really feeling it right now. I think that's really because of three reasons. The first one is, even though in my mind the progress is actually pretty continuous, you need to reach this level of reliability to really make any of these AI tools very useful. And I think we just crossed that, probably December last year, at least at OpenAI. That's where I thought we really crossed that threshold, where now we can trust these models to do a lot of the work that we are doing.

So it feels like a step function, even though I think actually in terms of capability, it's pretty continuous. So that's the first thing. The second reason is once you start having models that are really good, you accelerate yourself, especially in terms of coding, given that we all code internally. You accelerate yourself both for having these models train the other models, but also build the tooling that we need as researchers to do our job. And all this acceleration, I think, means that we saw these last few months going faster and faster.

The third thing that I think we are feeling is all of last year, we really built these reasoning models and we really saw pushing a lot on reinforcement learning. And initially, when we had, like, o1, o1-preview, even o3, these models were still optimized for what we call verifiable rewards, things where we actually have access to ground truth and it's easy to test whether you're correct or not. That is, for example, the case in math questions or coding competitions. And what I think we are realizing now is that we were able to take many of the tools that we built for these verifiable reward cases.

And we were able to use them more generally for reinforcement learning on real use cases. And I think that's really why we're feeling that right now in just real-world coding rather than competition. So we moved from competitions to usefulness to users, and that's what we are feeling right now.

It's a little bit of everything, but in general, given that these are agentic models, the longer—if you just think about it as every two minutes, there's a certain probability that they're wrong. The longer that they run, the higher the probability that the final answer is going to be wrong. So it's just something inherent in agentic models. And what we've been pushing a lot on is making sure that we decrease this probability of being wrong every two minutes.

So purely from a model point of view, of course, there's a lot of reliability that is also happening on the applied side. And the team at OpenAI has been doing an amazing job on that. But I'm even talking only about reliability of our models and basically making sure that we decrease the probability of being wrong. Great.

GPT-5.5, to be honest, is one of these models where everyone in the company was extremely involved in building. And I think that we really feel it now. GPT-5.5, it seems like all the stars were aligned. That doesn't always happen. And that was just a great model for this. So we did feel it. It's kind of funny because in general, with every model that is looking really good early on, we have a model, we all get really excited about it, and then tons of doubts start coming up because it's like, oh, everyone is hyping this thing internally, but actually it's bad at all these other things.

And then there's another wave where people start underhyping it, and it kind of goes through waves. And it depends when we actually ship it, how people feel about it internally. But that's true of most models that we have. GPT-5.5 was not that different in this case, but it definitely maybe had a higher amplitude of the wave. So people were very excited, then not as excited, and we shipped it, and people were happy externally.

It really depends. GPT-5.5, but it can—it kind of depends which part of the pipeline is training parts of the model. So we really have different sub-teams, including pre-training, and you have the mid-training stage, and you have some post-training. And usually, the closer you get to products, post-training being the last one, the faster the iteration cycle is. And if you're more upstream, the slower the iteration cycle is.

So it could go from, let's say, from months to days, basically.

Yeah, we definitely have different teams that are working on specific use cases and are pushing on these use cases. My team specifically is actually the one that is taking all these vertical improvements and trying to put them together in the final model. You could see it as a team that is doing both the smoothing function. So you have all these improvements, but you need to make sure that the model doesn't feel too spiky, doesn't feel differently on different verticals. And also, you need to have some teams that are working—and that's basically what my team is doing—on all the horizontal improvements.

Improvements. So there are many things like instruction following, function calling, or thinking about how much should a model think for different problems. Those are very horizontal, and that impacts all these use cases. So we have both these more vertical teams and these more horizontal ones, and both are very important to improve on the model. And the good thing is that these things can kind of be improved orthogonally. So you might have multiple different teams that are working on certain verticals.

And maybe for one model, there's only half of these teams that made integrations, basically, in the last run and improved the model on these capabilities. And maybe for the next model, it'll be the other half. So that's kind of, at a high level, how it works. One thing which I will say, because you asked also about what are the things that we are really proud about for this model, I would say two things. Number one is the efficiency of the model.

We really, really improved the efficiency of the model, and most of the tasks can be basically performed, I would say, like, 2x faster now with this model. So that's great. And the other one that I already mentioned before is this alignment of the company and making sure that everyone is working towards the same goal. And that really takes the entire company working towards this North Star of building one good model in specific timelines. So very proud of how that happened.

So that's what I mean when I say it's the entire company: it really comes from everywhere. It has to come from inference optimizations. It has to come from the model being more efficient in its thinking time. So you have basically every token that you think for—the usual plot that you should be looking at is x-axis, the number of tokens that you think for, and y-axis, the performance. So these are these test-time scaling curves that we look at.

And research basically tries to move this curve to the left. So think less to be at the same level or more correct. And then inference also deals with this x-axis, but switches it from number of tokens to actual latency. And the final thing that people care about is latency on x-axis, performance on y-axis. And this is where everything comes together. So yeah, that's why I always say I'm really proud of the company for this one.

Yeah, I would say there's three things that we do. So in a broad sense, we are under the post-training org, and my team is the Post-Training Frontiers one. So there are three things that my team does. Number one is we decide what goes into the final run. So as we talked about before, there's many verticals, and someone needs to decide what can go in, what cannot, and also provide the science experiments for people to iterate on something that's going to be representative of the final run.

So this is the first thing that my team does. The second thing that my team does is bringing everything together and actually doing the big run. As you might imagine, we train on a good amount of GPUs, so there's a lot of infra work that is needed, but also there's a lot of ML work that is needed by putting everything together and making sure things work well together. And then the third thing that my team does is horizontal improvements to the models.

Basically, there are some things that these vertical teams will not usually look too much at. For example, the thinking time, as I said before. So, how much should a model think for on certain answers? Or instruction following, function calling, things like memory, and general improvements to the model that are really across the stack. So that's what the Pushing Frontiers team does, and I'm leading that team.

Oh, it's a long story, but I'll try to keep it really short. Basically, I did my undergrad in biomedical engineering in Switzerland. I'm from Switzerland. And then I went on exchange in Canada, and I learned about Word2Vec. So, I don't know if you heard about this algorithm, but it basically takes words, which are something discrete, and puts them in a vector space. So, basically, a way to think about it is as a plane where words that are more similar to one another will be closer to one another.

So it brings these discrete words into some continuous space that is semantically meaningful. And I was absolutely blown away by that algorithm. And that's when I decided that I wanted to work on natural language processing and just understanding language. At that time, I was very wrong, but I thought that English NLP was basically solved or close to being solved. That was in 2017. So that was right when transformers started. And it was actually right before transformers. So I was very wrong, but I decided that I wanted to work on under-researched languages, and basically, I wanted to improve NLP on languages where we don't have that much data.

So I went to work for Grab in Singapore, and I was basically building the natural language processing pipeline for them, working with Khmer, Bahasa, Thai, Vietnamese, and all these different languages. And then I'm skipping a little bit. I did more academic-type work in different countries, and I ended up at Stanford, did my PhD there. And after this, had a small stint in startups and then went to OpenAI.

Yeah, but I always think it's very important for me to think about the positive impact that I'm having in the world, or at least that I'm trying to have. So that's why this thought is there. Yes.

Exactly. That was a fun one. GPT-5 was not that reliable, so I was a little bit stressed that it wouldn't work, but it ended up working.

Actually, right before we did that, like the last rehearsal, it did not work. So I got slightly stressed about that, but yeah, seems like live ended up working well.

O1-preview were really breakthroughs in the research community about having models that can think. And the longer they thought for, the higher likelihood they would be of being correct. So that was really a breakthrough. But initially, and if you look at old blog posts, you would mostly see math evals and also maybe coding competitions, but things that are really easy to test whether you're correct or whether you're not. And that also gives you some suggestion about how we were training some of these models.

And how I see maybe all of last year, and especially the end of last year and the beginning of this year, is that we were able to take these algorithms that work with verified rewards, like things where we can say you're correct or you're not, to the messy real world and really optimize for the utility that we provide to users and making them more productive. So I think that's what really changed.

Yeah, I would say that's—I mean, there's also another big part of it. Number one, basically the first thing is that, of course, when you develop a new method, the method is fragile and it's not that reliable, and it's hard to productionize. So this part also improved a lot. But then it's also really, basically, we had a tool that we could start optimizing for different things. And initially, when we were developing this tool, we were making a lot of simplifying assumptions in the real world, basically.

And now we are removing these simplifying assumptions and, at least in post-training, we are able to optimize really for user utility and make sure that these models are useful. And the tasks that we're looking at are useful. And that's why also now current evals look much more realistic. I mean, if you think about GDPval, or even if you look at SWE-bench Pro or SWE-bench, these look way more realistic than, let's say, some Codeforces or coding competitions that we were looking at with O1.

Yes, basically it's just a question of how much test-time compute we pour into the model, or we pour into this entire system that we're shipping. We've seen again and again: the longer the model thinks for, the better answers we will get. The problem is that these curves that we are talking about are definitely not linear. There's some plateauing effect, and they kind of look logarithmic in some sense, depending on which evals. So you can pour two times more compute and actually only get small performance gains.

I personally don't use Pro that much because I really don't like waiting. I'm pretty impatient, so I don't like waiting for that long. And I know that the probability of being correct definitely improves, but it doesn't improve enough for me to use it. But there are some people who use Pro and who really love it, especially for academic research. And I know a lot of mathematicians who are using it. And that's because they kind of just have this in the background that is running for maybe one hour, two hours, and they don't really need to iterate really quickly with the model.

And Pro is really good for that.

Yes. So if you go back to the plot that I was talking about, where on the x-axis we have latency and on the y-axis we have performance, we're basically moving this curve more and more to the left when we say that we improve efficiency. So we're becoming more efficient, or we spend less time to achieve the same performance. But what Pro does is that it extends this curve. So it says, "I'm going to think for much longer, but I will have a higher likelihood of being correct."

But every iteration of the Pro model also moves to the left. So it also becomes more and more efficient. The important part is there will always be tasks where you just want to maximize the probability of correctness and you don't really care about latency. For example, if I start a job before going to sleep, the model has like eight hours. It should just think for as long as it can.

And this is what kind of probably gives you—

So for the efficiency, okay, so there's two things. Are you asking for the efficiency? What does it mean?

Let me give you maybe a metaphor from humans. If you have someone who's an expert in a certain domain and you compare them to some undergrad that is starting in that domain, the undergrad doing that task might take like one day, two days, and will have to think through a lot of the possibilities and investigate because it never did a certain problem. While someone who's an expert in that field will usually just know what direction to take, and it will not spend the time investigating ten different directions because it knows that there's one that is more likely to be correct.

So this is the type of efficiency that we're talking about. It's basically models where we optimize more on real-world problems. And as a result, it was trained to figure out with a higher likelihood which paths of reasoning are more likely to be correct. So this is the part on efficiency. There's also what you suggested: part of it is the model knowing when it's going down the wrong path. But this is also something that the model can be trained for with reinforcement learning.

It's knowing, okay, that seems like not a great path. Let me backtrack and let me go on and test something else. And if you train the model less, it might realize that it's on the wrong path much later.

For pre-training, I can't talk in a lot of detail about what is happening internally, besides that the team has been doing a lot of good work and our models are really getting better and better. One thing that I do want to highlight when we're talking, for example, with efficiency: if you have larger models, the amount of thinking time, so the amount of tokens they will think for, will usually decrease. And the way that you can think about it is that, metaphorically, the model already thinks through its weights when it generates a certain token.

So you can decrease the number of tokens that it needs to generate for thinking by increasing the size of the model that you are training. So oftentimes, if you just increase the model size, if you basically pre-train larger models, you will get better efficiency. And the good thing with larger models is that they can be parallelized better at inference time. So even though you might think, okay, you actually generated fewer tokens, but by a larger model, so you actually might decrease the overall efficiency of the system.

This is not true, because the larger the model is, the more chances you have to actually optimize for inference on GPUs. So you will be able to make the overall system more efficient. So that's one thing I wanted to say with larger models, that they are actually giving you a lot of efficiency. Otherwise, in terms of pre-training, I think it's very interesting. I actually also thought maybe two years ago that pre-training was kind of hitting a wall. And when we see, for example, if we talk just about Anthropic, I mean, Opus seems like clearly just a much bigger model. When you look at the cost, the cost of the model, usually that's how you know.

By the way, if it's a bigger model, you just look at the cost per token. And clearly they're getting very good performance just by increasing the size of the model. So I think the field was very, at least part of the field was surprised about that. There were a lot of conversations about hitting data walls, and it seems like we did not quite hit it. So the larger the model is, the more data it needs to ingest to be trained. And it seems like different companies kind of found different ways to overcome the fact that we don't have that much data on the internet.

I think synthetic data can probably work well in a data-limited regime. I think multimodal is an interesting one. I definitely cannot talk about what we do internally, but I used to work on multimodal representation learning back in the day, and I always thought that it would really help your reasoning abilities if you have a lot of multimodal data. And I still think this, but, for example, if you look at Anthropic models, they tend to not be that good on multimodal, and they are still really smart.

So it seems that it's not as necessary as at least I would have thought in the past. I still believe that once we go to embodied agents, embodied AI, you will learn a lot about the world, and you will kind of improve general intelligence and usefulness to users by learning how the world interacts with itself. But at least looking, for example, at Anthropic models, it seems that they don't need that much multimodal data to have strong models.

Yes. The intuition that I think many people had, and I definitely felt for a long time, is that it's hard to understand the world only through text. And it's hard to understand what physics is without really seeing what—for example, you can't understand gravity without really seeing things falling. And when you look at our models, I mean, they kind of understand gravity without having seen that. But it still seems not obvious. It still seems like they would get it more, and they are still missing some common-sense aspects.

So I do feel like we will improve the common sense of our models by having them interact in the real world. But we are still pretty far from that, I think. And by we, I mean just generally the academic community and the AI community seem pretty far from that.

World models in the sense that, yes, you can try to replicate or simulate things, basically work in an environment that is simulated? Yes. The problem is simulations are always going to be really hard and not going to be truthful. So I think there will always need to be a little bit of training that will need to happen in the real world to make sure that the model realizes these mismatches between the simulated world and the real world.

And I think we as a field have a tendency of optimizing something that is simulated or not quite realistic past the point where this is useful. So that's something that I think we should always be careful with: we spend a lot of time and effort on optimizing something simulated and not quite realistic. And it's great at the beginning, but at some point, once you start optimizing too much for something, it's not representative of the real world. And people continue doing that just because that's what they've been doing for a long time.

So I just think people need to realize when to stop that. I don't work with these types of synthetic environments as much, just because I don't work on embodied AI. So I don't know if we're at that yet.

Mid-training. It's just this idea of something that's between pre-training and, as you might realize from the name, the post-training part of the pipeline. And really, the idea is if you have high-quality data that is more representative of what you really want in your final model, you should overtrain on that data. So, taking a step back here, pre-training, what is it? Pre-training is basically trying to learn everything from the world by learning everything from the internet at a high level.

The problem is that most things on the internet are not really useful. If you think, for example, about Wikipedia or GitHub, which is coding data, it just seems like there's way more information in there than some random forums that maybe don't have that much information. For example, ads. There's also lots of ads on the internet. You probably don't want to train too much on that. But in pre-training, we train on everything. And in mid-training, we basically overweight this type of high-quality data that we think is more useful for training the final model.

And this is something—I can't talk about what's happening at OpenAI—but it's something that is definitely happening in all the academic community right now, and all the open-source models have this stage of mid-training.

It kind of depends how you define the term and where you put the boundaries. In my mind, post-training—I'll take it from a very broad sense—includes all the reinforcement learning and the training for our reasoning models. It's just the idea of having something that knows everything about the world and making something that is useful to people. So, pre-training, I think about it—the metaphor that I like giving is you go in the library and you have a lot of books about everything.

And in theory, you can find all the information that you want in the library. But it's much more useful to talk to an expert who has learned these books and who you can ask questions to, and they can answer and understand what you're actually looking for. So this is kind of the goal of post-training at a very high level: making something that is useful to users and is easier to interact with. So there are multiple stages. I'll talk only about things that are happening outside of OpenAI and kind of the usual stages.

There's usually some SFT that is happening.

Supervised fine-tuning.

Supervised fine-tuning. And that's actually what, early on, most of the models that were post-trained were only doing: supervised fine-tuning. The idea is that if you have humans that can give you the desired final answer, if you have humans that can give you the gold answer, you can basically clone the behavior of the human. So this is what we call behavior cloning. The problem with this is that you will never get better than what your ground truth gives you. And humans are actually pretty limited in many senses.

So you will never overcome the human labelers that you're working with. The reinforcement learning stage goes from behavior cloning to really optimizing rewards. So the idea is, I don't know what the ground truth is. I don't know what the perfect answer is, but here's how I would say whether the answer is correct or not. And here are the things that I want in the answer. And what you do is you start optimizing. You start having a model that tries to get more reward, basically optimize this reward function. That's what we call it. And it goes beyond what you currently have, what humans can do, or at least what the humans that you're working with can do.

So this, I would say, is the two big stages. Then in reinforcement learning, that depends on which models are being trained. At least in the open-source community, it seems that there are different ways of doing that. Reinforcement learning when you have verifiable rewards, so reinforcement learning where it's really easy to say whether something is correct or not, and you can really kind of have a binary reward for this.

And that goes back to how we talked about o1 and o1-preview in the past. And then you have reinforcement learning without verifiable rewards, where maybe I could do pairwise comparisons. I can say this answer is better than this other one, but I don't really know. I cannot quite say this is the perfect answer. So, of course, it's a continuum and there's everything in between, but I would say these are the three high-level things to think about when you think about post-training in general and how people are usually doing it in the open-source world, is that they take SFT, they clone the behavior that you can collect online or from humans.

And then once it's already at a pretty good level, they just do this reinforcement learning to go beyond what we currently have. Because if you just started from reinforcement learning, it would be very inefficient. Because the problem with reinforcement learning is that you have to stumble across the right answer, basically. Because how reinforcement learning works is you sample many times, essentially, from the model that you're training, and you say, this one is correct, this one is not. And you say, do more of the one that is correct.

So you have to stumble across the right solution. So you're much better off first getting as close as possible to the best you can do, and this is behavior cloning, and then doing reinforcement learning.

It's really hard to say because pre-training, when it's trained on all of the internet, arguably already has all capabilities in it. So it would be even harder to answer this question scientifically because arguably everything is already there. What I would say is that if you look at models that we were training or post-training, like two years ago in the open-source world—for example, I worked on one of them, Alpaca, where we used 50,000 examples for SFT—and now when you look at reinforcement learning from models like Kimi or from DeepSeek models, it seems that they are closer to one million data points.

So definitely people scaled up the reinforcement learning stage a lot. And from this, it seems that they've learned new capabilities, like this reasoning aspect, this fact that you can check your answer and try to improve it. So you can really think for longer to get a more correct answer. So all this to say that arguably everything is already in pre-training, but we were definitely able in the last year and a half, even in the open-source world, to have more capabilities.

I would say most people who did not work in reinforcement learning in the academic and research community up to two years ago probably thought reinforcement learning just doesn't work and is too finicky to work with. I used to be that type of person. And actually, when I saw ChatGPT come out, they had this blog. I was not at OpenAI at the time. I saw this blog that says that they use reinforcement learning. And my first thought was, I can do the same without reinforcement learning.

Because this is just an overcomplicated method. And this is actually the project that we started working on with Alpaca, was exactly, let's try to reproduce that only using SFT, just by doing this behavior cloning. And, for example, Yann LeCun famously gives this metaphor that reinforcement learning is just like the cherry on the top. So I think that was really the intuition that most people had. It seems that after crossing a certain scale of models that know basically everything about the world and what we call good priors about the world, it seems that reinforcement learning just started to work.

And this is not only with LLMs. Robotics seems to be entering the same stage where they're realizing that actually it used to be very finicky, but now that we use models that know already everything about the world, it actually learns pretty well. Now, to answer your question about what is still complicated with reinforcement learning, one is an infra aspect. So, just systems in general. Reinforcement learning, you have, at a very high level, basically to sample, as I said before, many answers and say what is correct and what is not.

And this sampling is just very expensive, and you have to do it at scale. The other issue that also in the open-source world people are seeing right now is that when we are training more agentic systems, you only know whether you're correct at the end of your very long rollout. So you get very little information per token of whether you were correct or not. And it's hard to basically do attribution. It's hard to say what part of your entire answer was the one that led you to being correct.

So that's more of an issue on the machine learning side. The ideal world in machine learning is when I can say exactly, like, this thing was good, do more of that. And the problem again with these agentic systems and reinforcement learning with these agentic systems is that you don't really know which part was good or not until you arrive at the end. That's another big issue for reinforcement learning.

I can't talk about what we're using, but, for example, in the open-source world, GRPO seems to be working very well. And people used to have different methods like PPO and DPO, and people seem to have really converged to this one. The big difference with other methods is that you, again, do this simple method that I told you about: sampling as many answers as possible, and you say which one is correct. So in some way, GRPO is a very simplistic method.

And in general, we saw over and over again in machine learning that the simplest method where you can scale up in terms of compute is usually the one that ends up working the best. And that is kind of what is happening here, at least in the open-source world.

Yeah, that's a great question. I think how it usually works is that it starts being craft. People just try out many things, and they start building a mental model of what works and what doesn't. And over time, we move from this craft land to more science. Science, or more scientific approaches, are really the ones that first end up working. It's very rare that you take a really scientific approach and you say, "This is the optimal thing to do," and you do it and it just works.

There's some sense of alchemy. People just have a good flair for something, and they make it work. And then other people, or that person, start trying to improve what we are doing by being very scientific. And I would say this happens over and over in machine learning. So first craft, then science, and both are really important, but it's different stages of the pipeline. In terms of engineering, this is definitely something that is always necessary. So I would say most researchers have moved to being relatively good at figuring—at least, I wouldn't say good engineers, but good at working in complex systems and figuring out what they need to try out.

And the systems and the infra that we have have become more and more complicated. So definitely the work required changed over time.

To be clear, I talk more about reinforcement learning because this is the part I know the best, and this is what I've worked on for a long time. We talked about mid-training before. All these things are also extremely important, and you can improve it in different parts of the pipeline. As I said before, the closer you are to the final stage of the model, usually the smaller the scale of the training becomes. So you can iterate fast on that because now you can iterate in terms of days rather than months.

So usually people start from this fast iteration loop, and then they go deeper and they make bigger changes across the entire stack. So this is not to say that only reinforcement learning matters. I'm really not saying that, but it's just that that's where people will start doing changes, and then that will permeate and we will go deeper into the stack. So this is how it works. And in the open-source world, it's very much like that too. I think you see way more post-trained models than you see new pre-trained bases.

And you see way more improvements in the algorithm. And that's why we talked about GRPO, DPO, PPO. There are so many XPOs, and that's because people can iterate really quickly on this final stage of the pipeline.

For sure, if you optimize more on specific types of problems, you'll be better in that setting. I would say, at least my intuition is that it's less about the exact problems that you're optimizing on, and it's more about the class of problems that you're optimizing on. So, for example, if you are really good at math competitions, your model will probably be pretty good at coding competitions. So it's not about the domain; it's more about the skills that are necessary and the way to think and the horizontal capabilities that you need for performing these tasks.

And that's what I think you're usually seeing when some model is really bad at something: it's actually bad at that in any domain, in any language. So you have to think about this domain and then this generalization of this domain, not necessarily per-domain capability.

It's definitely something that we actively push on. I think people are realizing, us and also other companies, that we are moving towards this world where we want to really make products that are useful and improve productivity of people and help people in their day-to-day life. So I think there's a very active move to deciding what are the domains that we should be prioritizing. Now that we know we have an algorithm that we can apply in different places, what we are constrained by is more collecting the right data, having people who really care about a certain problem work on that problem.

But there are not that many people who can do these things. So you really need to prioritize. So this is a very active, proactive kind of approach here. And in general, I would say the performance of the model really depends on the number of people who care about the final output of the model and who are looking at that model. So if they start looking more at specific verticals, these verticals will improve really quickly. But again, we don't have that many of these people that can do these things.

So I think there are different axes of generalization. One, there's algorithmic generalization. And that's really, can I use the algorithm that I developed or this black box that I developed for domain A, and can I use it for domain B? And again, even talking about the open-source world, it really seems that people are able to do that. They take GRPO, they apply it in many different places, and it just works. So that generalization seems to be relatively good, which is why we are seeing a lot of progress.

Otherwise, it would be hard to make progress. Then there's the generalization of the model that is trained on one particular dataset. And this is what I was alluding to before, is at least my mental model is that generalization happens in terms of capability. Like if the capability is the same, you will see generalization across domains. Again, like different languages, like coding, like you can optimize for C++ coding, for having a good C++ model with very little RL in C++, partly because this pre-trained model has seen all of C++.

And so it already kind of understands the basics of that language. So that type of generalization definitely happens. The generalization that I think is harder is when we don't have these horizontal capabilities. So I'll give you one concrete example. If my model is very intelligent in terms of being correct on competitions, I usually take that example because it's somewhat contrived, at math competitions, coding competitions. From a human perspective, people that are good at these things are usually just smart. And if they are smart, or someone might think that at least, that they're just smart.

And if they're smart, they can actually do other things too. But that is really not true. And that type of generalization is really not true, because many things where we need to have humans working on expert domains, the world is very messy, and these coding competitions and math competitions are extremely well specified. And you need to have the capability of understanding underspecified tasks, understanding how to deal with the messy world, and understanding what are even the resources that you need to answer the question.

If you look at the math competition, you usually have everything in the prompt. It's like you have five lines or maybe 15 lines, and it's all the information that you need to answer this question. In the real world, if I'm a consultant, if I work in finance, I need to go on the internet. I need to find and extract different information just to understand, before doing any of the reasoning, just to be able to do that reasoning. And this type of horizontal capability is the thing that doesn't usually generalize. If you have that horizontal capability, but in many cases we don't have that horizontal capability.

So yeah, that's why we hallucinate, actually, in every domain. When you have hallucination in LLMs, if a model is really bad at saying that it doesn't know, that usually happens in every single domain. You won't have one domain where the model is extremely calibrated about its knowledge and another domain where it's not.

John Schulman has a great presentation about that, I think from one or two years ago, where he was saying that if you do behavior cloning, so this SFT that we talked about before, you will basically reward and optimize for hallucination. If your model doesn't know about something, but now you say that the right answer is to say something. So I'll be very concrete. If the model doesn't know about a paper, and now in an answer that is given by a human, you say, "Here's where I got the information."

And then you cite that paper. What you're actually optimizing the model to do is cite something that doesn't exist, because it doesn't know that that paper exists. And so John Schulman had this great presentation saying SFT is going to force hallucination. While in reinforcement learning, given that, as I said, you sample from the model in the first place, it's extremely unlikely that it samples something that it doesn't know and it's correct. That's extremely unlikely. So you will never reward that behavior.

You will only sample things that it doesn't know and is being incorrect, and then you will kill that behavior. So hallucination, at least the intuition that people have, is that it can come, for example, from SFT, and it can come from this post-training pipeline. But if you have a good reinforcement learning pipeline, that shouldn't happen too often.

In domains, usually not. What will happen, though, is you will make decisions based on which domain we optimize for. And if you optimize for one domain, you will be able to optimize less for another one. So it's not necessarily that optimizing for one thing will make the other one worse. It's just that, as a result, you can optimize less for the other one because you're compute-constrained, you're data-constrained. You have your human bottleneck also in terms of that work.

What does happen is you can have negative kind of generalization, like bad generalization or negative transfer, more for these horizontal aspects of the model. So I'll give you a very concrete example: explicit instruction following versus implicit instruction following. If I have a model—and this is, we often hear, for example, from OpenAI models—that they tend to be really good if you tell them exactly what you want. But as a result, sometimes we hear also that they're less good if you are not as specific about what you wanted.

For example, if I make a typo and I say, change this file, and I make a typo in this file, an extremely good model at explicit instruction following will change the wrong file, the one that has a typo. But humans would probably realize that you made a typo. And as a result, there are cases where this explicit instruction following goes against this implicit instruction following. So you will have cases where basically these horizontal capabilities go against each other.

Yes, we can. I don't think there's anything really deeply special about these domains where we cannot optimize and where we couldn't get the same with other domains. The but is for at least two reasons. The first one is most of the people working on these models are pretty good at coding, and they really care about coding because that's what they use as their day-to-day kind of drivers. And there's nothing better than the user being also the one who trains the model, because then they understand the issues.

It's very hard to really, for me, for example, it's very hard to really understand what should we change on the legal aspects of the model if I don't understand anything about the legal domain. So that's one thing. The other thing that you will often hear about, and I mentioned also briefly before, is this kind of verifiable rewards. There are domains where it's easier to say whether something is correct or not. For example, in the case of cyber, you mentioned that before, that cyber has been improving a lot, the cyber capabilities of our models.

And this is because in cyber, it's extremely easy to say, are you correct? Did you find—did the cyber issue that you find is a real issue or not? It's very easy to test it. And so there are domains where reinforcement learning is just easier to apply, but there's nothing, I would say, in the capacity of the model that is constraining the model to be as good at legal and medical and other domains. So, the short answer is we know less about these domains, and definitely there are some domains that are easier to optimize for in reinforcement learning.

Evaluation has been harder and harder as models become better. And that's because the tasks that we ask of the model become more and more general and more and more open-ended. So now I maybe just say, "Build me a website that does X." Well, before, in the past, I would just be like, "Hey, is there a specific bug in this implementation that you have?" And it's much easier to say whether there's a bug because I can inspect, I can know, I can have a human that says, "Here are all the bugs that you have," and then you can apply that automatically.

While the website one is very hard to know what is the optimal answer, because there are many good answers. There are many good ways of building a certain website. This open-ended nature of models really makes evals harder. There's also another issue: models, in specific axes, are becoming better than the majority of humans. And so we have fewer and fewer humans that can actually evaluate these models in particular axes. So that's definitely a constraint. Another one, to be honest, is kind of cultural.

Most people want to improve the model, and they think that the best way to do that is kind of training the model, when in reality, finding issues and making sure that we can quantify improvements is just as important, if not more important. But there's always this cultural gap. That was especially true, I would say, in the academic world up to two years ago, when evals were always fixed, benchmarks were always fixed, and even datasets were kind of always fixed, maybe, let's say, four years ago.

And there was a mentality shift of, like, okay, data is actually critical. And now there's a lot of people working on data. And I think evals were still not quite there. People don't really fully—everyone knows that it's important, but people don't really understand how impactful it could be to work on evals. So actually, my first project at OpenAI, I just came in and I was like, "I want to work on data and evals because I know that this is the thing that no one is working on."

And as a result, I know that it's super impactful to work on that. And yeah, the tide is shifting, but not fast enough.

It's really fundamentally the same method. Most of the things that we do in evals, especially now that we have reinforcement learning, could just be applied nearly exactly as is during training. So that's another reason, actually, why evals are so complicated, is that every time you build an eval, you actually build a way to build training datasets. So now you're going to optimize that training dataset. Well, even if it's not that eval, it's going to be the same type of data, and now you're going to do super well because we have this generalization of capabilities that I was telling you about.

You will learn that on that other dataset, and now you'll become really good at that eval, and that eval will become obsolete really quickly. So that's also an issue with evals. But yeah, to come back to your question, model as a judge, it's really important. And I think it's probably one of the most important things because as we get better models, we have this self-reinforcing loop and we have this capability flywheel where better models become better teachers for other models. And this is really important for training, but then you can also do the same thing for evaluation.

So a lot of my team works on that, and I think what's really critical is to work on this model-as-a-judge kind of framework.

In terms of progress, as I was saying before, I think it's always continuous. Now, the feeling of discontinuity will happen. It did happen three months ago with coding, or four months ago with coding. And I think that will happen now in every other domain. Most people are not feeling the capability and usefulness of our models the same way coding and software engineering are feeling right now. So this will definitely permeate, I think, through many other verticals.

Now, in terms of capability bumps, in terms of, let's say, the verticals that we're already looking at, I think it'll be more continuous, and there will never be big discontinuities. Most of them are always local discontinuities, but you zoom out and it always just feels pretty smooth. It's not always like this, but that has been the case most of the time, and I definitely cannot predict when is the next big discontinuity.

I'm extremely excited about continual learning. I think we haven't quite cracked it. I mean, we have Codex Memories, and that is helpful, but it's definitely not the end state. I have a friend who always tells me about, again, another type of plot that we should be looking at, which is: x-axis, time; y-axis, utility that you provide to users, or usefulness, basically, of the models. And right now, actually, most models at day zero, if you just drop them in a company, arguably they're more useful than most new employees.

So they start higher at T-zero, but then across time they're mostly constant because they don't really learn company knowledge. They don't really learn to be more efficient over time at doing the things that they are doing, while humans learn really quickly. And what is important is this integral, or the area under the curve of these curves. And as a result, I think humans are still more useful in many cases. And that's why what we will need is continual learning, to make this curve monotonically increasing over time and basically make models more and more useful the longer they work in a certain environment.

So I'm extremely excited about it. I'm actually surprised that we're not quite there yet. Three years ago, when ChatGPT came out, I remember I was doing a startup with friends, and we were thinking about working on continual learning and personalization and memories in general. And we were like, "Ah, OpenAI is going to do that in the next six months. They have all the data. They're going to figure it out, and they have all the users, and the models are going to learn super quickly from users."

And three years later, I don't think we're there yet.

It's a good question. I actually don't quite know, to be completely honest with you. I don't quite know why it's taking us that long to figure it out. It's this type of domain that I think, if we really put enough resources behind it, we would figure it out. Of course, especially when we talk about this memory inside of a company, there are big questions about permissions. And there are a lot of questions about privacy and what you can share and what you cannot across models, across users.

But for a single user, even for a single user, we're not quite there. And I don't quite know why. At least at the high level that I can talk about, I don't know why.

Yeah, I think harnesses can really improve the capability of a model right now. I think, given that we are seeing this really fast progress in terms of capability, I personally wouldn't push that much on the harness unless the harness is for a very concrete goal that you're trying to achieve right now. So certain companies, if they are focused on a specific vertical, want to go from this 80% reliability to maybe 85%. Harnesses will give them that.

And I think that's very important, but they need to do it while knowing that they will have to retune that harness in the future. And I think that's totally fine. If you try to have a general harness that will sustain over time, I don't think that will work. Harnesses for specific domains are a short-term thing that you need to do. I think there will always be so much you can do in harnesses. And if anything, I think everyone should do more of that if they have a specific problem in mind, because we're leaving so much on the table without a good harness.

Arguably, I think if we froze the models that we have right now and you really worked on the harness, and maybe we also spent more time training with a great harness, I think people would really feel the AGI in every single domain, or could already feel that in every single domain. But given that we're not freezing it and we're going to continue training better and better models, I think we don't really understand what the final harness will be, and it will always change.

There's so much space for external companies or startups pushing on specific verticals. I think there's so much space for that. The reason why is because a lot of people kind of think about intelligence, in quotations, and raw capability as being the real bottleneck, but I don't think that's true. I think most of the time, the bottleneck is the last mile. It's making sure that the model has access to the right permissions or also has access to the right connectors and things like this.

And we are going to be very focused on this general aspect. And I think there are other companies that should be focused more on the verticals and providing maximum value of what we currently have. So I think there will always be a lot of space left for this last mile in different verticals. And I would highly encourage people to continue working on that. And maybe one day, when we stop making horizontal progress, which I don't think is anytime soon, maybe we will start focusing on that.

But yeah, that's not what we're doing now.

Great. Thanks, Matt.