Sonnet 4.5 & the AI Plateau Myth — Sholto Douglas (Anthropic)

How you doing? Great to be here.

Yeah, I think it's a proxy for a couple of things. One is that there's now this two-paradigm regime where previously you did pre-training scaling and reinforcement learning scaling, and now we're in a mix of the two, basically. And so I think that gives you more opportunities to update models, because it means that you can make advancements along multiple frontiers. And then that means that you sort of end up shipping more frequently. We're now 2.5 years after ChatGPT. And so the post-ChatGPT investment cycle is finally hitting where compute availability is increasing and all of this.

And so it means that you should expect, actually, the pace of progress to increase, because there's lead times in commissioning chips, basically. So even if you wanted chips last year, it would have been impossible to get them because TSMC was booked out and so forth. So finally, this year is where the compute supercycle is beginning properly, in effect. Yeah.

So we release models along three categories, three tiers: Opus, which is the smartest model; Sonnet, which is the mid-tier model; and Haiku, which is the fastest, cheapest model. One of the interesting things about this most recent release is actually Sonnet is smarter than Opus. And this has happened before. In fact, this happened last year. It's a reflection of fast progress because it is cheaper to train mid-tier models than large models. And so what happens is that you end up doing a lot of progress on smaller models.

Eventually, you need to choose when to scale up and sort of get the benefits of scale in a model. Often, you make progress fast enough that your mid-tier model is great anyway, and it's actually better than the large scale-up model that you did previously. And I think this is also a little bit of a reflection of the reinforcement learning paradigm, where you can take a model and you can train it and extend it with reinforcement learning, basically. So that allows you to take a mid-tier model and make it as good as a larger-tier model of six months ago or three months ago.

Yeah. So I think, how far back do you want me to start? From the beginning.

Yeah. So a couple of things. One is that growing up in Australia, there's a very traditional set of paths you can take. You can become a lawyer, you can become a doctor, or you can go into finance. Australia is a wonderful country in so many ways. In particular, the quality of life is so high that it means that people just choose these default paths, have a fantastic life. And I was very lucky in some ways. My mum was actually frustrated in her ambitions.

And so this meant that I had the perfect mentor throughout my entire life. She studied medicine, went on to do emergency medicine in South Africa, but wasn't ever quite able to break into public health in the way that she wanted to. She wanted to do systemic change in public health. And at the time, that was just very difficult for a woman. So instead, I had her full attention. Growing up, when I did an exchange in China, I got this dossier this thick of China's political economy and different actors in the current startup ecosystem and this kind of stuff.

So I had this wonderful, constantly driving education in a really supportive and wonderful way. I also was lucky enough to get into fencing. And through fencing, I had the experience of becoming one of the best in the world at something via repeated effort. I became top 50 in the world, at my best, 43rd. And it was partially a consequence of—well, I think in large part due to having a coach and perfect mentorship that was one of the best in the world.

He moved to Australia because his wife was Romanian. He had just coached Italy to the gold medal in the Olympics. She was facing discrimination in Italy. And so I had, on the one hand, perfect academic mentorship, and on the other hand, perfect athletic mentorship and a proving ground to grow up watching these people on YouTube and then become one of the best in the world at something.

In some ways, yes. Or in other ways, like an introduction to, you can watch these people on YouTube and analyze what they are doing to become who they are, and replicate that. And you could be part of that world. All it just takes is intense amounts of effort.

Right, exactly.

Yeah. And I mean, I think we should see the same thing with AI, right? Like, in the same respect, everyone will now get a perfect tutor. I then actually had that experience again with AI. Fencing wasn't something I wanted to do ultra-long-term. I wanted to take a shot at the Olympics and then try and progress into working in technology, basically. I was very lucky to read a Gwern essay on scaling, where he basically details the scaling hypothesis. After reading that, I was like, oh my God, this is absolutely, clearly, AGI progress over the next decade is going to be one of the most meaningful things to work on in the world.

It's the largest lever we have to meaningfully advance the world. And so I started doing my own research on nights and weekends, and I was, like, partway through undergrad.

So this is last year of undergrad and the year after.

I did computer science, robotics. And I sort of vaguely grew up looking up to Elon Musk and this kind of stuff. I wanted to build rockets and Tesla, but I didn't have a concrete idea of what actual problem I wanted to solve. Reading that essay was the critical hinge of, okay, AGI is possible this decade. It seems like the most meaningful thing in the world to work on, and I need to figure out how I can demonstrate that I should be working on this.

I was at the time working on robotic manipulation stuff. And so I started working on scaling up robotic manipulation, trying to train general foundation models for robotics from the bedroom, which is now a big thing. There's a lot of general foundation model for robotics companies. It was a little bit early then, but I rigged up my own simulator, collected a lot of teleoperation data, trained models, got a loan of TPUs from Google. Eventually, some people at Google noticed the work I was doing and said, hey, this is great work.

Would you like to come work with us? It was actually very fortuitous because, for example, I didn't get into the PhD programs that I wanted to. I applied to a couple of PhD programs here after undergrad and didn't get in. But I was very lucky that the work that I was doing really resonated with Google. And so they reached out.

Right.

I think they're very highly correlated, but I think the signals that are usually used to gate academia are, like, there are dramatically more people that satisfy the criteria of being really effective than there are that have the correct signals that would then enable them to progress to the next stage in an academic career. For example, if you're here in the US, you end up doing, as an undergrad, research that can get you a NeurIPS or ICLR paper, whereas in Australia, it just isn't the case.

Right. I remember Pieter Abbeel actually once visited our lab in Australia and asked people to put their hands up if they were going to NeurIPS, and no one put their hands up, not even the PhD students. So it means you don't have, again, that mentorship aspect that is so important. And so you don't get a chance to develop problem taste on the things that mattered, and therefore you don't have the correct signals that indicate you would have high potential for academia. I actually think that right now a lot of the signals we look for aren't traditional PhDs or anything like this.

I mean, this is obviously very useful, but the fastest route, or the most immediate one, is whenever we see a really good blog post where people have done an incredible amount of work in an independent fashion, it's one of the highest-signal things there is. One of the examples I love to use here is this guy called Simon Boehm, who's one of the leads on the performance team at Anthropic. And he's published, to date, the best guide on how to optimize a CUDA matmul on a GPU.

It is simply the world's best CUDA matmul guide. No one has done this for attention, right? If someone did this for attention, then, I mean, we would reach out with a job interview offer the next day, right?

And in fact, someone did it for TPU and for some kind of attention the other day. And we were like, this guy, immediately, let's send out a request to interview. So I think there is actually an absence of agency, there's an absence of taste, and there are still many ways to demonstrate this, usually by producing a world-class artifact in some independent fashion.

Yeah, I think it's growing quite dramatically. I also think we have done a pretty good job of growing people. And I mean, I think Anthropic has taken many, many junior people and grown them into really fantastic researchers and engineers in quite a deliberate way. So I think it's definitely growing.

So Google noticed me. I started at Google, I think, like a month before ChatGPT or something like this. So it was actually a fantastic time to start at Google, because the entire company was suddenly forced to react instantaneously and compete with OpenAI. So it meant that there was this gap of, I guess, the typical command structures and everything were not well suited for that particular battle. It wasn't a preexisting org. Gemini was sort of forged out of the merging of Brain and DeepMind.

It meant that there was just a huge gap in terms of agency, really, of figuring out what we needed to do, doing it as fast as possible, organizing people together to work on important things. And so I ended up, one, getting the chance to develop a lot of taste by working closely with people in those early months of Gemini, but two, also quickly got the opportunity to step up and lead various parts of this. So one example of this is we just didn't have an inference stack that was at all sensible for the modern world of LLMs.

And so we had to notice that, design one from scratch. A lot of the things you now see in the sort of SGLangs and stuff of the world are things that we had to derive from first principles at that point in time. And we wrote the inference stack. This ended up saving several hundred million dollars, I think, even over the first six months, and meant that I was then trusted to solve both hard technical and sociopolitical problems. Because one of the interesting things about the inference stack as a problem was that it was both a really large technical challenge and a large sociopolitical one, because the ownership of the preexisting stack was distributed across five or six different teams.

And it meant that actually enacting change was quite hard. And so it was a challenge along multiple dimensions. That then led to me having the trust to solve problems of this form across other parts of the ML stack. And so, for example, later on, when the thinking stack was started, the reasoning stack, I was in charge of research infrastructure for that to get us an RL codebase that could actually allow us to do large-scale RL and reasoning and this kind of stuff.

And then the move to Anthropic. So the move to Anthropic was in February of this year, and I think it was motivated by a couple of reasons. The number one is I'm just really excited by how deeply every single person in the company cares about how the future goes. And I think that's one thing that really struck me, is everyone at Anthropic has an articulated theory of why what they're working on contributes towards a better future, whether that is AI that is better in ways that can help people improve their lives, or whether it's because it's AI that's safer and more controllable and aligned with our civilization's interests, or even just more deeply understanding what's actually going on inside this AI and trying to better forecast the progress curves and where we think we're actually headed, or policy.

And probably such a strong advocate for policy in many ways.

Yeah, I think there are. For example, DeepMind, if you wanted to solve science, is the best place in the world. I think that DeepMind will directly contribute to more scientific discoveries from AI than anything else. Absolutely. And I think it's just so well set up to do this across every aspect. You've got both the direct scientific efforts like AlphaFold and the material science work and this kind of thing, and also generally large efforts to make AI scientists and all that.

Whereas I think Anthropic has been laser-focused on two things. One is long-term AI alignment, and two is near-term economic impact. So Anthropic has been laser-focused on coding and computer use and things that we think will make a direct impact on the economy within the next six months. One thing that Anthropic noticeably hasn't focused on, compared to DeepMind and OpenAI, is mathematical reasoning. DeepMind and OpenAI have been pursuing mathematical reasoning because of the implications for science and for scientific progress.

And because I think so many people there just love maths so deeply and would love to see the field progress.

We've had to reluctantly sacrifice our focus on that because we want to focus on—well, it's partly for many reasons, but we want to focus on near-term economic impact with the models. And then much of our research along other dimensions is focused on that.

I had a really interesting discussion about this with a biology friend where we were comparing taste across biological research and ML. I think one of the most important things is mechanistically understanding exactly what you're trying to do and having an important simplicity regularizer. When you think about taste in ML, it's often the crucial ingredient that allows you to decide what goes into your large training run when you have imperfect information. Because we can study very deeply what the impact of an architectural change is, right?

But past a point, past a certain level of scale, you have to guess whether or not the impact of that change will compound with other ones, whether it will conflict. Because you can't test your full-scale run, right, N times. You only have one shot at that.

And so a lot of taste comes from being able to make good inferences about, do we think that this ultimately will sort of deliver increasing returns to scale? It also comes down to, do I think this direction of research is worth pursuing? Because often our baselines in ML are so well tuned that it's very hard to beat them, even with what is theoretically a better method, because there are so many small tricks that are required to make a machine learning method work, and they can fail for any number of reasons.

It's not like building a bridge, where you actually have a pretty good idea why a particular shear was introduced. It can be all these quirks. And so knowing whether it's right to push along that direction or to give it up and try something else is another question of taste. And I think it always comes back to the simplicity regularisation. People love to be clever. We all do. And that's sort of the bitter lesson, I think, is maybe the best expression of this, where generations of people have developed clever methods of encoding priors about how they think an artificial intelligence should reason and encoding it into the model.

And all of this gets wiped out by scale and through planning, basically like search and learning. And scale applies to those two things.

Yes, exactly. The methods that can take advantage of compute, in particular search and learning, will wash away all tweaks. I think I can offer a couple of examples of this in some ways to make it more concrete. One of the reasons that convolutional neural networks were more effective is they encode a prior. And convolutional neural networks, in many ways, you can think of as a little square being drawn across an image, so to speak, that nearby pixels are related to each other.

And this is a very sensible prior, right? Because if you throw a picture at an AI model and you don't tell it anything about the world, it has to learn that nearby pixels form curves that then form other things. And so there's this hierarchy of abstractions. But ultimately, that is not true of all images. And so convolutional neural networks will be better than a more general vision transformer for the vast majority of images up to a certain amount of scale.

But past a point, actually, you need to be able to flexibly integrate information across the entire image. And a similar example in language might be, well, we know a lot about grammar, and so you might actually want to decompose a sentence into the constituents—the verbs and the nouns and how they relate to each other and so forth—and provide that explicit structure to your AI algorithm. But then what happens when you want the model to write poetry or to write code? All of a sudden, these assumptions have to be thrown away.

And so you can't generalize across poetry and code and writing.

So you can do actual numbers up to a point. The way to sort of illustrate or think about this is: you are testing a system at multiple levels of scale. And actually, the analogy to biology was, if you think about it, you might test a new therapeutic in a cell and in mice and in model organisms. But that's no guarantee that it will work in a human, right? So you test across multiple different scales and multiple different model organisms, and it seems to work in basic single-cell bacteria, seems to work in mice, maybe works in monkeys.

That gives you a lot of indication it's going to work in humans, but it's not a guarantee. So at that point, you need to understand the underlying mechanisms of how does this thing work, like what receptors is it binding to, and so forth. And in ML, it's exactly the same, right? You have your different model scales and you figure out, well, okay, it's delivering benefits at these model scales, and I think it should work because mechanistically, I understand what this is doing to the learning dynamics of the model.

And then you can have confidence that it's going to work. But if it's like, oh, it's a hack and we don't really understand how it works and it's really complicated and introduces all this stuff in the code, then—

Yeah, or in general. I mean, I think a good example here is I once asked this question of Noam Chomsky, and he was like, yeah, maybe like 10% of my ideas work. And that's Noam, right? An absolute genius, one of the best in the field. So if only 10% of his ideas work, then I think that establishes a bound on the percentage of ideas that work. Most don't.

It's one of the things that at both Anthropic and at DeepMind we really tried to build, which is a culture of safe experimentation where people were trusted to explore ideas for a long time out in the wild, because you often need months of independent research to really prove out a novel research direction that is a substantially different one. It's hard. Particularly, I think it's hard actually less from the compute cost of the experiments and more from a cost of time and focus, because there are so many remaining wins, I guess, even in the current architectures and paradigms and everything. Maybe there's so much low-hanging fruit, right?

Like, a really high-ROI use of your time would probably be to just go and look at the data and think hard about what the model is learning or doing and make some tweaks to that. You could even just—the simplest things in the world will still deliver massive gains. And so asking people, or giving people the time and space to breathe, and saying, well, we know that there are short-term things you could be doing, but actually we want to try and develop a more general or fundamental technique that allows you to scalably do this in the future, is important.

There's this tension between doing things at scale and doing things that don't scale, right?

So, non-transformers, non-RL? Yeah, I think this is another way in which Anthropic and DeepMind differ a little bit. Anthropic is a very focused bet. We think that AGI is within reach in the next couple of years. We think that it's the current paradigms, or something not crazily dissimilar to them. Maybe there's something new, but it's not like we think it's some crazy out-there research program, right? Really, for the last five or six years, Anthropic's ethos has been: scaling compute with broadly the current set of techniques, AGI is tractable within those bounds.

DeepMind has a much broader scientific culture because it has the resources to do so, right? Anthropic has to be a focused bet. DeepMind has the time and space to be like, well, we're happy to bet on something that is really far outside the current paradigm. And I think, depending on which kind of question you want to ask, whether you think the really focused bet or the wide exploration of different and novel architectures is better, that's one of those sort of research ethos differences.

Not to say that Gemini itself is a very focused bet, but if you look at Gemini as like 1,000 people, then there's still like 10,000-plus people doing all kinds of really long-term foundational research at DeepMind.

Yeah, we're really focused on coding for two reasons. The first one is that we think it's the thing that will allow us to assist ourselves in AI research fastest. So there's this notion of automating AI research, right? We think that one of the most important signals of the speed of takeoff, the speed of progress, is how much AI is able to assist AI research.

And so prefetching this is really important, we think. Secondly, we think it's the nearest-term tractable problem domain in terms of economic impact. For Anthropic to be a viable research program that can research the things that we think are important requires economic return. And coding is a huge market full of people who are really, really, really keen early adopters, who love trying and switching things, who are really excited to play with new tools. There's massive, massive demand. There's dramatically more demand for software in the world than there is good software.

We've seen that in every previous iteration of compilers and general web abstractions and so forth. There's just a booming demand for software. And so, basically, the models are better at coding earlier than anything else because coding is a uniquely tractable problem in some respects for the techniques that we have, in terms of the data exists in many ways. You can containerize and run things in parallel. You can run unit tests, and so you can verify something.

Self-driving is uniquely hard, right? You need the car to work first time, kind of. Whereas coding, the models can fail 100 times. As long as it succeeds once, then that's fine. So there's this tractability, there's this replayability that doesn't exist in other fields that touch the real world in some ways. Like, you don't want a lawyer arguing your case that is an AI, right? Because what if it gets the case wrong?

Sorry. Too bad. So, as techniques develop, coding is uniquely tractable. And you can see that, right? Already, I myself am dramatically higher productivity when I'm using the AI tools to write code. And I have a friend who manages nine Claude Code instances, which is just like a crazy number. I don't know how he does that. I can only handle two. So it's maybe like a skill issue on my behalf.

So SWE-bench is the current benchmark for how we measure coding progress in the outside world, which all the companies use to evaluate against each other. It's an imperfect benchmark in many ways, right? It is like 50% one particular web framework and this kind of thing. But what it does do is it takes real-world scenarios of work that people have done, which is submitting a pull request, so a change to a codebase.

Stuff that's on GitHub, right? And it checks whether or not the model is able to do that same pull request and pass the same tests. And this ends up being a pretty decent proxy for a couple hours of work from a software engineer. These changes aren't incredibly complicated, but they're reasonable complexity, a couple hours of work. We moved recently from roughly 72% to roughly 78% in SWE-bench, which is a pretty substantial step up. I think it's worth pointing out that as recently as a year ago, I think we were under 20% or something like that as a field.

So there's been dramatic progress in the ability of models to do this unit of work that a software engineer does. I think that SWE-bench is imperfect in a lot of ways, and it's probably pretty close to what we call saturated. One interesting thing to look at with AI benchmarks is you see that they lose their utility past a point because they no longer disambiguate the differences between different models of high capability. But the models, one, are SOTA. They're the best in the world on SWE-bench.

It's a decent proxy. We're also, I think, more excited by the fact that a lot of our customers and partners are really excited by the model. So one example of this is the Cognition folks and Devin found the model so useful, they had to rebuild their architecture around it.

Great blog post on this, right? I think that's the real measure of whether or not a model is good, is whether or not it enables people to do things that they couldn't do before. And really, coding as a whole has been transformed in this way over the last year. 3.5 Sonnet, which is the first really strong agentic coding model, the first model that you could ask to do something in front of you, and it sort of was able to interact with your codebase on your computer and do it.

In many ways, this model is what caused PMF for Cursor. 3.5 Sonnet, because they were in the right place and they were able to capitalize on that model as offering a coding experience that didn't previously exist.

And then actually Cognition and Windsurf went for an even more ambitious target. So basically there's like a spectrum of agency here where either you can ask it to do 30 seconds of work or a couple of minutes of work. 3.5 Sonnet. Then roll into this year.

Right, exactly. Bet on the exponential. So what I think a lot of coding startups are now asking themselves is: what can they now do with models that are capable of independently pursuing goals for substantially longer than previous goals? Before, you had to supervise the models every 30 seconds. Over time, over the next couple of months, you're probably going to end up in a situation where you only need to supervise the models every 10 minutes, 20 minutes or so. That's a pretty dramatic change depending on the complexity of the task.

Even we have a couple of examples, I think it was mentioned in the blog post, where we asked it to build something that looks roughly like a chat app, something like Slack or Teams, and the model just worked for 30 hours. Like, it was just spinning there on a computer for 30 hours and came out with a really good working Slack-like, Teams-like app. It's pretty incredible. That is nowhere near built into any of the existing products. Maybe Cognition is—Cognition, I think, has always bet on a longer-running, more independent agentic suite.

And maybe this is the moment that really hits PMF for them, for example. Yeah.

Just coding, I think.

Yeah, it is there. It's reading files and writing code and running tests in exactly the same way that a human would. Basically, you can think of the model as running in a loop where it can constantly decide what to do. People often mention something called tool use, and tool use is the ability to—in this case—use tools like read file, write file, et cetera, or run code in the terminal.

And it is sitting there in a terminal on a computer in a loop, just constantly looking at the current code, deciding, "Oh, well, it can't quite do this yet, so I'm going to work on that next." It's often making plans, particularly to run for 30 hours. One of the things that we're pretty happy with about the recent launches: we finally taught the models to use what's called memory. And we've built that into the agentic harness. So it's able to create a Markdown file of to-dos and things that it thinks are important to do, check them off, work on them, and check whether they've been completed.

There's almost this self-verification loop. One of the things that people were worried about with language models over, I think, a year ago or so was that they would fall off track, they wouldn't be able to self-correct, and that this would basically ruin their utility. I think one of the things that's maybe remarkable about the current generation of agents is that they can self-correct. In fact, they're astonishingly good at self-correcting. And this emergent ability has been pretty helpful.

Right.

Yeah, exactly. If you can maintain long-term coherency, then the model is able to do things that it couldn't possibly have done. If I asked you to just, in a single stream of thought, write a working version of Slack or Microsoft Teams, you wouldn't be able to do it, right? You have to sit there and take notes, do this closed-loop feedback system. So long-term coherency is really important, and it's something that we think is just really critical for this.

I think a good way of measuring this is to look at the METR evals. They're probably my favourite eval at the moment. And what this eval is, is they've taken a bunch of tasks which humans do, particularly in the machine learning or programming context, and they've annotated how long it takes a human to achieve strong performance at those tasks, and then they ask AI models to do them. And what they found is that there's this really strong relationship between progress and the time horizon over which the AIs are able to complete tasks.

And so I think it's like every couple of months, the time horizon that the AIs are capable of doing is doubling or something crazy. Maybe every six months the time horizon doubles, which is just utterly insane.

Now, again, like all benchmarks, this one is imperfect, right? It only measures pretty simple tasks. It only measures, I think, 50% success rate or something like this at the task, not like 99% success rate, but it's a good directional measure. And it certainly resonates with my own experiences of, as I've been using the recent models, I start to feel if I just set everything up right, I feel like I could leave this overnight and it could just churn away and it would probably have something pretty useful for me in the morning.

Yeah, I think in this case, the Slack-like thing is a pretty good example, where it's a significant piece of software. It's really like an end-to-end working piece of software, which often takes a bit of time, like not an MVP demo. Other things I think are interesting is your machine learning experiments and stuff like this are pretty interesting. You want something that's able to propose an experiment and write a bit of code, run some initial tests, come back later, et cetera.

It opens up the world pretty dramatically. Basically, working software rather than demos, I think, is the key thing.

Fascinating. Now, I'm not saying that the models will spin you up a full working software right now. It's not going to spin you up a Slack competitor.

Yes. Right.

And it got some pretty complex features. Like, it got Artifacts. Artifacts is a feature where the model is able to write code, and then the results of that code are displayed in the web browser. Claude.ai with Artifacts, with everything else. I can't quite remember how long that one took, maybe a couple hours to do. But basically, regard this as the first halting steps of this. It kind of works. Sometimes it won't work. Sometimes it will. Over the next six months, over the next year, expect dramatic progress here.

And look at where we are now versus where we were a year ago. And the difference is, I expect the same jump, basically.

I think the biggest things here are, or the question that we often ask ourselves is, what is preventing the models from working for longer, basically? Or when do you need to intervene? And I quite like the model of interventions in a Tesla sense as an example, because right now you need to intervene quite frequently, but it's usually on questions of taste rather than questions of raw programming ability. It's not like the model is unable to, when it's decided to do the right thing, do it.

But sometimes the models take shortcuts, and sometimes the models forget the overall structure of what they're doing, and they lose themselves in the context. They're doing a locally sensible change, but it doesn't actually make sense in the global context of what they're trying to achieve. And so I think a lot of the improvements, both that we've made and that are still to go, are on this taste and context, basically. It's on making the model better able to decide smart things about the overall structure of the program that it's going to do, and not take shortcuts and write sensible and good code.

Memory is also very important because the models do eventually run out of context. And being able to manage memory over time and even learning from experiences is something which would probably help this a lot. You don't want the model to be constantly rediscovering facts about how a particular system or codebase works. And now this is actually one of those areas where the question of taste or the bitter lesson comes up, because you can imagine us going and launching a massive effort to teach the models coding taste.

And that could be one way that you solve taste. You have heaps of human software engineers decide, well, no, this is good, or this is bad, or whatever. Where does taste come from in software engineering? Or what do we regard as taste? It's typically that it easily sets you up to make changes later on, or so on and so forth. Or it's easy for maybe multiple agents to communicate with each other and collaborate. Often, good abstractions are something that you or I could work together on in a codebase and not conflict with each other.

Right. And so there is this question of how much do you focus on teaching the model coding taste via getting software engineers to decide what is good or bad? Or should you be creating a society of models that all have to together code a giant monolithic codebase, and if they're arguing, then it's bad? You can sort of imagine the spectrum of potential strategies, and picking the right one there is a difficult thing.

That I can't really talk about. Yeah, I mean, I think it's important to recognize that it's not one individual breakthrough, really. It is the continuous application of lots of different things across the entire stack for many people. And it's mostly just a function of compute in many ways. There are obviously individual breakthroughs, but fundamentally progress has been pretty smooth. Like, on the METR eval, if you look at progress over the last two years, you can plot it with a straight line.

And so, similar to Moore's Law of the past and this kind of thing, even Moore's Law is made up of lots of individual improvements. It's not any one critical breakthrough. It's more the accumulation of a huge amount of work in an environment where there's a sort of exogenous force of compute pushing progress forward.

For those listening, a good way to understand, at a high level, the difference between pre-training and RL: pre-training is like skim-reading every textbook in existence, and RL is like doing the worked problems and getting feedback on whether you were wrong or right. And there are actually a lot of things that you can only learn via RL. A good example of this is the skill to say, "I don't know," in response to a question. Because in pre-training, remember, you're trying to predict what text is going to come next in all of these textbooks, the entire internet in the world.

And so the only reason you would say, "I don't know," as a pre-trained model is if you think the character that you're modelling in the text would say, "I don't know." Like, if it's a likely completion, right? Not whether you, in fact, don't know, but whether you think that the sort of player that you've pulled from this cast of characters that you could model would say, "I don't know." Whereas in reinforcement learning, you could, in theory, set up a battery of tests where there are things the model knows and things the model doesn't know, and you could reward it for correctly answering things it should know and penalise it for falsely answering when it doesn't know.

And what it will then learn to do is, it will learn to look up information inside itself and assess its own confidence in whether it knows that information. So saying, "I don't know," or solving hallucinations intrinsically requires reinforcement learning in many ways. So that's one example. There's a whole bunch of things you can't otherwise know. I think also an important change in this sort of era of reasoning models and RL on language models is, at the end of last year, RL on language models finally started to work.

And I think OpenAI deserves a lot of credit for releasing the first serious RL-plus-LLMs release with o1. And I think this really kicked off a pretty substantial change because it opened up a new axis of scaling. There was pre-training scaling, and now there's test-time compute and RL scaling. I think this is something which all of the research labs were investigating already. One of the reasons that DeepSeek was able to follow so fast was that they'd actually already released papers in the direction of doing RL on language models before, for example.

And so it was already an idea in the air, but OpenAI deserves the credit for crystallising it, releasing it, and detailing the first public existence of those scaling laws.

Yeah. One way of thinking about this is test-time compute is doing a lot of reasoning, and then RL is the feedback signal on whether or not that reasoning was right or wrong. And so test-time compute is a way of answering questions that are hard for you to answer. Let's say I ask you a question that you just know off the cuff, off the back of your hand, basically. It's not like from a field that you really know or whatever heuristic that you've already done.

You've already baked that into your muscle memory, so to speak. But for something which requires you to really think and really learn, like when you're first doing math, if I ask you a basic times table right now, you can say that like this. But if you're a kid, you have to do out the math and all this kind of stuff. You need to do the reasoning chain to learn it, and then you get feedback on whether it's right or wrong.

So test-time compute lets you do harder problems than you can currently do off the cuff. And RL then allows you to sort of distill that back into the model. It's almost like a ladder. You can constantly do slightly harder problems because you're learning strategies to do harder and harder and harder problems.

In some ways, it's quite funny.

Okay, yeah, how do I say it? Let's take the DeepSeek paper, for example. In the DeepSeek paper, they detail, one, an approach that works, and two, a lot of approaches that don't work. Actually, some of the approaches that didn't work were the approaches that led to AlphaGo's success. One of the craziest things about RL on language models in the RL from verifiable rewards regime is it's almost the simplest possible thing. It's almost too simple to work. And this again comes back to that question of taste, where really, I think a lot of people thought this was just too simple to work.

And so they tried more complex methods that ultimately ended up being harder to get to work. And there may still be juice in those methods, but it was actually really important to nail the simple thing first. And so I think people were almost too ambitious with the RL strategies that they tried initially. I think there's also a minimum bar in LLM quality that is required. You need the model to be able to solve meaningfully difficult coding and math problems before you can get that feedback loop of, well, you solve these ones right and you solve these ones wrong.

And I think also one of maybe the unintuitive things is those reasoning chains of tokens. People for a long time thought that you'd need to do something clever to give the model long-term coherency. You have to remember that two years ago, 8,000 tokens was a long context for a language model. Five years ago, 8,000 tokens was long. And now models are using 8,000 or 30,000 tokens to reason about something, right? So there was this real phase shift in, oh, language models are smart enough, underlying priors, that they can solve sensibly difficult questions.

They're actually reasonably coherent at longer contexts than we thought they would be coherent, and this ability to reason in long chains of tokens can emerge naturally with the right feedback signal. And this is a little bit counterintuitive. I think most people wouldn't have expected off the bat that the ability to reason would emerge naturally. There was a lot of thought that you'd have to structure it, you'd have to provide strategies for it to do reasoning, you'd have to build all these things, prompt it and hint it and this kind of stuff.

It actually turns out, well, no, give it math questions, tell it whether it got them right or wrong, and the model will learn. It comes down to a bit of a lesson in scale and search: just allow the model to search, have enough compute to run the experiments, and the model actually ends up figuring out a really effective and sensible strategy.

Yeah, there's, like, a minimum base model quality, minimum amount of compute for RL, sort of trust in the ability for long-term coherency, doing the simple thing that works. These all sound obvious, but they're actually a little bit counterintuitive sometimes.

Yeah, I think it's sufficient.

Yeah, there's a few definitions that one could use. I think a useful one is better than most humans at most computer-facing tasks, because I think that's a really important moment for the world where we go, okay, intellectual labor is addressable via this set of algorithms, and that totally changes the world. I think there are other definitions that are stronger that you could use. One of those is stronger.

Yeah, sorry. So, I mean, harder to meet, maybe. Yes. Because you could have this and it could still not learn as effectively as humans, right? We learn and generalize from very few examples. We have incredibly high, what's called, sample efficiency, whereas AI models need hundreds or thousands of times more experience, hundreds of thousands of lifetimes, basically, to learn the things that we learn. And they can, over those thousands of lifetimes, learn the skills that we do to an incredibly high degree of accuracy.

I think one of the important changes over the last year has been that RL has finally meant that we have sort of this algorithm that allows us to take a feedback loop and turn it into a model that is at least as good as the best humans at a given thing in a narrow domain. And you're seeing that with mathematics, and you're seeing that with competitive coding, which are the two domains most amenable to this, where rapidly the models are becoming incredibly competent competition mathematicians and competition coders.

Right. There's nothing intrinsically different about competitive coding and math. It's just that they're really amenable to RL, more than any other domain. But importantly, they demonstrate there's no intellectual ceiling on the models, right? They're capable of doing really tough reasoning given the right feedback loop. So we think that that same approach generalizes to basically all other domains of human intellectual endeavor, where, given the right feedback loop, these models will get good enough that they are at least as good as the best humans.

At a given thing. And then once you have something that is at least as good as the best humans at a thing, you can just run 1,000 of them in parallel or 100 times faster, and you have something that's actually, even just with that condition, substantially smarter than any given human. And this is completely throwing aside whether or not it's possible to make something that is smarter than a human. It seems entirely plausible, right? The brain is ultimately a biological computer.

It seems possible to make a better one. The implications of this are pretty staggering, right? Which is that in the next two or three years, given the right feedback loops, given the right compute, given the right elbow grease and this kind of stuff, we think that we, as the AI industry, are all on track to create something that is at least as capable as most humans on most computer-facing tasks, possibly as good as many of our best scientists in their fields. This is really wild.

It'll be sharp and spiky. There'll be examples of things it can't do and this kind of stuff, but the world will change.

Yeah. I think that it's true that our models don't learn anywhere near as efficiently as humans do, right? They take 1,000 lifetimes to learn. But this is, I think, fine because they can live those 1,000 lifetimes, whether in simulations or doing a job at 1,000 firms and so on and so forth. I think that maybe I would disentangle. There's two arguments. One is architecturally that transformers are insufficient. I don't think that's true. I think we haven't yet really found anything that transformers haven't been able to model, provided sufficient data and sufficient compute.

I think RL as an objective is a pretty powerful one. Rich Sutton is actually a big fan of RL as an objective. He just thinks we're actually encoding too many priors in pre-training and this kind of thing.

Yeah, it's not an adequate representation of the world. I think so far the evidence indicates that our current methods haven't yet found a problem domain that is not tractable with sufficient effort. Things that would make me eat my words is if there was some domain that we put a lot of effort into that just didn't move, the benchmarks just didn't move, and we just couldn't make any progress for a year, then I would be like, okay, yeah, there's some fundamental limitation here.

But instead, what I just constantly see is every time we make a benchmark that measures something we care about, progress is incredibly rapid on that. Yeah, I think this is worth crying from the rooftops a little bit: guys, anything that we can measure seems to be improving really rapidly. Where does that get us in two or three years? I can't say for certain.

But I think it's worth building into respective worldviews that there's a pretty serious chance that we get something that is AGI.

Exactly. And I mean, people have said that we're hitting a plateau every month for the last three years. And if you look at where we've come over the last three years, it's incredible. I think that one other thing that makes me think, God, we're not anywhere close to a plateau, is I look at how these models are produced, and every part of it could be improved so much. It is a primitive pipeline held together by duct tape and the best efforts and elbow grease and late nights.

And, God, actually, I remember—I don't know if this is a good analogy or whatever—but I remember I went sailing with a couple of friends a few months ago. And the boat was so well designed. It was just clearly the product of millennia, or centuries, of accumulated human design and effort. And I was like, wow, this is what it feels like to be in the accumulation of a lot of human effort, right?

It's actually pretty hard to beat today's best sailboat designs. Five years of best effort, last-minute desperate effort, and there's just so much room to grow on every part of it.

One of the things that I was really excited by, actually, was that GDPval that OpenAI released. Claude Opus 4.1 was the leading model there. And I think that's a really interesting and really good eval because it demonstrates such a breadth of tasks across all parts of the economy.

I've wanted someone to do this for a long time. Take the Bureau of Labor Statistics. I think the most important input into policy would be: take the Bureau of Labor Statistics, take all the jobs there, break them down to tasks, and see whether the AI models are able to do that, and measure progress over time. Right. And this is obviously going to be an imperfect measure. We'll probably reach better than human on GDPval, and it won't change anything economically because it'll be all the connective tissue and all the context, and actually the tasks won't be representative.

But again, we'll then find better ways to measure these difficulties, and we'll keep pushing benchmarks. I've wanted someone to do this for a long time. I'm really glad they did it. I'm really glad that our models were general and generally strong and sort of showed up top across all the areas. And I think policymakers should really look at this and extend it and really make an effort in investing in figuring out whether we are on track for what I've been claiming we're on track for.

Right. We can measure this.

And we should be.

Yeah, I think the most actionable piece of advice is: keep planning for a world where you, as an individual, have more leverage. Right now, I can use two coding agents to do twice the work that I could have done before. If coding agents progress in the way I've been saying, in a year or two, you'll be able to manage a team, basically, that works 24/7 for you, doing work. I think we should expect, in the digital domain, for individuals to get dramatically more leverage over the next couple of years.

I think there are many incredibly important problems we're going to try to—our world is so imperfect in so many ways. People still live in dramatic poverty. Health and medicine are unsolved. Housing is completely unsolved. The world could be a million times better in so many different ways. And what I hope is that people take, initially, models giving us leverage over the digital world, and then hopefully models giving us leverage over the physical one through robotics, to dramatically improve it.

Right.

There's this thing called Moravec's paradox, right? Which is that things which we find really easy, like manipulation, picking up objects, are really hard for AI. But maybe things which we find hard, like reasoning through mathematical problems, are easy. I actually think Moravec's paradox is a little bit fake, and I think this is mostly a question of data availability and RL signal and stuff. And I think one interesting reason to look at this: if you look at robotic locomotion, so the ability of robots to walk around and balance and stuff, and look at the videos of the Unitree robots, the difference now versus two years ago is crazy.

These things are incredibly agile. There's this video, I think, of someone kicking one over, and it literally does a Matrix kind of get-back-up thing. It's crazy. This is because locomotion is a really easy RL signal. And right now, locomotion is kind of solved, to be honest, with basic RL. Manipulation is a bit harder, but there's a few things that make me think that robotics is going to work. For starters, I've seen incredible progress from the robotics labs over this year.

Really, they've gotten to the point where they can do pretty interesting basic physical tasks. Two is the existence of a large generator-verifier gap, which is that one of the things that makes improving our models hard is we constantly need to find people who can beat the models at the things we want to improve them on. But with robotics, we're making really smart general models. So you can actually have these as teachers or judges for whether or not the robot is doing the right thing.

If I say, "Stack the red block on top of the blue block," we can then ask the language model, "Did it stack the blocks appropriately?" If so, give a reward. If not, don't. So you can use the generator-verifier gap to give models feedback. And finally, for a long time in robotics, people thought they would have to solve long-term coherency and planning. And that's also something that language models have made easier. They can break things down into multiple steps. So all the robotics labs are focused really hard on making great motor policies, and they're making incredible progress.

It's mostly just a data and feedback loop question.

Thank you very much.