Open Source AI Strikes Back — Inside Ai2’s OLMo 3 ‘Thinking"

I want to add more color to these things. I think Luca's underselling the base model. We're going to talk more about this, but over this year, a lot more people have been releasing open models, especially large open models. But some people are starting to not release base models. We have a bunch of DeepSeek-sized giant MoE base models and a bunch of small base models. But, for example, Qwen 3, which everyone accepts as a research standard and an industry standard, they don't have this 32B base model.

OLMo 3 32B is still the best base model. The upside is that we have all the data, so people can actually do some sort of continued pre-training and hopefully make it a bit easier to modify and understand the behavior. That's exciting for us, though. The actual potentially best-in-class thing. It's also a fully open thing, which is not something we get to say a lot at Ai2. Sometimes it's like, oh, we replicated this and now you can do it yourself.

This is actually a good thing. And then 7B models, which Luca was saying, used to be this huge industry standard where there's just so many of them. It's still a standard size category, but there aren't quite as many models there as there used to be. Especially the instruct models are less common, and this is up there with one of the best in the world there at that size category again. OLMo 1B is one of the most used models on Hugging Face of all time, and this should be better than OLMo 1B, I hope.

I hope that holds up for people. We can release more and fix it. But that's just trying to give—we might not be at this frontier scale, but these are things that are still widely used in the world. Then it's the first fully open reasoning model where we show doing RL on base models and distilling from bigger thinking models and all these things that people have seen a ton of times throughout the year. I think another thinking model is like, ooh, what is this one for?

But we have all the data and we show people what to use it for. So I think a lot of times with our, especially open post-training, it's just like the datasets become a standard. So it's like our Tulu 3 dataset from last year, which we use for OLMo 2, is in, like, the Thinking Machines Lab Tinker API. And we want people to use this data, modify it how they need to, and look at the different training stages.

So we have that part.

I would say in post-training, we're the best models that don't start with Qwen 3. We're reasonable to say that they are comparable to Qwen 3. On some benchmarks, we beat them, but on some benchmarks, they're way ahead. I think a lot of people are like, we don't know what Qwen 3 puts exactly in the training data, so we don't know if some benchmarks they benchmarked MATH a little harder than we did. I mean, we try to hill climb on benchmarks to make our model good.

I think there's always some level of this. But in that, it's in the same ballpark, plenty of things. We're hoping that there's use cases where people that use Qwen 3 8B or 32B are willing to switch over and get some value out of this and maybe modify it to their own use cases. But it's like, Qwen also releases great models. So it's like a never-ending uphill battle that motivates you to do better, to try to get close and compete with what they're doing.

It's like they released these Qwen 3 VL, their vision models, and on text-only benchmarks, it's way better than the models they released in April. So it's like, okay, that's the new baseline, and most people don't know about it because they think it's just a vision model, but it's actually a much better text model. The bar's always rising. But at the 7B scale, NVIDIA had Nemotron Nano V2, which is a 9B hybrid model, which I think is almost equivalent to our 7B model.

These are good models. There's not that many of them that are in these size bands that are really strong. Happy to be there and happy to point out other people are doing great work here. It's not like we can ignore Qwen. That's a losing strategy.

We've always got questions about intermediate checkpoints during SFT or other fine-tuning stages. Now we have intermediate checkpoints during our supervised fine-tuning for reasoning and for instruct, and then also for our multi-day RL rounds. At the end of these, we have intermediate checkpoints. So people that are looking—a lot of people like to understand checkpoints and do research on them, but don't have compute to train. And now it's like, okay, this is all there.

Yeah, if I try to make a list of actual models, I'm going to forget some because there are so many that are notable. I think starting with DeepSeek, as you mentioned, is definitely the important thing. And then if you talk to people building models in China, a lot of the consensus is that DeepSeek showed us that AI could be a big deal and that a lot of these companies were like, oh, we should do what they did. So there's just a ton of labs that have popped up over the year.

Z.ai and Kimi Moonshot had already existed. These really stepped up to be much more known names, especially if you're following Western, SF-centric discourse. These are things that people are using and talking about, which is a kind of big change. But there's just this huge mass of models coming from China. You have everything like Ant Group is releasing trillion-parameter MoEs with really strong benchmarks. Meituan, which is the Chinese equivalent of DoorDash, which is just another big tech company in China. The standard way of developing language models has become to release them openly, and that whole ecosystem is going forward with this, figuring this out.

When, at the same time, there was a big change in leadership at Meta, and Llama's future is less known, which was really the paradigmatic definition of open-source AI. That line of thought just ended. So there's this big vacuum of influence which has been absorbed by the likes of Qwen, DeepSeek, Kimi Moonshot, in terms of who's trying to build things with open models. That's a big shift. I think there's a lot of discussion within the U.S. that there's good reason that we should own, we should at least have influence over the whole technological stack, and that includes open models.

Realistically, it's the big tech companies in the U.S. that'll capture the downstream value of that from having the researchers be in close proximity and speaking the same language and used to the infrastructure. I think this is something that we've seen for decades in the tech industry, so I don't think I need to explain it that much. There are people that are really starting to wake up to this. I think June, July is when the Chinese model providers were really becoming like, you could not ignore them.

That's when we had the Kimi K2 Instruct, GLM, and that's kind of just continuing now. So I think when we're recording and releasing this podcast, there's a lot of interest in, like, what are the U.S. companies going to do to respond to this? I know that NVIDIA is making a lot of noise here. They invested a lot of money in Reflection AI, and there are other players that are trying to get going. But urgency—and we don't have a lot of compute at Ai2—but if we can make a dent in this in some model sizes that people actually use, I think that—we focus on researchers.

I think dense models are great for researchers, so they take a little bit less compute and engineering resources to use. And that's—I do think that there's more U.S. participation. I mean, OpenAI has released some models, where they're just established in a different way.

The other famous quote was a Martin Casado quote in The Economist where he said 80% of companies are building on Qwen. That has been corrected, where it's 80% of companies building with open models are using Qwen, which is like 16 to 24% of his portfolio, which is still a lot. It's a meaningful amount of people who are trying open models for things, and most of them are using Qwen. Then there's the likes of Cursor, who released their own model, Composer 2. It's suspected that it is built on a large Chinese MoE of some sort that was released openly.

There's some obvious tells of it switching to Chinese and things like this, but that is the sort of company that doesn't want to pre-train their own models but has immense value in specifying models for their use case that is just going to build on these great models. I think they would want more options to choose from as they try to sell into more markets. I think realistically, it's a thing where a lot of U.S. companies don't want to deploy Chinese models. I think currently a lot of the stated reasons are just unknown unknowns and things you can't prove.

You can't prove that the models aren't doing certain backdoors, where I'm fairly certain they definitely aren't now. But just because you can't prove it makes this kind of weird market dance, which is like, yes, these are stochastic things that are kind of amorphous. And it's like, I don't love being in the middle of this as a researcher, but it's like, I would like to just provide information and good things that people actually really want to use and leave all of the geopolitical and other messaging to people that have probably, realistically, way more on the line than I do.

Like, I don't know, we work in a nonprofit. I have my dog.

Historically, the U.S. has a lot more willingness to pay for services. I hear anecdotes from people that know China a lot more than I do that are like, yeah, medium- to large-, billion-dollar-plus valuation companies in China will just pirate SaaS software. I know that sounds worse than it is, but it's just, I think the thing is that U.S. companies are used to paying for services, and that API model and paying for tokens has been proven as a very good—selling tokens is a good business in the U.S. right now.

I think there's a lot of debate over profitability, but the demand and usefulness of these tokens is high. So I have a lot of belief that there can be profitable businesses from selling tokens. Where I think that AI will be embedded in very different ways when it comes to Chinese companies, and I've talked to a few of these labs, and they're like, in order to sell into the U.S. market, they will not pay for—they've said this—U.S. companies will not pay for services.

So they don't expect enterprises to sign up for the Kimi coding plan en masse, but they're like, we have a chance that they'll use our models. And it's like, that is a practical way to influence and getting a piece of the sharing pie. And it's like, the people building these models in China know the same things about the different ecosystems. That's why I've enjoyed starting to talk to them. It's like, oh, these people, that's the same thing. They see the same constraints.

It's not that complicated. U.S. can't ignore it. And that's their way to have a part in this ecosystem. So there's a mix of the DeepSeek standard, and then they're kind of like, yeah, this is something that works for us. Let's keep doing it. It's getting them a lot of mindshare and some use in prominent ways. So I think it makes sense.

I think any concerted response you only see when it actually is public. And I think there's a lot of investment at different stakeholders and conversations that are happening, but that's not that useful. So it's like, I don't have the proof for you, but I do think the right people are talking about it and want to invest more because, realistically, the cost is not that high relative to the trillion-dollar buildout of AI infrastructure. 0.01% gets us better, great open models. Like, we should probably do that.

I think that's actually not that complicated. It's just like, how do you get the $100 million line item to the right people that have the talent to do it? And they're like, oh, okay, the right incentives. It's just like, okay, it's a lot. Like, the Reflection AI news is big. It's like, okay, that's probably a good solution for a couple of years. Like, they have enough money and they have a strong base of talent. And it's like, okay, that's like a major checkbox.

We need to have diversity there because the Llama thing could happen again where it goes away, but looks like a small snowball, but hopefully grows in the coming months.

I would say I launched ATOM in July and thought it would get more visibility, but now I'm getting a crazy amount of media inbound and press inbound, and everybody wants to share the plight. So it was like, okay, I guess I was just four months too early, but that's what I don't really mean. It's like, just today I saw Bloomberg published a post that pretty much had the same title as my Kimi post from July. I was like, okay, I'm glad that people are paying attention now.

It's like, better late than never.

A lot of people have heard about inference-time scaling, which makes sense. If you spend more compute at inference time, you get a better answer. A thinking model is really a way to train the model to exploit that a lot. So you spend a lot of tokens, which are usually hidden from the user as a long chain of thought, and the model therefore has this step change where it's way better at math tasks, coding tasks, agentic tasks. I think our future plans are adding more tool use to the models.

We're not talking a lot about agentic search or agentic code execution on the fly and stuff for this model. But building thinking models is the gateway to doing a lot more interesting things, like Claude Code. Maybe we'll have OLMo Code next year and all these things that we want to do. The thinking model has just been the thing in 2025 that uses a lot more compute per answer. The model gets way better. I don't like thinking models, but it's fine. No, they're good.

They're very useful.

Yeah, I say I wear many hats to try to get the things that I want to do done. I showed up to Berkeley as an EE PhD admit in 2017, and then I saw that AI was happening and I decided that I wanted to try to do this, which started by going to all the names that people know, like Sergey Levine and Pieter Abbeel, and asking to be in their group. And then they respectfully said no. And then starts the long process of learning how to actually do it without being directly embedded in these elite groups, which was a mix of robotics and reinforcement learning and finding my way there.

My PhD was mostly in model-based reinforcement learning, and then my one research job was to go join Hugging Face when they said they were going to make an open-source version of DeepMind to do a bunch of research. Realistically, my job was not that impactful or useful at Hugging Face until ChatGPT came out, and then I was like, oh, I should maybe just learn about RLHF. That got very immediate traction as somebody trying to work in public with the team there.

So Lewis Tunstall and other people at Hugging Face are still doing a great job on this, and we worked together for a while. And then mostly I was just getting burnt out on remote work and met Luca in Hawaii at a fun conference and was like, wow, I could have real-life friends. And I joined AI2 to work in person and tried to do the same thing, which kind of takes an evolution of the OLMo story, which was just like I had a lot of motivation in trying to figure out these—what was mostly reinforcement learning from human feedback at the time.

And make versions of these post-training techniques public. Then that evolved through both OLMo, and we have our post-training methods named Tulu, which is like, we spent a long time trying to replicate what we thought was close to Llama 3 post-training with multiple stages and optimizers, which is the project that came up with the name Reinforcement Learning with Verifiable Rewards with a bunch of people. So it's kind of this evolving journey at AI2 in search of impact, which is what we think people are actually doing.

And then largely, the opportunity that Luca and I and others at AI2 fill is that there's so much money in AI, and it only becomes increasingly so, that the amount of people that can talk about these things in public and educate and get more people involved by spreading knowledge is ever smaller. So I describe my career journey as a lot of it is filling that vacuum and thinking about what's impactful. So it kind of pulls you when there's such a void. It has a sort of gravity to make it clear what you should be doing.

Yeah. I mean, I've been here for two years, and I wasn't famous when I joined. So let that be told to people looking for new jobs: you want to find a job that you can grow into. And I think AI2 has been a really, really good place for that for many people because you have independence and are encouraged to go forth and do things and not be a cog in a broader grind-out-language-model machine, which is important, but it's harder to get visibility.

I'm going to jump in. It's cool because it's the namesake of our team name and has been for a long time at AI2. And it was the thing, it was the main competitor to Hugging Face Transformers, and they ultimately outcompeted AI2 as the thing that people use for that because they had a very different model and amount of support. Luca can keep going. Luca knows a lot more.

Sure. I recently gave a talk on this at the Conference on Language Modeling, so I have them on the top of my head. I think I could provide a personal motivation for this, which I think, as researchers, we're closely embedded in the community, and we see that there are a lot of people that are starting to do this reinforcement learning research after DeepSeek-R1 and Qwen3, between 1 and 8B parameters. I think something that particularly motivated a lot of the fine-grained details that we might not have time for in this podcast is that there's some questions hanging over the data used for Qwen when doing this RL research.

Specifically, there's two papers. One is "Spurious Rewards: Rethinking Training Signals in RLVR," which is one that I was on with a lot of people at UW and Ai2. And then another one, which was "Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination."

I think the thing to know about this is that you can trigger—you'll hear this rant on the technical side later. It's a lot of background on understanding what these algorithms are. But essentially, the question mark is: did Qwen include training data that is too close to the evaluation targets so that the research is picking up on weird behaviors within the model rather than the fundamentals of what this reinforcement learning is doing?

It's a gray zone. I think all the frontier labs will do this to some extent, which is how they're tasked. You have a team member that's tasked with improving an evaluation, and then the easiest way to do this is to train on test. But they all have dignity as elite scientists, where they won't do this. The next closest thing is you do some sort of paraphrasing of the test set to create new training data. Therefore, you're not technically cheating, but you're potentially cheating.

It's like, where in the spectrum of you scrape GitHub for math problems versus you paraphrase the evaluation set? Where do you draw the line on actually calling it cheating? Different people have different answers, but mostly I think a goal that we kept coming back to, because we understand that OLMo is not—you can look at the numbers, we're getting close to Qwen3 with reasoning or without—but this is not a 600-billion-parameter model that people are going to immediately download and run OLMo code on or anything.

But we want to make sure that our core audience could do the research that we want to do with confidence and debate it. We want to give people access to every stage, and you can then see how this impacts this new important area of research. We're going to talk about six stages. One is large-scale pre-training, which is this training on all of the internet, predicting next tokens. Two is what we call mid-training, which it's debatable whether or not it actually should exist.

What technically it is, is you train on higher-quality web data and with a change in the learning rate. Three is long-context extension, which is absolutely essential for these reasoning models because they generate so many intermediate tokens before sharing an answer with you. Luca has a lot of battle stories from that. Then we go into post-training, which in our case—those three building blocks of pre-training are, I would say, more set and super essential. Then post-training, when you approach this, you have a bag of tools, which are optimizers, and you apply them in the order that suits your model depending on size and capabilities you want.

We'll talk about things that we did, which is instruction tuning, preference tuning, and then we did some reinforcement learning with verifiable rewards again. But if we were to train a model that was 10 times as big, all this post-training stuff would change. But the pre-training and mid-training and long context would actually become looking pretty similar. It's kind of a difference across two phases of training, where post-training is a bit of an art and you have to do what is best for your specific use case, and that'll change.

But we can go through these two.

I would say that generally the reason why discussions are hard right now on whether or not people should care about pre-training or post-training is that we optimized pre-training for multiple years, and then there was a lot of untapped potential on this type of RL. What is said is that OpenAI figured out a whole bunch of tricks to get o1 to work, and then it showed that this area was possible. And then this year has been a race to capture low-hanging fruit on RL.

I think that's kind of the biggest story, is why we have all these crazy new models that appear, like o3, with this thinking and tool use, which are just downstream of, oh, we could do very different things because we have such a good platform of these malleable pre-trained models that we've been iterating on for a long time, that this RL stuff, we just kind of could have tapped into it much earlier. But there's a lot of potential. Yes, the rate of improvement right now in RL is higher, but at the end of the day, it's going to be a dance between both of them, where you need a better base model.

It's said very commonly that a better base model and a bigger base model is much easier to improve with RL. So if you take that as one of the core things of doing RL research, it's pretty obvious that pre-training is very important to enabling that.

My take is that a lot of people are being exposed to Rich Sutton for the first time, and Rich is a font of wonderful ideas, but often not ones that are going to be immediately practical. This is how you get things like creating reinforcement learning, but not necessarily things that are going to impact what GPT-6 is. So I've been on the critiquing Rich train for many years before this in terms of making people try to interpret his ideas as realistic. I think the one from 2021 or 2022 was his Reward Is Enough paper, which essentially is an argument that a reward function is sufficient to get any intelligent agent that you want.

So I think that that's actually, rather than the technical debate, is an entertainment of the whole community being nerd-sniped for the first time by that.

Okay.

Okay.

Wow.

No, Luca loves data. Luca likes to be in a dark room grinding out tokens to train the models. That's the emotional backdrop for this. It's very painful.

Yeah. One of the things, especially for a model like OLMo where we're scrappy and putting everything together over time, is that one of the biggest changes is that when reasoning models become popular, the in-vogue evaluation suite of the industry shifts to add a whole bunch more new things in. So one of the things that happens at every stage is that even if a lot of the data has overlap, you mix it in a different way. I think Kyle Lo and Mei Chen, that's another researcher and an intern, did this whole mixing procedure that we use across all these stages.

Just to upweight the math, code, and reasoning stuff to make sure that what happens later in post-training is much more tractable and that all this stuff is set up. That's the type of thing that we have to do that's baked into everything. Then post-training, I think for this model, everything we're doing is operating on the assumption that this is about a 7B model. We are very narrowly focused, and therefore we're going to do what many people have done, which is called distilling from bigger teacher reasoning models.

I think distillation is described as when you take the outputs from one model and then you fine-tune on it later. I think there's been a lot of broader discussions on this in the community. Then this supervised fine-tuning stage, or SFT, or instruction tuning, is all about just getting the best traces from reasoning models out there or the community and then just teaching your recently trained base model to behave really, really closely to what is going on there. In our case, we took a mix of existing datasets like OpenThoughts-3 and modified it, which is from Bespoke AI Labs, a startup.

Then we also generated a whole bunch of new data. So we ended up using a mix of teachers from DeepSeek-R1, DeepSeek-R1-0528, which was their updated version, and then Qwen's reasoning model, QwQ. These tend to be pretty strong teachers.

Essentially because our small model is not going to be able to output as strong of text. There's a kind of fork in the process where I'm talking about a small model. If we had a bigger model, what we would do is do a lot of reinforcement learning to start, and the model then would take time to learn these interesting behaviors and have strong performance. But with a smaller model, the ceiling on that is fairly low. It just doesn't have the capacity to learn from these harder math problems. So what the common practice is, is you take the absolute best reasoning models you can get that are openly available with a good license, where you can just generate new data yourself and train on it and release it to the community, which is something we've been seeing a lot of this year.

Therefore, the models that are closest to the frontier in performance with a good license all happen to be Chinese models throughout the year for this case. I think in our case, even if GPT-OSS had existed, I don't think we would have used it for synthetic data in this because that model is really designed for tool use, which is something that we did a bit of in this project, but not in the sense that that model is, which is this many-hop agentic reasoning with search and stuff.

The DeepSeeks and Qwens of the world are just powerhouses at generating math and code answers and other things and being generally robust. Five million reasoning traces, mostly on math and code and STEM, but also on chat and other general capabilities. The model really absorbs a lot at this point. I think if you were to have told us last year, when working on OLMo 2, looking at it, that if we had a similarly sized OLMo model that gets like 95 on math and like 70 on AIME on these crazy math evals, it would have been surprising.

But this is just what you can get when you can extract data directly from these really powerful models and distill it down. I think realistically a lot of companies are going to want to do this because you can do this for your domain. I think we threw a blanket on, we want all of these evals from instruction following and make sure that you can actually talk to the model and not have it just become totally broken. But you can do this in any specific task you want if DeepSeek has coverage on it, and it's very efficient.

That is the foundation. If you're training a small reasoning model, you need to do this. Then the other things after are how do you extract more performance, and they quickly become more technical or done because we want to do them and maybe not efficient in our time. So 90% of our time this summer is having great people battle reinforcement learning infrastructure because when you generate a lot of tokens, the time or compute increase and memory increase is quadratic. Therefore, you pretty much encounter every possible bug in your framework or every possible corner case that'll make your job go to a halt.

But most of the performance is through this SFT and the preference tuning that comes before it. But the RL is like, we need to do this in order to build the infrastructure for many of the future OLMos that we want to build later this year, where they get bigger and they can do more interesting reasoning with tools and so on. So it's kind of like a nuanced point of the model. It's like, yeah, we'll show you that we got a couple of points out of doing RL at the end, but really the RL tooling is something that is so crucial to doing the next models that come from here.

So it's the same loss function as pre-training, which is you're predicting the next token. In this case, what it looks like is a question could be like an AIME-style, really hard math question. It would be like, list all the prime numbers within some constraint of x and k. And it's like this one sentence that is really hard, and then the model generates 30,000 tokens of, let me think about this and do this, and to test this, I'll have to use this theorem and hypothesis.

We were talking about token intuitions for a bit, but 30,000 tokens to solve a math problem is pretty mind-bending. So if I were to sit there and read this, it would be hours of me just trying to read this one math solution. So these models are very unintelligible in many ways. I think the reasoning models sometimes will go into a bout of guess-and-check for hundreds of attempts before realizing that they can no longer guess-and-check. I mean, this is our reasoning model.

I think the frontier models could have probably done this and fixed this issue, but there's just really, really, really odd things in these tokens. But even with that, doing this next-token prediction is an incredible foundation of performance that many people use. So it's not matching any sort of human reasoning or things that people might want it to be doing, but it is teaching the models their own language of breaking down problems step by step in order to solve a goal.

Yeah. Luca, do you want to jump in? Do you have things too?

Yeah, so this is one of the things that is thought of as like, hey, let's try this. We're not sure if it'll work, kind of later in the process, when you spend a lot of time on other things, and it works very well. I think DPO, or direct preference optimization, is not exactly new. I think it's a way of optimizing for preferences. It's related to this whole RLHF thing that we mentioned. Technically speaking, in one sentence, it's an analytically derived loss function that is essentially applying stochastic gradient descent to the RLHF objective.

So it becomes much easier to implement than other things. We used this in the past with OLMo 2, with Tülu 3, Tülu 2, other OLMos. The question was, can we apply this out of the box on top of a reasoning model? We knew that it works in many different situations, because we weren't sure what would happen with these long reasoning traces being included in the loss function and so on. So then, essentially, there's a student, Scott, that has been working on what he calls the delta learning hypothesis, which is an intuition for understanding DPO as being more about the contrast between your chosen and rejected examples.

The core of preference learning is that you have pairs, or some grouping, of completions to the same prompt. You have one question with multiple completions, and his intuition and work is showing that this contrast is more important than the exact magnitude of goodness of the answer. So what he did is he spent a lot of time trying to come up with a good pairing of reasoning models, which are open source, or open weights, and they have a permissive license and they include the reasoning traces, because we kind of need this, and you need them to be sufficiently well spread out.

So we spent a bunch of time generating this data and doing some normal kind of like, let's fiddle with the learning rate and small things. And it was kind of just like, yes, this works. After we did it, we saw that Hugging Face did something similar with SmolLM. They trained a fully open 3B model where they pre-trained it as well. The funny thing is that we converged on using the same Qwen 32B and Qwen 0.6B. The problem is that these small Qwen models and these small public reasoning models are actually so strong that getting a sufficient delta to another model to apply this preference learning technique was kind of hard.

Our past techniques, we kind of had groups of models we sampled from, but as these open models are getting better, these samples become too homogeneous for the learning signal to exist. So it's a cool experiment because it validates this hypothesis of the changing tides. If you think about years ago with Alpaca and stuff, those models were so broken that having this group had enough variance and contrast in it where we could do a different type of preference learning. Where now, you have to look really closely at the completions and make sure that there's a learning signal for the models.

We did this and it kind of gave us a boost across the board, I think. Sometimes things look very easy when you've done careful data work and set up to understanding your optimizers. I think Luca described pre-training as very scientific and post-training as the Wild West. I think there's many analogies. So it's like, we had me that made this SFT dataset where I was like, we had a bunch of cloud credits and they were running out and we were behind, and I just generated as many completions as possible.

So it's a few billion completions from DeepSeek over the weekend. You're like, oh, we'll mix it and filter it later. I applied filtering and the answer was like, oh, we just include almost all of it. We did very little. We would have liked to do more if we had resources for longer, but sometimes there's low-hanging fruit and doing the obvious thing yields a lot of results. This SFT and DPO stage, in a lot of sense, are that, and then this RL stage is extremely hard technical grinding, week in and week out, to make the tools even run at all.

And the disparity in post-training is like, yeah, that kind of tracks to me. You just have all these checkpoints flying around and it seems like chaos, and then something that's extremely obvious gives you a massive gain. GPT-5 level to almost Qwen 3 level. It's just the thing that you apply to get there is sometimes really obvious. And I think the frontier labs are much further down this path, where they take these low-hanging fruits so fast. But as a smaller team that's trying to map to what the changing priorities of the field is, sometimes it's just turning the crank on this really straightforward thing.

All right.

I mean, I think that I'm the person that got to bring it publicly to the world. It's well known that people across industry have been doing this for years. And then the technique started to get far more impactful. It's broadly taking existing reinforcement learning algorithms, or downstream evolution of proximal policy optimization, PPO, which is an evolution of REINFORCE. And then DeepSeek had their group relative policy optimization. I always try to say group robust. I think it's group relative. And all these algorithms are really quite similar, and you're training the models with whether or not they got the answers right or, in the case of code, whether or not the tests execute and don't fail.

I think one of the famous examples is that doing too much of this, or racing to get the low-hanging fruit from this RL approach, is what makes all these code models do all these try-except things to avoid errors because they accept all the errors. I think that is just because the gains that you get in the model being useful are so much higher than the annoyance and the fact that it also does these stupid things. And we'll fix the stupid things eventually. In the case of this OLMo model, it's not anything crazy.

We cast a wide net on RL math problems. We do some data comparisons to see which data we think is the best for teaching these models. We do mixing with code and precise instruction following. This mixing is effectively when you tune the big set that you have to what you've known from many experiments and to the specific model checkpoint that you're on. So if you have a really strong model and you show it really easy math problems, there's no learning signal.

And if you have a really weak model and you show it really hard problems, it gets them all wrong. There's no learning signal. So the learning signal is all from the gradient of: you sometimes get it right and you sometimes get it wrong.

RLVR, the verifiable rewards, is in the name. I think essentially the reward that you get from the environment, which is the completion or the grader, is whether or not you got the problem right. In RLHF, the reward is essentially a reward model which is rating the quality of the response based on a proxy to what humans would like. So it's described as being much—the RLVR reward is much easier to understand because these reward models tend to have a lot of problems, and you can over-optimize them much more easily because the reward models will pick up on features that are maybe emojis or something like this that you don't actually care about, whereas RLVR is much better matched to performance characteristics rather than style.

There's many ways that your tooling could fail. I think where most of these processes are set up right now is that you have a set of generation GPUs, which look like something like vLLM, and you have a set of training GPUs, which is some distributed learning framework, which is where you actually have this RL update and loss function. Therefore, you need to have some sort of system that orchestrates the two and passes information back and forth. This kind of information passing back and forth is really annoying.

It's a systems problem because you have distributed error handling and things like this. A common case is when you have the most basic approach: you'll have one generation, this one math problem, the model is thinking and thinking and thinking and thinking. So you have all these GPUs working on one problem. Effectively, your whole system is somewhat idle, waiting for the answer. There's many other small things like this, which is this long-context generation just uses so much memory that you then need to introduce different types of parallelism and stuff to do the generation effectively.

And there's just a lot of subtle numerical issues. So I think it's just kind of stress-testing a lot of the post-training infrastructure that we have had by turning up a lot of different things that could go wrong to the maximum. I think the things that the open community struggles with is that vLLM and Hugging Face use different kernels to do the actual internal computation of the model. These kernels are the things that make things like vLLM really fast. But this then results in subtle numerical differences between the completions that you're generating from the model and then the log probs that the thing that's doing the loss function actually generates.

If you look at the math of these RL algorithms, it's assumed that those are from the same distribution. Therefore, this is a big root cause of a lot of numerical problems. Then if you look at what we're doing, and a lot of labs have done throughout the years, they do different fixes to change these dynamics. Thinking Machines had a famous blog post, one of their first blog posts, on deterministic vLLM to make it exactly deterministic. That is really useful, and people think it is key to their Tinker API and doing other sorts of RL things where you just have complete control over sources of nondeterminism, and that could just be numerical lack of robustness in RL.

There's also these discussions on if the open labs have worse algorithms than the closed labs. In reality, it seems like most people are using something like an evolved version of GRPO, which is a bit simpler than PPO. Some labs might be using a learned value function. It's not that important about the details, but what happens is that each lab finds the set of tweaks that they need to get really stable RL performance. In the RL literature, historically, there's a pretty low bar on the amount of changes that are needed to call it a new algorithm, but it's realistically an implementation detail.

Everyone finds their stable configuration for operating, and it's really dependent on the tool. Yes, you could say that they have a different algorithm, but it's also not really something that you could easily exfiltrate from a lab because it's dependent on many layers of the stack and maybe what chips they're operating on. So it's just one of these things where training these models is complex, and the kind of quick quips could never reflect that.

But also, I think that I'm very far from believing in any sort of singularity being possible due to these things like complexity. On one hand, we talked about all these things which are low-hanging fruit to improve the models, and I don't doubt that—I mean, Sholto commented this on the pod and other places—where at these labs they still see low-hanging fruit in improving the models. In many ways, I don't think that their approach feels that different. They've just refined it relative to what we're doing.

But at the same time, as things get complex with tools and adding more layers to the stack, and you have to build a product to scaffold it, if the requirement to get the best out of Claude is to use Claude Code, which is some magic product in prompting relative to GitHub Copilot, this is one thing that you're going to need to get right in order to get AGI, along with all these tool uses and stuff. So, as any system gets more complex, the pace of change is slower.

I think any tech company has seen this. And then, realistically, there's going to be physical constraints on the amount of infrastructure that we can build. So this belief is simultaneously giving us these new data centers and, I personally think, hopefully new power generation, but there's a cap. All these things are plateauing, and then you 10x the compute and you get a big jump. You can't do this forever. So, realistically, there's going to be some physical constraint that kicks in at some point.

But balancing that with complex systems and the low-hanging fruit results in—I think these researchers are going to grind out improvements for multiple years, but never in a way that results in this kind of accelerating well that we get drawn into. So it's like, I don't know, in some ways it feels like I'm having my cake and eating it too, but it seems like the likely outcome if you look at other types of technology.

Yeah. And I think they have these milestones, like AI automates research engineering, and then AI automates AI research and development, where it's like each of these are incredible jumps in performance. And I think what's more likely is this messy co-evolution. They deserve credit on their marketing and getting this impact, for sure. But even they are now like, oh, maybe we should have called it AI 2028 or AI 2029.

I think the AGI word is actually pretty not useful. I think that how I describe it is that big tech has all collectively realized that these language models plus scaffolding is going to unlock absolutely incredible value. And I have very high probability, barring extreme geopolitical situations, that big tech executes on this vision across the two to five years to build 95 to 98% of the way there of what you can do with our physical power constraints and what an LLM's ability is. And I think that that will be extreme.

The transformation from that by 2030 is going to be so powerful across society. There's a bunch of long tail. There's going to be mass societal readjustment to what the internet and media and information means within five years. And that's mostly why I do this. And I think debating whether or not it's AGI is kind of secondary to the fact that this is coming and we want people to study and understand what is happening.

So I think there's a lot of interest in AI outside of the CS majors and AI people of the world, where it's informing policymakers. I think it still takes a long time for information to diffuse, and there's often not that many people that are engaging in this that are doing it just purely for this kind of—you can call it alignment and concern. There's just a lot of general noise, and I worry about concentration of power or all sorts of many things. And it's just trying to upskill people into understanding AI so they can be engaged, engaged citizens and think about how it affects their domain.