DeepMind Gemini 3 Lead: What Comes After "Infinite Data"

Thank you. Hi, Emmanuel.

Yeah, I'm not sure it's a big secret, at least from my perspective. This seems quite normal. I think people sometimes have the expectation that from one Gemini version to another, there's a big thing that changes and that really makes a big difference. In my experience, there's maybe one or two of those things that make a larger difference than other things, but it's really a combination of many, many changes and many, many things from a very large team that actually makes Gemini 3 so much better than the previous generations of Gemini.

And I think this is probably a theme that will recur later, but it's really a large team effort that comes together in a release like Gemini 3.

There's two things. The first one is it's still remarkable how much progress we're able to achieve in this way. It's not slowing down. There's so many of these knobs and so many improvements that we find on a day-to-day basis, almost on a day-to-day basis, that make the model better. So that's the first point. The second point is we're not really building a model anymore. I think we're really building a system at this point. People have sometimes this view that we're just training a neural network architecture and that's it, but it's really the entire system around the network as well that we're building collectively.

And so that's the second part.

The benchmarks definitely keep improving. And if you look at the prompts and how the benchmarks are set up, they are becoming increasingly difficult. And even for me, who has a background in computer science, some of the questions the model answers would take me a significant amount of time to answer. This is just one view. It's the benchmark view. And we evaluate those frequently, et cetera. We're being very careful about holding out the test set, but still, there's often some fear of overfitting to those and just benchmarking.

is what people call this. But that's one aspect. I don't think those fears are very well-founded. But the second aspect, and that's the one that really fills me with confidence, is the amount of time people spend using the model to make themselves more productive internally is increasing over time. Every new generation of models, it's pretty clear the model can do new things and help us in our research and our day-to-day engineering work much more so than the previous generation of models.

So that aspect should give us confidence as well that the models are becoming more capable and actually are doing very useful things as well.

I think it's easy to say we're on track in hindsight. I think if I'm being honest with myself, I think we're ahead of where I thought we could go. Starting work on LLMs in 2019 or 2020, it's kind of hard to believe the scale of everything we're doing, but also just what the models are capable of doing today. If you kind of looked at scaling laws back then, they were definitely pointing in that direction. And some people really believed those deeply.

I'm not sure if I would have bet a lot on that actually materializing and being where we are today. One interesting question that follows from this is, where does that take us if we assume the same kind of progress we've seen in the last five years? I think it's going to be very cool, what's going to happen in the next few years as well.

I think, yeah, that's part of it. So on the science side, I think DeepMind historically has done a lot of work, and for sure there's a lot of work in that direction as well. I think we will be able to make some large scientific discoveries in the next few years. That's one side. I think on the other side is in my day-to-day work as well, both research and engineering.

I think it's not so much about automation, but more about making us go faster and spending more of our time in the research part at a slightly higher level. A lot of the day-to-day work in research on language models is dealing with quite complex and large systems at the infrastructure level. So actually, quite a bit of time is dedicated to running experiments, babysitting experiments, analyzing a lot of data, collecting results. And then the interesting part is forming hypotheses and designing new experiments.

And so the last two parts, I think, are something where we'll be very much involved. The first part, I think, especially in the next year, with more agentic workflows being enabled more and more, should be able to really accelerate our work there.

For the first question, there's definitely similarities between what the different labs work on. I think the base technologies are kind of similar. I'd be surprised if we weren't all training transformer-like models, for example, in terms of the architecture side. But then there's definitely specialization, I think, happening on top of that, and different, maybe, tree branches in the tree of research that are being explored and exploited by the different companies. I think historically, for example, DeepMind has—and still, I think, on the vision and multimodal side, we've been actually really, really strong.

And that continues to be the case today and shows in both how people use the model, but also in the benchmarks, of course. And then other things like reasoning, et cetera—OpenAI came up with the first model, but we also had a strand of research on that. So there's similarities, but it's not exactly the same, I would say. For the second question, I don't know if I have a good answer. One thing that's clear is, to make progress on a model like Gemini today, you do need a very large team and a lot of resources.

Now, that doesn't necessarily mean that what we're doing today is optimal in any form, and some disruptive research could definitely come along and allow a smaller team to actually take over in some form. This is one of the reasons why I actually enjoy being at Google so much as well: Google has this history of doing more explorative research and has a really high breadth of that research, and that continues to be the case, mostly in parallel to Gemini. But we're definitely able to also utilize that and bring some of those advances into Gemini.

I believe so. There's groups doing research on the model architecture side for sure within Google and within DeepMind. Whether that research will pan out, it's hard to say, right? It is research. So very few research ideas work out.

It definitely helps. I think it's an important part. Research versus engineering is also interesting. I think over time, that boundary has blurred quite a lot because we're working on these very large systems now. Research really looks like engineering and vice versa. And I think that's a mindset that has really evolved over the last few years at DeepMind, especially where maybe there was a bit more of the traditional research mindset before. And now with Gemini, it's really more about research engineering.

The infrastructure part is also very important. We are building these super complex systems. So having infrastructure that's reliable, that works, that's scalable, is key in terms of not slowing the research engineering down.

I am one of the Gemini pre-training leads. So what this entails, it's a mix of different things. So part of my job is actual research, so trying to make the models better. But these days, it's less running experiments myself, but helping design experiments and then reviewing results with people on the team. So that's the first part. The second part, which is quite fun, is more of the coordination and integration. So it's a fairly large team at this point. It's a bit hard to quantify exactly, but maybe 150, 200 people that work day-to-day on the pre-training side between data, model, infrastructure, evals.

And so coordinating the work of all of these people into something that we can build together is actually quite complicated and takes quite a bit of time, especially time to do well. To me, this is super important because actually being able to get progress out of everyone is really what makes us make the most progress, rather than enabling maybe one or two or a small group of 10 people to run ahead of everyone else. That might work for a short period of time, but over longer periods of time, what's really been successful for us is being able to integrate the work from many, many people.

I grew up a bit all over the place in Europe. I moved around quite a bit. So I was actually born in the Netherlands, and I moved when I was seven to Switzerland. So my dad is from Switzerland and my mom is from Germany. So I did most of my school and the beginning of my high school in Switzerland, mostly in French and also in German in parts. And then at age 15, I think, I moved to Italy, where I finished high school until I was around 19.

And at that point, I was going to go to ETH Zurich to do my studies, but I think just by random events, one morning I just looked up the top universities in some kind of ranking, and I saw Cambridge was at the top. So I thought, I'll just apply. Why not? And, yeah, a few months later I got the acceptance letter. So I decided to move to Cambridge, where I did my undergrad and master's in the Computer Lab.

My dad has a technical background. So I remember, when I was 10 or 11, starting to program a bit with him and learning. And I kind of always liked that. And then I always had, like, easiness in math and science at school. I remember never having to really study for math exams, but always doing quite well. That definitely changed at university. But that was my high school experience.

Yeah, so that's, again, a bit of a lucky moment, I would say. One of the lecturers we had in my master's was someone who was also a researcher at DeepMind. And I just remember at the end of the last lecture, I was packing my stuff, and I was like, oh, I'll just ask him for a referral. What's the risk, right? He might just say no, but whatever. And so I actually took the courage and I went up to him and asked if he would give me a referral.

He was like, sure, send me your CV and I'll see what I can do. And that's kind of how I got my interview at DeepMind. This was in 2018. And so I joined DeepMind at the time, just DeepMind, not Google DeepMind, as a research engineer after university.

Yeah, so at the beginning, having joined DeepMind and DeepMind being known for RL, the first project I managed to work on, or decided to work on, was something on the RL side. So specifically, we were training some unsupervised network to learn keypoints on Atari environments and try to get the agent to play Atari. So I did this for about six months, maybe. It wasn't enough, or in the sense I didn't like the synthetic aspect of this. I always wanted to work more on real-world data and have more of a real-world effect.

I think in general, I like to build things and build things that work. I don't really like the academic pure research part. And so that kind of drove me to start working on representation, so creating these, or training these neural networks that have good representations to do different tasks. And one funny anecdote here is something I tell a lot of the people on my team, but the first effort I joined on this was called Representation Learning from Real-World Data. And at the time, we had to add this "from real-world data" to the name of the project because people would assume otherwise it would be synthetic environments or synthetic data.

And that definitely has shifted completely since then. So yeah, that was kind of my first project on that side, and specifically LLMs and transformers. We were looking at architectures like Transformers and models like BERT and XLNet that were learning these representations and trying to improve those representations and do research on that side.

Yeah. So after that, we started working on scaling up LLMs and LLMs in general. So we started this work first on Gopher, which is, I think, the first DeepMind LLM paper that was published. So already at that point, it was a team of maybe 10, 12 people. So already at that point, it was pretty clear you couldn't just do that research on your own. And this is really where I started doing pre-training and pre-training at scale, and developed my research taste, but also what I enjoy about this.

So we trained the first dense Transformer model. I think it was 280 billion parameters, I think 300 billion tokens at that time, and trained that. We definitely would not do things like we were doing them back in the day, but it was great and a very fun learning experience. After that, there were kind of two projects that emerged. The first one was Chinchilla and the second one RETRO. So in Chinchilla, we were reexamining how you should scale the model size and how you should scale the data, especially from a training compute-optimal perspective.

So the question is, you have a fixed amount of training compute: how do you train the best possible model? Should you increase your model size or should you increase your data size? And there was some previous work in this domain from OpenAI specifically that we reexamined, and we actually found that you want to scale the data side much more quickly than what was thought before, rather than scaling the model side. Funnily enough, this is still really relevant in our day-to-day work today, especially because it has a lot of implications on the serving cost and how expensive it is to use the models once they're trained.

The other line of work was more on RETRO, and this is more on the architectural innovation side of things. So here we were looking at how you can improve models by giving them the ability to retrieve from a large corpus of text. So rather than having the model learn and store all the knowledge in its parameters, you give the model the ability to look up specific things during training, but also during inference.

Yeah, it's very important these days. And it's quite hard to quantify. But the first thing that matters, maybe, is that your research is not standalone. This is what I was mentioning before, but your research has to play well with everyone else's research and has to integrate, right? So let's say I have some improvement on the model, but it makes the model 5% harder to use for everyone else. This is probably not a good trade-off, right? Because you're going to slow down everyone else and their research, which would then cumulatively slow down the overall research progress.

That's the first thing. The second thing is being allergic to complexity, but complexity is quite subjective in terms of what people are familiar with. But still, we have a certain budget of complexity we can use and a certain amount of almost research risk we can accumulate before things go bad. And so being aware of that and managing that is very important. So oftentimes we don't necessarily want to use the best-performing version of a research idea, but we'd rather trade off some of the performance for a slightly lower-complexity version because we think that will allow us to make more and more progress in the future.

So these are kind of the main two things, I think, around research taste.

Yeah, definitely. That's also an important part. I think some people have that much more than others, and a lot of experience really helps. But for sure, we are bottlenecked on the research side by compute. If we had a lot more compute, I think we'd make a lot more progress a lot quicker. And so you have to guess, to some extent, which part of the research tree you want to explore, and then within that, what are the right experiments.

But then also knowing most research ideas fail, right? And so you need to figure out at what point have I done enough in this direction to know to move on to something else, or should I keep pushing? And then the other interesting thing is, especially in deep learning, a negative result doesn't mean something doesn't work. It means you haven't made it work yet, often. And so being aware of that as well is quite tricky.

This is part of what I spend a lot of time thinking about as well. There's always critical-path things to be done, or like this part of the model needs improving, or we know this part of the model is suboptimal. So we invest quite a lot in just fixing those immediate things. There's a few reasons for that. The first one is we know this will make the model better. So it's a fairly safe bet, but also we know that things that don't look quite good or quite perfect often tend to have issues later, either when you scale up or when the model just becomes more and more powerful.

And so actually being very diligent about tackling those and fixing those is really important. So that's kind of the first part. The second part is slightly more exploratory research. So ideas that could land in the next version of Gemini or the version after that, that have maybe a bit bigger effect on the model performance, but aren't quite validated. How we balance these is, I don't think I have a very clear answer. It's also a bit periodical. So when we're doing a scale-up, for example, there's often slightly more exploratory research because there's nothing right now that needs to be fixed in parallel.

But just before we are ready to scale up a new architecture or a new model, it's very much like, let's de-risk the last pieces. It's very execution-focused.

This is why I like Google as well. There's actually very little of that, I think, because all of the leadership has a research background. They're very much aware that, yes, to some extent you can force and accelerate specific benchmarks or certain goals, but in the end, the progress and making the research work is really what matters. So I personally, at least on a day-to-day basis, never really feel that pressure.

At a super high level, we have a pre-training team and a post-training team. On the pre-training side, we have people working on the model, on the data, the infrastructure, evals as well. Very important. I think people often underestimate the importance of evals research, and it's actually quite hard to do this well. And then, yes, there's a post-training team, and of course there's a large team working on infrastructure.

At a high level, I don't think the architecture has changed that much compared to the previous one. It's more of what I was saying before, where a few different things come together to give a large improvement. At a high level, though, it's a mixture-of-experts architecture, Transformer-based. So from that perspective, if you squint enough, you will recognize a lot of the original Transformer paper pieces in that.

At a high level, the Transformer kind of has two blocks. So there's an attention block, which is responsible for mixing the information across time, so across different tokens. And then there's the feed-forward block, which is more about giving the memory, but also the compute power for the model to make these inferences. And those operate on a single token at a time, so they operate in parallel. So in the original Transformer architecture, this is just a single hidden-layer neural network.

So it's a dense computation where the input gets linearly transformed into a hidden dimension, you apply some activation function, and that one gets linearly transformed again into the output of the dense block. So that's the original paper. And then there's a lot of work before Transformers as well on mixture of experts. And here the idea is you kind of decouple the amount of compute you use with how large the parameter count is. And so you dynamically route, effectively, to which expert you want the computational power to be used on, rather than having that coupled.

Yeah, what this means is that there's no specific model trained to handle images and a different model trained to handle audio, a different model trained to handle text. It's the same model, the same neural network that processes all these different modalities together.

Yeah, this is a really good question. There's kind of two costs to this. I would say that the benefits largely outweigh the costs here, and this is why we train these models. But the first cost is maybe less obvious to people, but it's this complexity cost and this research bit I was talking about, because you're doing a lot more things, and especially different modalities interact in some ways. This can interact with different parts of the research and has a complexity cost. So we have to spend time thinking about these things.

The second cost is, yes, images are often larger in terms of input size than pure text. And so the actual computational cost, if you do it naively, is higher. But of course, then there's interesting research to be done on how you make these things efficient.

Yeah, the discussions there, to me, were always slightly strange because my experience didn't match those. I think what we've seen is scale is a very important aspect in pre-training specifically, and how we make models better. I think what's been the case, though, is that people overvalued that aspect. So it is a very important aspect, but it's not the only aspect. So scale will help to make your model better. And what's nice about scale, it does so fairly predictably.

And that's kind of what the scaling laws tell us, is as you scale the model, how much better will the model actually be? But this is only one part. The other parts are architecture and data innovation. These also play a really important part in the performance of pre-training, and probably even more so than pure scale these days. But scaling is still an important factor as well.

I think the way to put this is these all compound. So scale is one axis, but the model and data also will make the actual performance better. And yes, sometimes the innovation part outweighs the benefits of scaling more. And sometimes just raw scaling is the right answer to make the model better. So that's on the pre-training side. And yes, on the RL and RL scaling side, I think we're seeing a lot of the same things we're seeing in pre-training, or we saw in pre-training.

What's interesting here is because we have the experience of pre-training, a lot of the lessons apply, and we can reapply some of that knowledge to RL scaling as well.

Yeah, it's a mix of different things. So the data is multimodal from the ground up, and there's many different sources that go into this.

Yeah, so synthetic data is interesting. You have to be very careful in how you use it because it's quite easy to use it in the wrong way. And what's often the case as well with synthetic data is you use a strong model to generate the synthetic data, and then you run smaller-scale ablations to validate the effect of the synthetic data. But one of the really interesting questions is: can you actually generate synthetic data to make a model that you want to train in the future, which will actually be better than the model that generated the synthetic data in the first place?

Can you actually make that one better as well? And so we spend a lot of time thinking about this and doing research in this direction. The other part of your question, “Are we running out of data?” I don't think so. So there's more. We are definitely working on that as well. But more than that, I think what might be happening instead is kind of a shift in paradigm where before we were kind of scaling in the data-unlimited regime, where data would scale as much as you would like.

And we're kind of shifting more to a data-limited regime, which actually changes a lot of the research and how we think about problems. But one good analogy of this is before LLMs, a lot of people were working on ImageNet and other benchmarks, and there was a very data-limited regime as well. So a lot of techniques from that time start to become interesting as well.

Yeah, unfortunately, I can't comment on the specifics.

I believe so. And this also falls into the previous question around synthetic data you were asking, and our approach to that is similar.

To use the famous analogy, a model can learn like a child. Just to maybe clarify what I said earlier, in the data-limited regime, I didn't necessarily mean with less data, but rather with a finite amount of data. So the paradigm shift is more from, like, we have infinite data to we have a finite amount of data. The second point is, in some sense, model architecture research is exactly what you mentioned. So when you make an improvement on the model architecture side, what it typically means is you get better results, a better result if you use the same amount of data to train the model.

But equivalently, you could get the same result as the previous model by training on less data. So that's kind of the first aspect of that. But it is true, in terms of the volume of data needed today, we're still orders of magnitude higher than what a human has available to. Of course, there's the whole evolution process as well, which I find these high-level discussions quite hard to understand or follow because you have to make so many assumptions to convert that amount of data into what is today's pre-training data.

But at least at first order, it does seem like we're using a lot more data than humans do.

Gemini 3, I think, had a really good leap in the long-context capabilities of the model. And I think that's really enabling the ability of models and agents today to do this work where you have maybe a codebase and you do a lot of work on it. So your context length really grows. I think there's going to be a lot more innovation on that side in the next year or so to make long context more efficient, but also just to extend the context length of models themselves.

So that's on the capabilities front, I think, is something where pre-training specifically has a lot to offer, and it is very interesting. Relatedly, I think for us, at least on the attention side, we've made some really interesting discoveries recently that I think will shape a lot of the research we do in the next few months. And I'm personally very excited about that. Again, I think I want to emphasize the point that I made towards the beginning, but the way things work is it's really a culmination of many different things.

So there's a lot of small, medium-sized things that we can already see coming up where I think, wait, we fixed this issue, we fixed this bug. This is interesting research that shows promising things. And all of these things coupled, I think, will drive a lot of the progress. It's interesting thinking about RETRO that we talked about a bit earlier.

So first of all, I think RETRO was really about retrieving information rather than storing it, not necessarily about making models smaller. So it's about how we can use the model to do more reasoning already in a pre-training sense of reasoning, rather than just store the knowledge. So this is still very much the aspect today. The interesting part is the iteration cycle, maybe, of pre-training used to be a lot slower than that of post-training until fairly recently.

And so making these large changes on the pre-training side is quite costly in terms of risk and how long it takes. And then you have approaches like RAG or search, which you can do during post-training and iterate much more quickly on, which give very strong performance as well. I think deep down, I do believe that the long-term answer is to learn this in a differentiable end-to-end way, which means probably during pre-training, or whatever that looks like in the future, learn to retrieve as part of the training and learn how to do search as part of the larger training.

And I think that that's kind of where RL scaling maybe starts that process, but I think there's a lot more to do on the architecture side. But this is something that we'll see in the next few years and not immediately, I would say. The one thing I want to highlight is people often talk about model architecture, and that's definitely one part of what makes pre-training better, but there's other parts as well, like infra and data and evals specifically, that don't always get the same mention.

Evals specifically are extremely hard, and it's even harder in pre-training, I would say, because it kind of has these two gaps you need to close. So on the one side, the models we train regularly are much smaller and less powerful than when we scale up. So that means the eval has to be predictive of what the performance will be, has to still work for the large model, and point in the right direction. So it has to be a good proxy on that side.

And then there's a second gap as well, which is when we evaluate pre-training models, there's a post-training gap as well. So the way the models get used is they don't just get used after pre-training. There's more training happening after. And so the evals we use in pre-training, or pre-trained models, have to be good proxies of what happens after as well. And so making progress on evals is really important and quite hard, and has also driven a lot of the progress we have in terms of being able to measure what an actual improvement is on the model or on the data side.

Yes, to a large extent, and more and more so, because what we found is that external benchmarks, you can use them for a little while, but very quickly they become contaminated. So they start to be replicated on different forums or different parts of the web. And then if we end up training on those, it's really hard, basically, to detect leaked evals.

It's a majority of post-training, I would say, but there's definitely some parts of it which are relevant to pre-training. I can't go into too many details here, but some parts are relevant to pre-training, and then we do think about that as well.

This is an interesting question, and I don't think I have a definitive answer, but you don't want the model to do these terrible things. So at a fundamental level, you do need the model to know about those things. So you have to train a bit, at least on those, so that it knows what those things are and knows to stay away from those. Otherwise, when a user would mention something terrible, the model wouldn't even know what it's talking about, and it might not be able to say, "This is something terrible."

I'm not allowed to—I can't comment too much on specific things.

Okay, fine.

Yeah. I think this has been covered quite a bit in some of your previous podcasts as well. It's about generating thoughts. And so, rather than just doing compute in the depth or on the model side, you also do compute and allow the model to think more on the sequence-length side of things. So the model actually starts to form hypotheses, test hypotheses, invoke some tools to validate the hypothesis, do search calls, et cetera, and then, at the end, be able to review the thought process to provide a definitive answer to the user.

That's fair, yeah.

Yeah, this is, I guess, what I was mentioning before around my own work especially. I think that's interesting. A lot of the work we do on a day-to-day basis is more execution-based, babysitting experiments, et cetera. And I think this is where I at least see the most impact from those. Bringing it back to the topics of pre-training, I think that the perception and vision side is very important for this because now you're asking models to interact with computer screens.

So being able to do screen understanding really, really well is critical. And so that's an important part on the pre-training side, at least.

Yeah, this is interesting. I think you can probably ask five different researchers and you'll get five different answers. There's also this notion of a large-model feel. GPT-5 historically had some of this, presumably, where larger models maybe feel differently. I wouldn't put it in these terms specifically, but I think vibes comes down to this, and actually pre-training probably plays a larger role today in some of that, and how the model feels in general, than post-training. I think this is, yeah, this is in general for vibe coding specifically.

I think that's maybe more of an RL scaling and post-training thing, where you can actually get quite a lot of data and train the model to do that really well.

Yeah, I guess continual learning is about updating the model with new knowledge as new knowledge is discovered, right? Let's say a new scientific breakthrough is made tomorrow. The base model we trained yesterday wouldn't actually know about it in its pre-training. First, I think a lot of progress has been made on this front in the last few years. I think this is mostly around pre-training, around search: use search tools and then make search calls. Then they would have access to that new information.

In some sense, this is also what RETRO, that we talked about, was doing by retrieving data and then trying to externalize the knowledge corpus with the reasoning part. So that's the first part. I think the second part, on the pre-training side specifically, is what I was mentioning about long context as well. And one way of doing this is if you can keep expanding the context window, the model keeps getting more and more information in that context. And so you kind of have this continual learning aspect as part of that.

But then, of course, there's more of a paradigm shift. Maybe this is what people discuss: can you change the training algorithm such that you can continuously train them on a stream of data coming from the world, basically?

Yeah, there's a lot of, again, there's a lot of small things right now that accumulate. So that's kind of the first thought that comes to my mind. And that historically has really driven progress. So I wouldn't just bet against that continuing to drive progress. The things I mentioned before around the long-context architecture and long-context research is one aspect, I think, on the attention mechanism as well on the pre-training side. And then this paradigm shift from infinite data to the limited-data or finite-data regime is something as well, I think, where a lot of things will change and there's a lot of interesting research.

That's kind of on the pre-training alone side. The other side, which is quite interesting today, is these models—the amount of people using these models is growing quite rapidly. And so more and more, what we have to think about on the pre-training side as well is how expensive is the model to use, to serve, and have really deployed at a large scale? And what things on the pre-training side specifically can we do to make this model have better quality and maybe be cheaper to serve and consume fewer resources during inference?

One thing that's becoming increasingly important is being able to do research, but being aware of the system side of things. So we are building these fairly complicated systems now. So being able to understand how the stack works all the way down from TPUs to research is kind of a superpower, because then you're able to kind of find these gaps in between different layers that other people weren't necessarily able to see, but also to reason through the implication of your research idea all the way down to the TPU stack.

And people that can do that well, I think, have a lot of impact in general. So in terms of specialization, it's really, really thinking about this research, engineering, and systems aspect of model research, and not just the pure model architecture research. That's one. I think personally, I still have a lot of interest in this retrieval research as well that we started with RETRO. And I think it wasn't quite ripe until now, but things are changing. And I just think it's not unreasonable to think in the next few years, something like that might actually become viable for a leading model. And why was it not ripe, and why may that change?

I think that's around the complexity side of things I was mentioning, and also the fact that, for the capabilities it brings, you can iterate much more quickly in post-training. So what I was saying with search and post-training data, you can give very similar capabilities to the model in a much simpler way.

I think it's got a lot better. I think maybe two years ago, what I was seeing is people were still trying to very much create specialized models to solve tasks that were maybe within half a year or a year of reach of generalist models. And I think people have caught up to that much more and now kind of believe that for generalist tasks, or tasks which don't require extremely specialized models, trying to use a generalist model—and maybe not the current version, but the next version—might be able to do that.

And then, so what that means is research in terms of how you use models and the harness, et cetera, is becoming increasingly important. And also how you make models and these harnesses more robust to making errors and recover from such errors.

Yeah, I think so. Maybe look at what models were able to do a year or a year and a half ago, and then look at what models are able to do today and try to extrapolate that. I think the models—the areas where the models are improving—I think will continue to improve. And then there's maybe some areas where there's not been that much progress, and that might be more interesting areas to do research. I don't really have a specific example in mind right now, but that would be the general advice.

What I like very much about my day-to-day is working with many people and being able to learn from a lot of researchers. And that's what drives me to a large extent. Every day I come to work and I talk to really, really brilliant people, and they teach me things that I didn't know before. And so I really like that part of my job. As I was saying multiple times at this point, there are just so many different things that will compound and different things where there's headroom to improve.

I'm really, really curious because right now I don't really see an end in sight for that kind of line of work to continue giving us progress. So actually being able to see this through and see how far this can take us is really interesting. But at least for the next year or so, I don't see this slowing down in any way.

Thank you, Matt.