Ex‑DeepMind Researcher Misha Laskin on Enterprise Super‑Intelligence | Reflection AI

Yeah, thanks, Matt. Thanks so much for having me.

We're a research and product company in AI that was founded by myself and my co-founder, Ioannis. We are both longtime researchers at DeepMind, and generally the makeup of the team is a lot of folks who did large language model training and reinforcement learning, building reinforcement learning systems within companies like DeepMind, OpenAI, and Anthropic. We came together to build this company that co-designs research and product to build superintelligence. I think that superintelligence can sound like a very abstract thing, and I think what makes doing this at a startup or a smaller place special is that you can be very concrete in what that means to you.

Because I think that generally, when people think about superintelligence, they might think of something that becomes really good at math Olympiads or coding Olympiads, competitive coding, things like this. But these things to us don't really—I mean, they're impressive, but it's unclear how they're useful. Like, what is the superintelligence doing that is useful for kind of an end user? But if you co-design a product and the research alongside it, you can be much more opinionated about where it is that you want to go.

And after talking to a lot of organizations and enterprise customers after we started the company, we realized that probably the form factor of an organizational superintelligence, like the thing that's really going to help organizations get a lot of stuff done, is probably going to be something like an oracle. Like an oracle that understands the entire organization extremely deeply, that can answer questions at the level of, let's say, the senior-most person on that team, right? It kind of has that, like, a principal-level engineer or the senior-most sales leader who has full context on the company.

And it'll probably be superintelligent also in the sense that it's doing all those things at once, right? As opposed to having these distinct different functions. And so that is what we think an organizational superintelligence is going to look like, that it's going to be a system that really deeply comprehends the organization.

I would actually say that superintelligence in these contexts of the large lab context is actually just being used synonymously with what AGI used to be used for. There's a scene in Indiana Jones and the Raiders of the Lost Ark, the first scene where he goes and tries to steal a sculpture, and he kind of takes a sculpture and replaces it with a sandbag and hopes that nothing changes. And I kind of think one of those—

Yeah, the analogy stops here unless we really figure out safety. But I think that what happened is that, by many accounts, many people would argue that AGI hasn't been achieved, but now some people might argue that AGI has. And so it's unclear that this is a binary event that hasn't been achieved yet, even though that was the case before. And so I think the goalposts were just moved. It's like, oh, what I meant by AGI was actually superintelligence.

So I think that is to say that it's as vaguely defined as AGI was before. And I don't think there was any agreement on what we actually meant by AGI.

Okay. All right.

Yeah, I think it's more the person who—on every team, there's a go-to seasoned, experienced person who's very much in the weeds and understands everything. Getting systems first that are like that, and then they can hold a lot more context in their heads and sort of be even better than those critical members of the team. I think that's kind of like an oracle for an organization. That's what a superintelligence looks like. Because once you deeply understand the problems that you need to solve in any discipline in enterprise, acting upon them to actually go and solve it is the easy part.

And so we're really focused on coding to start for a number of reasons. We kind of think you solve this problem, it sort of solves the more general problem.

One reason is that the way today we think of coding is just for software engineers. But when you build a coding model, you've just trained a language model that can interact with a piece of software through code. And so the way these language models are going to interact with any piece of software, not just software engineering software like Salesforce and other CRMs and creative tools and so forth, the majority of those interactions are going to be through function calls, APIs, so through code.

So today we think of coding models as something that is just for software engineers, but that's not true. If you build that system, it can actually interface with any piece of software. It's sort of what you've built is kind of the hands and legs of a digital AI, right? In the same way that a humanoid robot can do anything with the same hands and legs that humans do.

Yes, you can kind of think of the model as the brain, and then what people call scaffolding, agent scaffolding, that's sort of the affordances, right? The things that you can actually do. All these affordances for software, at least for digital intelligence, are going to be primarily through code. That is, the other option is teaching a model how to drag a mouse around, and it's called computer control. Some of that will happen, but I just don't think that that's going to be the majority way in which a language model interacts with software.

So if you solve coding, you've just solved how a language model should interact with software?

No, I think that's very much correct. It's more that code is kind of intuitive to LLMs because they were trained on the internet initially. And so they kind of have almost a muscle memory for code. It's kind of like humans have evolved to have a spatial, geospatial muscle memory. And so we learn very quickly how to use our hands and so forth as toddlers. Code is actually very unintuitive to us, right? That's why the GUI was invented, because the first way to interface with computers was through code.

Very unintuitive for humans. So you invent the GUI. For language models, it's the opposite. They've never seen any GUI data really on the internet. There's no mouse movement data out there. So the thing that is native to them is the exact opposite of what's native to us. So if you build things that are really good at coding, you're sort of just amplifying the existing base knowledge of the LLM. It's just intuitive to a language model. So that's why coding.

But then within coding, you can kind of think about what it would take to create a superintelligent coding agent. You basically need a system that can generate code really well, and you need a system that comprehends large codebases and all the knowledge around them, like everything that's in all the docs and the stuff that's kind of tribal knowledge that's in engineers' heads. And I'd say every coding tool today is focused on the code generation piece in an autonomous, semi-autonomous, in-your-IDE, in-your-terminal way.

But it's all around code generation. And as a result, because companies today, I don't think, are really focused in earnest on the understanding piece, what we're building towards, I'd kind of call as almost today, maybe we have AI engineers that are at the level of an L4 engineer, maybe next year L5, L6, going to L9. I think the path there is clear. And what we're going to get to if we don't solve the comprehension piece is basically L9 engineers with amnesia.

So an L9 engineer, if they came into your company but had amnesia, they wouldn't be really useful.

Right. It's kind of, if you tell me exactly what to do, I'll go do it really well. But otherwise, I know nothing about your codebase, I know nothing about your organization, and I'm not going to gather any of that either. The thing to solve now, I think, is building what we kind of think of as the context engine. Like, what's the thing that's kind of the brain that your coding agents can query in order to have all the context that they need to go do meaningful work?

Not just junior engineering work, but the stuff that engineers spend really most of their time on, like critical infrastructure bugs, dealing with legacy code, things that if you actually look over the shoulder of an engineer at any large organization, or even a startup with a really sizable codebase, and you look at what they do, you'll find that a minority of their time is actually spent coding. Like 70% of their time, they're digging basically through information, trying to understand stuff, asking other engineers questions.

And you kind of want to build a superintelligence that has that same DNA. What it's spending most of its time on is collecting information, and then obviously, once it has it, it knows how to act on it.

Maybe it's almost like if RAG worked. So the purpose of RAG is to get all the information that your coding agent needs in order to do some work. But it's a very primitive form of doing this, and by and large fails. It's kind of—

So there are a few things about it, but the first thing is that it's kind of what we call sparse, because the stuff that it grabs from a large codebase has a lot of false negatives and false positives. And it typically only does it once, right? It'll grab it, and then that's all you have. And most likely, for any meaningful query, it will not have given you the information that you need to actually go do the task. So RAG agents actually are pretty weak.

What's happening now is that there's kind of a new kind of retrieval that I think people are calling agentic search, which is more what Claude Code does. And it's kind of an agent that uses the same kind of command line that a human does and goes and kind of looks for files and uses the same commands that an engineer does to go look for files and search for keywords and grab stuff. And it does this agentically. So as opposed to it just being a one-step process where it just pulls a bunch of embeddings in, it will use a file search tool, and then it'll look at the stuff that it did and think about it, and then use another tool and so forth.

It does store the stuff in a context. But the way I would think about it is that imagine you are dropped into a large, dark jungle, and all you have is, like, a tiny flashlight. That's basically what agentic search is today, where you're going and you're kind of exploring, and you have this tiny flashlight, and you have to then remember everything in your head. And obviously, that's a first form of comprehension, but it's also a pretty weak form of comprehension. It doesn't scale to large jungles.

Like, if you have a little tiny jungle in your backyard, then you might be able to navigate it.

A bonsai jungle. Yeah. So that's kind of how I liken the existing state-of-the-art agentic search, whereas the kinds of systems that you need to build are ones that basically expand the kind of lens, like the aperture of your beam, so you're seeing more, are smarter about where they go and look for stuff, and how they remember it. These are basically fundamental research problems, like memory. Long-context reasoning is what people call it, but I'd say it's just expanding the aperture of your beam, figuring out how to actually store it, and also what sources of information should you be pulling on other than just your code.

Because information for organizations lives in their chats and their project management tools, and a lot of it is in their team's just brains, right? So oftentimes, when that senior engineer leaves who understands some legacy piece of code, all of a sudden that becomes known as a haunted graveyard because no one else understands it. And so how do you build products that capture that knowledge that's in people's heads? That's why I kind of think that you can't—it's so hard to think about superintelligence in the abstract because half of it is a product problem.

The comprehension problem is sort of, I would say, deep. And what people used to say, like, when a problem was AGI-complete, it just meant that the problem has so much depth to it that if you solve it, you get AGI. I would say this is a superintelligence-complete problem. Like, if you really solve this oracle for organizations, just for coding, you've basically built all the capabilities you need to have a superintelligence.

Yeah, I think that the notion of, in the sense in which coding is ASI-complete, is that you can use an intelligent coding system to build another intelligent coding system. If people like to do research in the abstract, that sounds awesome, because it's kind of like, I don't have to think about the actual problem that's being solved. If I just build an intelligence that builds an intelligence, it will figure it out somehow. I don't think that's how it works. I think coding intelligence that builds better coding intelligence will make algorithms more efficient, basically.

And maybe it's so intelligent that it can even start making all the product decisions for you in terms of what questions you should be asking users and what features you should be building. And to me, it actually seems like a much larger, more practical problem. There's almost no—I think research without co-designing product with it is sort of a meaningless pursuit now. That was only meaningful when the ingredients for how to build artificial general intelligence, or ASI, were not known. I think now they're known.

Yeah, I think we have everything, even though the capabilities will continue being tightened. But the sort of series of breakthroughs that need to happen, many of them have happened. There are still some, but I would say that the ones that come to mind, and there, I'd say, are four of them. The first ones were just making deep neural networks work. So that was just the ImageNet moment in 2012, when you can have a deep neural network that's classifying images at human and then superhuman level.

I think that was the first breakthrough. Second breakthrough was reinforcement learning, even though I think it was not clear maybe then, but now it's clear again. So reinforcement learning basically told us, how do we take a system, and if we have a reward, make it superintelligent? We actually—superintelligence has been built a few times now. AlphaGo was superintelligent. AlphaStar was—if you put more compute into it, it would have gotten superintelligent. So we've known how to build narrow superintelligence for a while.

So deep neural networks, reinforcement learning.

Then scaling up transformers, the GPT series of work.

Kind of '18, but then really, I think, started shining in the early 2020s with GPT-3. And that was both an architectural innovation and a data innovation. It was an architecture that could consume data on the internet. So it's kind of both hand in hand. Transformers are not just architecture; it was also data. And then I would say the last ones are RLHF, which was, okay, we know how to align models in the basic way. We know how to do basic reinforcement learning on top of language models.

And then reinforcement learning coming back again with the reasoning models. So if I was to summarize, I would say neural networks, deep neural networks, transformers and internet-scale data, and reinforcement learning in kind of its various incarnations. These are the ingredients that are required to build a superintelligence.

Was that a temporary solution, or are those parallel? I think they're parallel capabilities because they do different things. RLHF was more—the design there was more to align a language model, like a pre-trained language model. If you play with one of these base models before they're aligned, they're really useless. They feel like stochastic parrots. They don't follow instructions. They're extremely kind of high entropy. And the fact that you could just align them, tweak them to align them with something that is human-consumable, I think was a pretty big breakthrough.

And that was RLHF. Whereas RL with reasoning is more RL to drive intelligence capabilities, which is make it really good at coding, make it really good at math, make it really good at whatever target domain that you have rewards for. And you use these in tandem, where RL in the reasoning phase sort of really expands on a capability, and then RLHF aligns it to be kind of human-consumable. But they're the same thing. In fact, I would say it's just the same thing.

The machinery is the same.

Yeah, we're launching our first product, kind of the first milestone on the path to superintelligence. It's called Asimov, like the science fiction writer who had some thoughts on the subject. And Asimov is the best-in-class code research agent built for organizations. So it's different than a coding agent, which is a thing that will go and write code for you, but which today we feel is pretty context-poor. So if the task is well-scoped, coding agents today will do it pretty well for you.

If you are spending most of your time trying to understand some hairy problem in your engineering, why some infrastructure bug is happening, or really doing what engineers spend most of their time doing, which is this kind of work, there's no tool today that really unblocks them in that way. And so I'd say 70% of doing software engineering is actually a code research job rather than a code monkey job. And we're building a product that addresses that problem. So the majority of time that engineers spend unpacking problems and trying to understand why something's happening is oftentimes bottlenecked because they have to ask someone a question, and it takes a few hours for that person to get back because they're busy.

We've built a product that helps overcome these kinds of problems.

It's very similar in concept to deep research for code. A thing that will go explore your codebase and other sources of knowledge. It might take a bit longer than the traditional snappy ask-kind-of products, but it'll come back with much better answers. Things that make it different are that, one, engineering knowledge does not just live in a codebase. It lives in all sorts of other surface areas and other software products, like project management tools, chats, documentation, things like this.

And Asimov pulls from those things. So it's not just your codebase; it's aggregating all these sorts of information. It has this new concept that I don't think has been introduced to date that we call team-wide memories. So today, products have individual memories, which remember the personal preferences of the developer, but they don't have a team-wide organizational understanding. Suppose you have a senior software engineer that understands some microservice A there. You are not that engineer, and now you need to interact with their microservice, and you don't have the context they do.

So now with Asimov, it kind of organically captures that information as it happens in chats, but also engineers can just teach it directly.

Exactly. It's kind of building a permanent organizational system of record for your engineering knowledge to start. A side note on that is something that's been interesting is that some of the most excited users have been these senior staff-level engineers who are fielding people's questions all the time. And so what we've seen is that we'll go to an organization, and the first three weeks there'll be four senior staff-level engineers who are just populating its knowledge. It's kind of the first time I've actually ever seen engineers excited about documentation, effectively.

That's kind of a unique thing. And then the final thing is really around agent design that enables you to, what I was calling earlier, increase the aperture of your beam so that it's looking at much larger codebases than agents were able to before. This is still, in that sense, a work in progress, in that long-context reasoning is just a big fundamental problem. And I think there's a lot of work to be done there. But I think this new agent design is a step in that direction.

This is not unique to us. I think when I look at how state-of-the-art agents are being designed, they're being designed very much as these multi-agent systems. Agent design is effectively a big reasoning agent, and I think that's pretty standard, but it dispatches small long-context reasoning agents to go search for different relevant chunks of information in the code. And so I think that it's this kind of decoupling of a big reasoning agent, maybe with a smaller context, with a bunch of these little retriever scout agents.

It's a different design than what's happened to date, though. I'm sure that other companies will converge on it as well. You really want to design your agents for the problems that you're trying to solve. So if you want a really snappy agent that's going to answer things immediately, then this is probably not the best design for it, right? You might want something that does RAG, which is very fast, or a very basic search agent, kind of like what Claude Code or Cursor might do, which is something that just uses the terminal and the file system there and uses the same commands as an engineer.

That's a lot snappier than sending a lot of these retrievers out. So I think what we'll start seeing is this product co-evolving with the problems that you're solving, and there will be different agent designs for different problems.

No, I think that it's a really interesting question. And this is kind of one of the things that's interesting in research now, as opposed to before, is that before it used to be just around models, but now a lot of the research is in your agent design. And so what you ask is kind of an open question. There could be a hybrid system that kind of routes some queries to one agentic system and then routes other queries to another agentic system.

It could be that you figured out some elegant, simpler kind of multi-agent system that can do both things. It's kind of an open research question. It's very exciting. And in some sense, it parallels a lot of the unspoken research that was happening at DeepMind and probably at OpenAI as well during pre-language models. For example, projects like OpenAI's Dota 5 or DeepMind's AlphaStar project, which trained these expert-level agents to play pretty complex video games like StarCraft and Dota. A big question there that I don't think that many people appreciated is, how do you design the environment for your neural network to actually dispatch actions?

Right. So the most simple thing you can think of is, well, it learns to use a keyboard and mouse like a human does. That turned out not to work. So when you read the AlphaStar paper, you see that they actually figured out this particular way of factoring out the actions in order to make neural networks play StarCraft. And what that actually meant is that that was agent design. So now agent design is back in terms of designing scaffolding. Back then it used to be called environment design, but it's the same thing.

And a lot of the project in these big projects was not even on training the models. It was figuring out how the agents should actually interface with the environment that you're training it in. And where do you get your data from? What's your data? How do you collect it?

It's a really good question. So there are kind of a couple of interesting things to unpack there. The first thing I guess I would say is that, depending on the problem you want to solve, different types of search all fall into basically search. And maybe I would say that if you want really fast, but it doesn't really matter how accurate it is, or it just needs to be some kind of ballpark accurate, but really fast.

So, kind of the weakest form of search, RAG is great. Then there's this more agentic search that we spoke about, where the agent uses tools similar to the ones available to humans on a computer. It's kind of in between the spectrum of fast and slow, so it's slower than RAG, but it gives you better answers. And then even slower is what I'd call neural retrieval, which is you have a really long-context model and you ask that long-context model to retrieve stuff for you.

You feed it everything you can, maybe use multiple of them if everything doesn't fit in one. And then you ask that model to look at what you put in its context and retrieve the relevant stuff for you. That's kind of called neural retrieval, and that is going to take the longest. It's not guaranteed to be the best, but you can train it to be really good. And so that's kind of the spectrum of search capabilities as I see them today. The difference between something like this and MCP as it pertains to interacting with different sources of knowledge is that MCP is kind of like, it's stateless.

It's just a way for you to interact with another piece of software. But what you need to do here is you actually need to collate and index knowledge, right? You need to take data from software, store it somewhere, and make it searchable for an agent. That ends up being a bit of, I would say, a blind spot. There's a question of why hasn't this been done? One of the reasons is that, just from a—forget intelligence—just from a business model perspective, it falls into a bit of a blind spot for existing coding tools, which were meant to be served as SaaS offerings for a broad consumer base.

But an enterprise, like, this is key IP. It will not want this leaving into SaaS. And so you have to kind of rebuild your entire business to basically deploy the stuff on that enterprise's resources. And so most companies have been basically staying away from this problem of indexing and integrations at this level of depth because it requires them to basically change their entire go-to-market and business model. But it's also why it's really easy to switch around between various different coding providers today because they don't really integrate deeply.

And so you can try Cursor today, you can try Claude Code tomorrow, you can switch to Windsurf. And there's really—because they're pretty thin skins, I guess, on just the language model API—from a developer perspective, it's pretty easy to switch around and try different things.

We do have a SaaS offering because ultimately, if you flip something into on-prem, it has to start off as SaaS anyway. But the primary benefit to enterprises is we're not going fully on-prem today, but we are doing VPC, and that ends up being sufficient for a lot of big enterprises out there who already have their cloud infrastructure on AWS or Azure or GCP. This notion of it being deployable as VPC is extremely important to our organization. This is definitely, it's really a deal breaker.

Let's say before language models were useful, you kind of had to be in this world where you build the best language model and then you figure out the product. That's kind of the world we were in. And Anthropic spent a few years building a language model and then took off with Claude 3. OpenAI, GPT-2 is not really productizable. GPT-3, not really either. GPT-3.5 and 4, they were able to productize effectively. We're in a different world today where language models are pretty good, and so our strategy has been: we build this kind of multi-agent system.

Some parts we're training models for. We kind of see blind spots from third-party models, and other places the third-party model stays for today. Over time, we're going to kind of abstract all of it, but we're being a bit more strategic about which parts of the system you need to go after as a startup, because that's kind of what you have as a benefit as a startup, is that you can be a lot more focused on the problem at hand. Obviously, the downside is that you have to be a lot more strategic about the bets that you're taking.

You don't have the resources to go and train everything all at once, so you kind of have to take it one step at a time. And so in the long term, this is going to be a system that just learns end-to-end with reinforcement learning. And in the short term, we're applying reinforcement learning to fix problems that we're seeing as kind of blind spots in the existing sets of models when we're deploying them.

As a kid, I got pretty obsessed with physics and wanted to be a theoretical physicist.

As regular kids do. Well, if you're a Russian Jewish kid dropped in the middle of nowhere America, which is what happened to me.

Born in Russia, immigrated to Israel as a kid, and then immigrated to the States for the second half of my childhood.

Yeah, Canada is a big one.

I think the Soviet Union was an extremely technically academic culture. And a lot of the young scientists, when the Soviet Union fell apart, left to Israel, Germany, America, and Canada. And when we arrived in the United States, it was actually—

I was there for eight years.

Yeah, so from one to nine, I was there. I was just born in St. Petersburg, so I have no memories, obviously, of living there, just visiting. And then, yeah, arrived in the States.

In rural Washington State. So in Washington State, many people don't know this, but it has a hard line where the west side of the state is a lush forest, and then the east side of the state is a desert. And you can see exactly where that starts. It's not a gradual transition. It's just like trees, a dense forest, just starts somewhere. And then it transitions into immediate desert. I lived on the desert side. There's a national lab there.

My parents are chemists, and they got jobs in this national lab. The culture of that town was, it was one of the sites during the Manhattan Project. It was called the Hanford Site. It's where the plutonium was enriched. So it was the sister site to Los Alamos.

It is a vibe. Everything is themed around that event. The bowling alley is the Atomic Bowling Alley. The brewery is the Atomic Brewery. The streets are like Uranium, Mercury, Plutonium. The park there by the river—it's on the Columbia River—is called Leslie Groves Park, who was the ruthless general in charge of that project. So it's an intense town. Actually, the most intense thing is that the high school mascot, the town's called Richland, and the mascot is the Richland Bombers.

So they're B-52 bombers, and there are mushroom clouds on the basketball court, and that's it. It's an intense town.

Yeah, I guess I was—well, I didn't even think about the town's history as physics, even though that definitely is the case. But it was more that I was learning to speak a new language, had a lot of time on my hands, and my parents had their lecture books from—they bought this Feynman kind of series of lectures. And that was around, and I just spent some time reading it and just got into it. So that was the path to physics.

Ended up going through and doing a PhD in theoretical physics and actually defected to artificial intelligence. I realized that—first, I saw AlphaGo come out. And the short of it is, I realized that I had picked an interesting science, but not the science of our time.

That was kind of it. All the things I was learning in physics, all these kind of very interesting, great things were done basically 100 years ago, 60 to 100 years ago. And when you're studying something as a student, you don't really think too much about the timelines. You're like, "Oh, this is cool. This is so cool. I want to do this." But then, 60 years later, the field has crystallized in many ways, and it's not as dynamic as AI, where the frontier is just moving so fast.

And so when I saw AlphaGo come out, to me it just seemed like, oh, this is actually the science of our time, and I needed to do that.

I think it's augmenting. And in some sense, I think that part of what happened was that AI's impact in the world had clearly become hard to ignore. But there's no Nobel Prize for computer science. The Turing Awards have gone to AI for a number of years now. And so I think that the Nobel Committee, I'd imagine, felt like it needed to somehow shoehorn this. And so obviously there were very impactful AI breakthroughs with AlphaFold that resulted in a prize.

But what's interesting is that the Physics Nobel Prize was given to something that has not really had that much impact in physics. But I still buy it because there's kind of a physics smell to the breakthroughs that led to these systems called Boltzmann machines and Hopfield networks that Geoff Hinton and Hopfield got the prize for. They're very physics-y, and they look like the same exact objects that physicists study.

Exactly. And I don't mean it in a negative way. But yeah, I don't think that, as a physicist, when I look at, again, these objects, Boltzmann machines, Hopfield networks, very fundamental objects in the development of AI, even though they're not really even used today, some concepts from them are, those things haven't permeated physics in any way whatsoever. But they just look like objects that a physicist would study.

Kind of mathematically look very similar.

Yeah. Okay.

Had an interim where I started a small startup that went through Y Combinator, and it was basically doing machine learning prediction for inventory management. Really felt like I had to get on the frontier of AI research. I felt that was going to be where a lot of scientific impact accumulates. And so I ended up joining UC Berkeley as a postdoc, where I worked in this lab called the Pieter Abbeel Lab, which is one of these great labs for reinforcement learning and what was called unsupervised learning research, which is basically—large language models and diffusion models are the biggest outputs of unsupervised learning.

I didn't realize that at the time, but that was, in some sense, a—I wouldn't say miracle year, but it was a special year in that lab, given the people who were in it. A large portion of that lab ended up going to start impactful companies or being scientists doing very impactful work in large labs. To give you a sense, the first people I worked with were Arvind Srinivas, who now runs Perplexity, and Dennis Yarats, his co-founder. We worked together on some papers there.

Jonathan Ho, who is one of the inventors of diffusion models, was there and invented diffusion—the big paper that made them break through there. And this guy named Ajay Jain, who was on that paper, and they started Ideogram and Genmo, which are two startups in the video gen and image gen space. Deepak Pathak, who is the founder of a company called Skild AI, which is one of the premier robotics companies, was there.

2020.

Yeah, it was—now when I look back at it, it was a pretty incredible group of people, like Aditya Grover, who started Inception, which is a company that does diffusion models for coding. I was just thinking, I saw this guy's tweet, his name is Kevin Lu, the other day, who was in the lab, and he was an undergrad then, and most recently was leading a lot of the work for the mini models at OpenAI.

So just an incredible group of people at that time. Not obvious at all then. I mean, the research people were doing was very interesting, but I would never have predicted that so many companies would have come out of there.

And then I went to DeepMind. At the time, I was really interested in Toronto and New York. So I joined the group of this researcher named Volodymyr Mnih, who was largely credited with starting the field of deep RL. He was the first author of the Deep Q-Networks paper, which was the paper that got neural networks to play Atari. And his first set of papers actually largely defined deep reinforcement learning as a field then, and were basically DeepMind's claim to fame for a very long time.

And so I joined his group. The team we built together was called the General Agents team. And so the whole point was to do research for us to figure out: how do we build general agents? I think it was much more opaque then than now. And the big problem we were trying to solve is what people called, and still do, unsupervised reinforcement learning, which is really: how do you train reinforcement learning systems that are capable of assigning their own rewards?

If you don't have rewards without supervision, in the same way that you can give things some rewards, but kids and animals, when you look at them, they learn a lot in an unsupervised way. They interact with their environments without—no one is telling them to. And so we were thinking about how do we teach reinforcement learning systems in this way? I actually think the subject is coming back in vogue in the age of language models, with the question of how do you do pre-training-scale reinforcement learning?

How do you generate a lot of synthetic data if you don't have explicit rewards? I think it's actually a really interesting question now again, but that was the general agenda of what I joined to study. And the short of what ended up happening was that language models started working. And once they started working, that really changed, I think, my entire perspective on what problems matter and what didn't. Because a lot of the problems that we thought were fundamental problems were solved in this brute-force way for us.

Gemini 1.5, and then obviously 2 and so forth. And I joined with my co-founder. My co-founder, Ioannis Antonoglou, was leading the reinforcement learning team, the RLHF team. I joined his team and led a lot of the work for training reward models for Gemini and implementing the algorithms and so forth. And it was a very exciting time when I'd say a group of 10 to 20 people were—that was basically—all the people doing the RLHF work there.

After Gemini 1.5, we realized that language models crossed this threshold of utility where they're no longer research objects. They're going to be very useful. This was early 2024. We realized that the ingredients were in place to build a superintelligence. We felt that everything was there. There was one more piece to solve of going from RLHF to making reinforcement learning work. And that basically happened over the last year with reasoning models. So we felt that that would happen. Then the question was: superintelligence for what?

That was basically—we felt that you can't answer this question in the abstract by being a researcher that's really far away from product and customers. You really had to go in and define what that means from a product vision and what problem you're trying to solve perspective. It's like, we're not interested in building a superintelligence that will be superintelligence in mathematical Olympiads. And the difference between this era of reinforcement learning and the previous era of pre-training is that when you did pre-training, you made the models generally better at everything.

Reinforcement learning is much more jagged, right? It makes them good at what you want them to be good at. So just because you made them good at competitive code, that improves the general coding capabilities. But that does not mean that you'll have not even a superintelligence, but even just a useful intelligence for software engineering code. And I think an example of that is, I think Anthropic has done a really good job of building models that are meant for users of their products rather than benchmarks.

When I look at academic benchmarks, the Claude models are consistently worse than whatever else is out there, oftentimes not even close. They're consistently worse. And yet, from a user perspective, they're consistently better. Something has to explain that. And I think the explanation is that when you train large language models with reinforcement learning, they become jagged in the sense that they become good at what you wanted them to be good at.

Well, I think the Bitter Lesson actually doesn't say anything about generalization. The Bitter Lesson says that the systems that we should be thinking of and building are ones that scale well with search and compute. That's kind of it. And so what he's saying is, if models are limited today—this was actually, basically, the lesson was for researchers, but I think it's for product builders as well—if you're building your product with the assumption that these models are going to stay at their current intelligence level, and you make a bunch of hacks around your product to overcome those things, then in the next iteration of models, a lot of the hacks that you put in place will probably be Bitter-Lessoned.

And that was scientific researchers were seeing these kinds of limitations of models and plugging them in with kind of temporary hacks that would make them better at certain benchmarks. So the same lesson translates there. But the lesson is more to build systems that are good at soaking up compute and scale well with search.

Well, I think what's happening when we think of generalization, what's really happening is that if your training distribution is everything, then your test distribution just falls in your training distribution, and you have generalization. Maybe one point of view that I don't think that many people share, but I do think there will be a general superintelligence. But I think that it won't be one lab that has built it, but it'll be kind of the plurality, like the collection of all intelligences, will be a general superintelligence.

Because if you think of pre-training as kind of the soil or the substrate from which you can now grow superintelligence in various categories—medical superintelligence, organizational superintelligence, superintelligence for scientists and math—these are all different types of superintelligence. Some of them you can merge into one model. But I think that there'll be, if you think of different superintelligent plants growing from the substrate, yes, like a frontier lab will be able to capture some of them. But I think that there'll be new frontier labs built that build out, sort of grow other plants, and that the collection of this garden is going to be a general superintelligence rather than one company going in and growing all the plants.

I think from a research and compute perspective, that's possible. But from a product perspective, if you want to build organizational superintelligence, you have to go and integrate with all these customers, and you have to have solution engineers that support them, and you have to have salespeople that support them. And in this kind of rosy picture of a researcher who just trains models and hopes the model is a superintelligence, I just don't think that the world will play out that way, because it's meaningless if it's not coupled to a product and its deployment.

On the benchmarks, a quick side comment: I think that SWE-Bench is getting to saturation. And it's funny that even though you have these numbers, like 70% or something like that on SWE-Bench, those coding models are good, but they're not solving 70% of engineering tasks. So there's a sort of benchmark-to-real-world-problem misalignment, which is always going to be the case when your benchmark is not the actual thing that customers are using it for.

If there's one thing, I had fairly aggressive timelines on progress in my mind, and I would say things have moved faster even than I would have expected. I think that we've gone from autocomplete engines to things that are kind of semi-autonomous to now things that, for junior tasks, can just do them autonomously. It's pretty incredible. There is, in some sense, we are probably at an L4 kind of junior engineer level of autonomy, which is pretty incredible.

You still need to do code review in the way that you would do with an L4 engineer, but at a level of reliability where there will be some nits that you pick off, like you do with a normal engineer, but the thing they gave you is useful enough for you to review it in the first place, as opposed to, "This is just garbage, and why am I spending time reviewing this code?" So I think for junior code, little UI changes and small things kind of here and there, of which there are a lot.

So it's very useful. We're probably at L4 as a field. I think that over the next couple of years, the code generation ability will continue improving, and we'll have things that are quite intelligent. I don't know where exactly I'd place them because I think they'll be really good at generating code when you give them all the specs, all the requirements, exactly what needs to be built. But the whole job of a staff-level and above engineer is figuring out the requirements in the first place.

So that's kind of 70% of their time is spent on that: designing stuff, figuring out what the requirements are, planning in advance. Like, if I build this, will it conflict with these things? Soliciting information from other teammates. That's really what a staff engineer and above does. And then the implementation part is usually the sort of straightforward part. Okay, you spend like 20% of your time actually writing code and implementing things. And so the way things are progressing, I think we'll have very capable agents at the implementation level once you give them something to do that's really concrete and specific.

But if you don't solve this kind of contextual gathering or build out the contextual core, I don't think they'll be at that staff level as a whole.

Yes. So I think it's a very hard, very tractable problem. And the combination of this LLM with amnesia and the LLM's context core, together, will become the principal-level engineer, an AI engineer. And so I actually think that that's not too far away. That's, I would say, a couple of years away.

Yeah, that's right. It's kind of, at that point, from a build-out perspective, very similar, right? You're now on-prem or in the VPC of an enterprise customer. You have a centralized kind of knowledge around their code, and you've already centralized knowledge around other tools for them as well that are immediately adjacent to the next thing, right? If you're centralizing knowledge from Jira, that's both engineering- and product-management-adjacent, right? So I think at that point it just becomes adding other tools, other kinds of integrations, based on where you're seeing pull from the enterprise, and then enabling the ability to act on the user's behalf when they want to.

So instead of just being able to ask questions, enabling the actual agent to go and do stuff for them. So I think that that's going to come sooner than later, right? In the coding space, I think once you have that contextual core, you can integrate it into existing coding products, right? You sort of, like, if we think about the other coding products as being these autonomous software engineers with amnesia, you can fill that gap for them. Obviously, you can build out one of your own as well, but I think it ends up being kind of a notion of customer choice, right?

You want the overall solution to be the best for the customer. And you're, as a company, focused on what you believe is a fundamental building block of enabling superintelligence.

I think the entire makeup of the research team was earning a lot of money at big labs, basically, obviously, Ioannis and myself included. The thing to remember is that a lot of people get into this field because they're scientists at heart, or they're builders at heart. And so there is the financial element. You certainly need to pay enough, be generous enough, where that's not really top of mind. But people really care about discovering the next frontier and the next breakthrough, right?

The most exciting time to be in an AI lab is before it's obviously the frontier lab. I think the most exciting time at DeepMind was building Deep Q-networks and building AlphaGo. Those were the times, I think, where DeepMind was in its golden days. And similarly with OpenAI, it was building out the GPT series of models, like at the GPT-1, 2, 3 stage. And I think for Anthropic, it was really in the Claude 1 and 2 stage, where obviously now they're reaping the benefits of that breakthrough work that was done there.

We tend to attract people who have that kind of internal drive in them of wanting to be part of that next story, because they're already at a big lab, or they could join a big lab, and they'll always be able to. I mean, we'll see when ASI comes around, but that's not really that scarce of an opportunity.

When you actually look at which startups are out there that have the clarity and the team and potential of starting a new frontier lab, there are not that many. So there are actually a lot more spots at the big labs than there are at startups that have a shot at this. So I think that people end up, in a sense, self-selecting. We win over candidates over OpenAI, Anthropic, Meta, DeepMind regularly. Obviously, they get a lot more equity in this company as a percentage of its ownership.

And if they do back-of-the-napkin math of if they had joined Anthropic at this stage and got that percentage ownership, what it would be worth now, it would be absolutely generational.

So that tends to be, I think, if you don't have a good kernel of the initial team, if it's not strong enough, then it becomes very hard. But if you have a very strong initial team, and people see that potential for breakthroughs, then you become, in a sense, a scarce option because there are not that many places where you can do this. And later this year, we'll be shipping things that I don't think anyone ever thought a startup could do. I think that we're going to be shipping some things on the research side that I think everyone thinks you need to be a giant lab with 100,000 GPUs to do.

And I think it'll be quite interesting and surprising.

Yeah, we've built out, I guess, the company is—we kind of think about half product, half research. And so we've built out a research team, we've built out a product team. And then there's, I'd say, the majority of the makeup of the company is probably two-thirds of it is people who have research backgrounds at some of the big labs. Of those people, a bunch of them are kind of in this role that is between research and product. For example, the design of the agent, design research—that's very between research and product—or evaluations.

Like, what are you evaluating your models to be good at? That typically is something that's just on research. For us, it's kind of a cross-functional, end-to-end thing. The data that you're generating, the synthetic data that you're generating to train your models, that also cuts across all those things. So in a sense, I think it attracts, well, maybe people similar to Ioannis and myself that came into this and just wanted to be—we just did not want to maximize another academic benchmark.

We just wanted to solve real problems and have real evaluations. And so for those people, this ends up being a really good place. I think for people who would much rather kind of sit in a known entity and really focus on some specific piece of work of training the model, because they're so big that when you enter them, you're kind of given, okay, this is the sliver that you own. But that's really interesting nonetheless, because you kind of become a craftsperson.

So I think that's actually a very important skill to pick up. But if that's where people are in their lives, then, yeah, I think the big lab is definitely a better option for them.

Capital matters a lot. I think the difference is that you can't operate at 100x less capital than a frontier lab, but you can operate at, say, 10x—like an order of magnitude less capital—when you're really focused. So I think that capital matters a lot, and it really is determined by when you're ready to scale up your GPU count. That's basically it, right? I mean, there's obviously headcount and data and so forth, but the primary cost for any of these companies is their GPU expenditures.

So that's sort of, you kind of raise capital commensurate to when you're ready to scale to the next stage. And so I think capital matters a lot, but you can be a lot more efficient than traditionally frontier labs have been.

Yeah, thank you, Matt. This was fun.