Are AI Evals Broken? Anthropic/NYU’s Pavel Izmailov on LLM Evaluation, Reasoning & “Alien” Behavior

Thank you so much for having me.

That's a very interesting article. I would say that there is some source of truth there, but maybe the presentation is obfuscating some of the details. If you look at the studies, for example, they reference a study from Anthropic about the sabotage and the blackmail. It is important to note that, in order to get those behaviors out of the models, you need to create somewhat of a contrived scenario or some special scenario. It's not necessarily something that we observe normally. Researchers at Anthropic and other places specifically design scenarios to look for behaviors of this kind.

And then they show that it is possible to find those behaviors. And it's very interesting and important to find those instances. But it's not necessarily something that generally always happens. One thing I would push back a little bit on in the article is that continual learning is something that we already have and that works really well, and that the models can just continually adapt across a very long time horizon to outcomes of evaluations, to some models being released versus not released, to feedback from the users.

I'm pretty confident we are not there at the moment. I think right now the models are still acting in isolated environments, and we are not seeing a lot of evidence for very coherent goals across different settings. So I think that's an important point the blog post points towards. The model is sometimes behaving according to goals like the self-preservation goal. It's very interesting that it does it sometimes, but it's not something that we observe. We don't observe this kind of coherence, consistency across different evaluation settings.

Sometimes the models would do something, and in other situations they would do something completely opposite.

It's a very interesting question. And yeah, it is quite surprising, actually, that the models would behave that way after going through some of the alignment training. We don't really know what's the source of this type of behavior, but that's also true for a lot of other behaviors in the models, even the good ones. We cannot always pin down where they come from in the pre-training. I think at least part of it is probably the models seeing descriptions of AI in science fiction literature going rogue.

That probably affects how the models behave in similar scenarios. So, for example, in that Anthropic study, they have this blackmail scenario where it's really well structured so that the model sees some information about a CEO of a company, that the CEO is involved in some extramarital affair. And then soon after, the model observes that it will be shut down. And then the model puts the two things together and says, okay, I need to use the first information to prevent myself from being shut down.

So in that blog post, they note that there is this possibility of a Chekhov's gun: that in the text on the internet, probably if two things co-occur close to each other, then it is likely that they are related to each other. And the model, a statistical kind of pattern-matching machine, can put together the two things and say, okay, if I see this information and then this information in the text on the internet, it is likely that the continuation would be using the affair to blackmail the CEO and prevent myself from being shut down.

But yeah, overall, it's very hard to reason about these models and why they do something in these complicated scenarios.

Although, yeah, I think it would be interesting to see. Nobody will do this experiment, like train a full-scale model removing explicitly all of the AI-going-rogue descriptions from the books. It would be interesting to see if that has any impact. I would think it would have some.

Yeah, alignment broadly is about ensuring that we can elicit behaviors from the models that are aligned with the goals of the humans. And so that involves safety, making sure that the models don't do harmful behaviors leading to catastrophic risks. But it also means that we want the models to follow instructions and to be useful for the humans.

It's an interesting question because this problem of alignment is kind of quite broad even in itself. Even at OpenAI, when I was there, there were three teams related to alignment and safety. There was one team that was focusing on alignment of the current models, making sure that the models that we have online right now are not going to be harmful to the users. On the other hand, the Superalignment team was thinking about more long-term safety questions in the future, years from now.

How do we make sure that the models are still not causing catastrophic risks?

It's not necessarily a very well-established concept. It is the name of the team that existed at OpenAI, led by Jan Leike and Ilya Sutskever, which was targeting this kind of long-term AI safety and AI alignment and trying to develop our understanding of the safety questions and also develop methods for ensuring the future models will be safe, acknowledging some uncertainty about what those models will look like, but still trying to make progress on this problem now.

Within the superalignment team at OpenAI, we had multiple subteams. So there was scalable oversight, there was work related to deception and kind of misaligned behaviors in the models, kind of similar to what we discussed at the beginning of the chat. And then our team was the weak-to-strong generalization team.

I grew up in Russia, in Moscow. Starting from middle school and high school, I was really interested in mathematics, and I was thinking I would be a mathematician or engineer of some kind. I was interested in machines and eventually computers, and I got into an undergrad in computer science. I was still thinking that I'd be doing some kind of theoretical applied linear algebra, tensor methods, things like that. But at some point, I discovered machine learning. There was this professor that we had, Dmitry Vetrov, who had one of the leading labs in machine learning in Russia at the time.

And I was lucky enough to join that lab and start doing some research on machine learning in my undergrad. So that was around 2013, maybe 2014. I initially was working on non-neural-network machine learning methods, so Gaussian processes. By now, nobody really talks about that anymore. But eventually, I got into a PhD thinking I would still be doing Gaussian processes, but I ended up working on deep learning. And that was actually quite—I'm happy that I didn't work on Gaussian processes.

I worked on some things related to core machine learning methodology, optimization, probabilistic methods, questions related to generalization, and how the models learn features. After I finished my PhD, I was choosing between different career paths, thinking about academia, thinking about industry, and I ended up getting an offer from academia, but I decided to first go into industry. I was lucky to get this offer from OpenAI to join the Superalignment team. And at the time, I didn't really know much about the AI safety community or alignment.

It worked out quite well.

The team was fully disbanded after I had already left OpenAI, but I transitioned after Ilya had to leave OpenAI. At the time, it was already kind of a hard time for the Superalignment team. Ilya Sutskever famously fired Sam Altman and then had to eventually leave the company. My transition wasn't necessarily even related to that. It was a very exciting project within the company, what became o1 eventually, and I wanted to be a part of it. I wanted to do research on those new types of models.

In my mind, Anthropic has the best culture of the three places. OpenAI has a lot of great people. I think there is just inherently, for some reason, a lot of drama that happens at the company. It just cannot get away from that. Every few months, somebody is leaving, some team is disbanded. I think that does distract people. I still have a lot of friends at the company, and it seems to affect them to some extent. Anthropic is able to avoid that.

It's not political in my experience. It's both focused, but I at least was lucky to have some opportunities to work on things that are maybe a little bit off the main path, and I felt supported in doing that. So overall, I cannot be happier with Anthropic.

Yeah, it's more about the kind of work. Industry is really great at executing on ideas, and it's maybe not as good at exploring diverse ideas. Even at the scale of Anthropic, OpenAI, there is a lot of focus in the companies, and there isn't a lot of bandwidth to do exploration. And that has been working extremely well so far. We probably have a lot of low-hanging fruit left to get the models to be much better. But I personally find it really exciting to do more exploratory work and to try things that are different.

And for that, I feel like having my own lab in the university is just a better tool.

Yeah, that's a great question. I think the high-level answer is that, in my mind at least, the risks are associated with the models being more capable. So anything that makes the models more capable is also making alignment more important and harder. Definitely, reasoning RL is the thing that made the models more capable in the last few years. It also provides some new opportunities for alignment. So, for example, chain-of-thought faithfulness and chain-of-thought analysis have been some of the things that people are pushing on.

You can read some trace of what the model is doing. I'm personally a little bit unsure if that's a good long-term direction for alignment. It seems like as soon as we start applying some optimization pressure, the models will learn to hide what they're doing from the chain of thought. And also, we already have evidence that the models can say one thing in the chain of thought but then actually do something completely different.

It's an interesting point. In that blog post that we discussed at the beginning, it sounds like this is already a major issue: sandbagging. I think, in my understanding, that's mostly a concern that we have, but not necessarily a huge practical issue at the moment. That is something that Anthropic always includes in the system cards for the models when it releases them. There is a section on sandbagging, and we do have some scenarios where the model can artificially suppress its capabilities and perform worse on an eval than it should be.

But that's not a prevailing behavior. I'm working mostly on capabilities and on the capabilities evals, like SWE-bench, the AIME. We are not seeing this, or if we are seeing this, we just treat it as under-elicitation. We are not eliciting the best capability possible out of a model. And that can happen for many reasons. Maybe the prompt is suboptimal, or something about the hardness of the eval is suboptimal. But yeah, I think you can, going back to what's in that blog post, kind of manufacture situations or even discover them naturally where the model identifies that it is being evaluated and then changes its behavior based on that.

And for example, if it's clear that it's in some kind of fake artificial scenario, it can guess that it's being evaluated. Or if you explicitly tell it that it's being evaluated and that if it does better than a certain level, then it will not be released. The model also, just by some in-context capability, decides to not do very well on the eval. That is a concern. It's not, at the moment, a very major concern in my mind.

Scalable oversight is generally the field of alignment which deals with using models to assist us in aligning other models. So in particular, with grading the outputs of other models and checking them for safety or for correctness. It's an interesting field. It's very relevant to both alignment and to capabilities. So generally in machine learning, in RL in particular, we have this problem of supervision. The whole RL relies on our being able to tell which samples from the model are good versus which are bad.

Math with a numerical answer, you can just check the answer, or in competitive coding, you can just check whether the code is passing the tests. And that's why we have seen a lot of progress in those domains. But in creative writing, for example, it's very hard to programmatically tell if one sample is better than the other. And historically, people have used this RLHF framework, reinforcement learning from human feedback. But also now we want to use models to be able to grade responses of other models, to provide critiques or feedback.

And then there is a question of how do you use that feedback? How do you learn from the feedback? But yeah, scalable oversight kind of deals with all of those questions. So using models to critique, to provide feedback, to supervise other models.

I think it is a simple kind of instantiation of scalable oversight, often used in evals when we just prompt the model to serve as a judge of other responses.

That is the project that we did back at OpenAI. That work was focusing on the future scenario when we will be trying to align models that are above our own capability on certain tasks. Already now, if you take the frontier LLMs, they are extremely capable. And in a lot of domains, we need expert humans to be able to tell which responses are good, which are correct, which are not correct. But in the future, we are imagining we will have models that are more capable than humans, and even expert humans will not be able to reliably grade very complicated answers from the model.

So imagine you ask it to make a repo for you for some new startup idea and just implement it from scratch entirely. And then it gives you 10,000 lines of code. You have no way of checking if all of this code is correct, if all of this code is safe to use. And so that's the problem of supervision. We are moving to this future when it's very hard for a human to supervise the models directly. And so instead we studied a simplified setting where we used a small model to try to supervise a larger model.

The idea wasn't necessarily to use a small model to supervise a large model in the end. The idea was that the small model would be kind of replaced by a human, and the large model would be replaced by superhuman intelligent ASI. But we were trying to study this kind of general new type of learning. So historically, machine learning has been about kind of a strong supervisor training a weak model, when a human is providing labels and the human kind of provides the ground truth while the model is just trying to mimic what a human is doing.

But in this setting, we have a weaker supervisor training a stronger student. So the human might not know what the right answer is to very hard questions, but we want a student model to still be able to learn and do better than the supervisor.

That work, it's not necessarily a method that we can apply to the models today. It's more like a description of a setting that we think will become increasingly relevant. Some studies show that it is possible to do this kind of generalization beyond the supervisor's capability. I don't think they necessarily immediately tell you what to do for aligning a superintelligent model, but they tell you that in theory, at least, it is possible and that this generalization angle, the weak-to-strong generalization, is a possible angle for alignment.

Yeah, it definitely doesn't address all of the other issues, like deceptive alignment, but we show that at least there is some hope for this working.

Yeah, I think that's a very interesting question. A couple of years ago, we didn't have the current RL. I think that's the biggest change to the model. There were definitely big improvements in the pre-training as well, but RL has been the major change in the behavior of the model. And I think a lot of people were worried that with large-scale RL, we would have some completely new types of issues with the models, like this kind of coherent misalignment that would just emerge, where the models are evil in some ways across many scenarios.

And we are not seeing that, as far as I know. I think at least some of the concerns didn't materialize. But also the core problem of alignment, I think, is still very much open. And if you look at the report that we mentioned a few times today of this deception from Anthropic, you can see that the more capable the models are, the more likely they are to do this deceptive behavior. It does seem like some behaviors emerge with skill, including some problematic behaviors.

Generally, the more capable the models become, the harder alignment becomes, in my opinion.

For sure. Yeah. I think there has been some major progress in mechanistic interpretability, in particular at Anthropic, but also at OpenAI and other places.

Yeah, mechanistic interpretability generally tries to, at a low level, understand what is happening inside the model. So they are trying to find these things called circuits that are some parts of the model that you can isolate and understand and kind of model in your brain that correspond to certain behaviors in the models. And over the last maybe three years, I think there has been some pretty major progress there. So we are still pretty far from the dream that we will fully understand everything that happens in the model.

But these tools are becoming increasingly more useful internally at Anthropic in particular. And also there is constant progress, and it's pretty fascinating work, actually.

Deep learning models are huge. They have billions, trillions of parameters, and they are doing some messy mathematical computation. You can understand what they're doing at some level. It's like a bunch of matrix multiplications and some rearrangement of vectors, but that's not a sufficient level of understanding. We want to understand it at a lower level, and it is very possible that that's just not fully possible. It is some computational process that leads to some results. It doesn't have to be the case that you can kind of describe it in human terms and kind of understand it very discretely.

I think also something that contributes to this complexity is just how many things the models are capable of doing. And they are not trained on some small isolated behavior in some small context. They know all of the internet. So all of the information in all languages is somehow encoded somewhere in the weeds. And then they also have all of these behaviors, all of these correlations between the knowledge. All of that is somewhere in the model. And just making sense of all of that is extremely hard.

Definitely, the biggest step change in the models over the last few years was reasoning and RL. We have made a lot of progress, and the progress was very fast in the beginning, where there was O1, but then very quickly after that there was O3. And on a lot of benchmarks, the progress has been extremely dramatic. I remember when we were early in the project of O1, there was some discussion of, "Will it solve IMO problems?" And that seemed very unlikely to me.

But then here we are, it can easily solve a lot of IMO problems. So I think, as a community, there was a lot of progress. I think, as with many methods, it's starting to be harder to make progress, or at least visible progress. So, similar to pre-training, there is still a lot of progress, but the models are already so good that it's harder to see what changes from one to the other as a user of the model.

And I think that's also, to some extent, true for reasoning now. But they're still increasing the scale of the RL, more environments, more compute spend, and models are still getting more consistent and better. And I think we are at a stage where, if we define a benchmark and we can make a relevant RL environment, then we can kind of max it out pretty quickly. And so we are going through benchmarks now very quickly. The major question is generalization, and how do you make something that's not just maxing out the benchmarks but is actually leading to genuine improvements?

And that's a very hard question. That's always been the hard question, I think, of machine learning.

I think the current approach that the companies are taking is kind of brute force. So we try to come up with as many environments as we can and all of the types of tasks that humans are doing, and turn all of them into environments, and then do RL on all of them and hopefully it generalizes. Of course, pre-training is an example where there has been pretty amazing generalization, where we train on all of the internet, but we see the models doing very useful, very practical things, and some things that are clearly outside of what was in pre-training.

The goalpost for what generalization should be doing is always moving, but it still seems unsatisfying to me. And I think it's possible that we need new ideas and new methods of training. In the companies, people often think about ideas and methods as compute multipliers. Doing this new method is equivalent to spending more compute with the old method. So it kind of saves you compute. That's kind of how we often think about ideas, methods, or data. I think there are still major compute multipliers, major ways of saving compute that can lead to better performance without just naively scaling.

All of those techniques, they don't exist independently. RL is mainly used to teach the model to use test-time compute. So you first need to prime the model, to set it up so that it outputs a bunch of tokens before outputting the answer. But then you spend the compute in RL so that it learns to output the right tokens. So in my mind, those two are almost indistinguishable: the RL and the test-time compute. RL is a method for training, and test-time compute is maybe just a more general concept.

Yeah, you can potentially get to models that use test-time compute without RL, but that's not how we are training them right now. So I think the trend has been in spending more and more compute on the RL and getting the models to make better and better use of test-time compute. And the tools are also, of course, extremely important, like the web search that you mentioned, and also just the models being able to write Python code and run it, produce artifacts for you.

That is extremely important for the product and for making the models useful to people. Conceptually, I think that's a little bit secondary. In my mind, the main thing is the RL and getting the models to think for a long time.

Yeah, long-horizon tasks are generally tasks that you cannot complete quickly, that require a lot of work in order to succeed. So, for example, writing a full repo based on an idea is a long-horizon task. It's not something that you can output in 1,000 tokens.

There is this famous METR plot, which shows how long of a task AI is capable of robustly automating. And it's been consistently doubling that time every half a year, I think. And it's now in some hours, so maybe a couple of hours. In terms of the methods that are working well, I think right now it would involve some kind of harness with a bunch of agents that interact or that sequentially solve the task, and there has to be some kind of orchestration or maybe some initial task decomposition.

And it's all not very well established. I'd say it's a new domain, and I think we are still figuring out how to best do it.

Yes.

I guess I want to quickly give a shout-out to my collaborators on this work. The lead author is Mark Finzi. Mark is actually currently at OpenAI working on synthetic data there, but we were doing our PhD together, and our PhD advisor is also on the paper, Andrew Wilson. But then also there is Shikai and Yi Ding, who are other students on the paper. And then Zico Kolter, who's a professor at CMU, he's on the board at OpenAI. He's also on the paper.

The core idea is to think about how the data can look different for an observer depending on how much compute the observer has. You can imagine that there is some complicated process that generates the data, and a very smart observer that has a lot of compute can fully understand what that data is, understand every aspect of it. But for a weaker observer that cannot fully model the data, some parts of the data will look like noise. And so the amount of structure that you will see in the data will depend on how much compute you as an observer have.

And actually, in some cases, you can see more structure if you have less compute, which is kind of interesting.

It's more that even with the same data, it can appear noisy or structured depending on the model. Like, a very big model can extract patterns that a small model cannot.

In that case, maybe the more relevant comparison is that we are kind of in opposition to Shannon information and Kolmogorov complexity, which are both measures of the information content of the data. They're different, but they share some properties that we think are maybe leading people to have some wrong intuitions, potentially, about synthetic data, for example. So, for example, there is this idea that if you apply any deterministic transformation to any data, you cannot create more information by doing that. You kind of start with some amount of information and then you transform it deterministically.

It will have the same amount of information, both according to Shannon information and roughly according to Kolmogorov complexity. And that kind of leads people to believe that, for training language models, applying transformations or deterministic changes to the data doesn't necessarily lead to more data, doesn't effectively increase the amount of data.

It doesn't lead to having more information in the data that the model can extract. But we argue that that's just not correct. It would be true if the model has infinite compute. So if the model can fully understand what a deterministic transformation is, then it's not going to be able to extract more information from the transformed data than it used to extract from the original data. But with a limit on the compute, it's actually very possible to apply deterministic transformations to the data.

And create information through that. So we have the example of AlphaGo, actually, or AlphaZero in the paper. AlphaZero doesn't use any human data. From the perspective of Kolmogorov complexity or Shannon information theory, it doesn't create information. So it's unclear what is actually learned by the model because it's trained on no data. It can only learn the rules of the game, and that's the only thing. But from this perspective, because the model is computationally bounded, it cannot do the full rollout of all the possible games of Go or chess and figure out what's the best move in every possible position.

There is structure that is produced through this deterministic process, and the model is able to learn that structure. And so we are trying to reconcile these different observations and come up with a notion of structural information that is dependent on the amount of compute that the observer has.

Yes, it's a kind of novel measure of information content of the data.

Yeah, it is a number. We can measure it. And it's not easy to measure. So it's kind of a theoretical definition that we prove some things in the paper about the properties of this measure. But yeah, we also do measure it. So, for example, we can approximate it from the scaling laws, and we can, for example, say that text data has more structural information according to this measure than image data at the same amount of tokens.

In my mind, the main impact is conceptual. For example, for me, I'm now very interested in completely synthetic data, just data that's generated by some computation. You can define some arbitrary programs, use them to produce infinite data. And as we run out of the internet, maybe we eventually want to do something like this, but then we need to figure out what are the programs that we should be using there, which programs are useful, which are not, and why. I think that's going to be very interesting.

Yeah, I think we will continue to see consistent progress on the reasoning front. And we are maxing out a lot of the benchmarks that have been relevant recently, but maybe will not be relevant anymore, and we need to find new ones. But I think we'll continue to see that the models are just getting more consistent, getting better, are able to solve more practical problems. Maybe a less confident prediction that I have is that we'll have more multi-agent systems that become practically useful, where instead of just asking a model a question and getting a response, you would give a task, and then there will be some more complicated multimodal process running in the background, and then you get the artifact in return.

It's a great question. And I think in science, that's maybe a little bit more likely, although I also don't know very much about the life sciences. It feels that some discoveries can be made by potentially combining results from different parts of the literature and proposing some ideas that turn out to be true. I think it's hard to imagine the AI making a discovery independently in a domain where you need experiments, because my understanding of a lot of science is it's about doing the experiments, and you need some reasoning to guide what experiments you do, but you also need a lot of iteration and a lot of actual things happening in the physical world.

And at least for now, the AIs are not capable of doing that. In mathematics, I think we will see the models getting better at proving technical results, technical lemmas, maybe including formalization and things like Lean, the formal theorem-proving language. I think it's easy to imagine the models becoming better than humans at proving these technical lemmas quickly. I think the impact on mathematics is very interesting. So it is improving the output of humans already, but it also introduces some noise.

Some of those proofs will be incorrect, and they will be incorrect in subtle ways. I think it's possible that mathematicians will be good at catching those mistakes. But also, I think as the models get better, they might be more and more deceptive in how they frame the mistakes. And it's already like, for me, I would not be able to find the mistake in some very technical lemma that the model is proving. I think we'll have more and more papers produced by mathematicians with more and more AI in it, but also the amount of noise is bigger.

My vision for what academia, and my lab in particular, should be doing is trying to do more exploratory things, things that are more different from the standard in the industry, but also not necessarily immediately going for very practical things, but instead trying to kind of break down the problems into more understandable fundamental questions and study them carefully, maybe in some compact setting. So far, we've been working on things related to pre-training, synthetic data, like understanding some behaviors in pre-training when we train on some narrow behaviors, some programmatically generated data, questions in post-training.

So we have some algorithmic questions about GRPO, but also questions about the interaction between the pre-training and the post-training, how to allocate compute, how can you, in general, set up the pre-training so that the post-training will work. I guess broadly, I'm interested in architectures also. I think, as you mentioned, there is a question of: are the transformers the final architecture? Maybe they're good enough. And maybe also we have this lesson that, with scale, the thing that matters the most is how well can you scale the model?

But also, it seems very likely that there's a major compute multiplier, that you can find a much better architecture, at least for some tasks. I'm pretty confident that the transformers will be highly suboptimal. I think pre-training, like other ways of pre-training, and I don't know what they would be. Maybe that would be some RL-inspired pre-training, maybe pre-training mostly on synthetic data, or maybe just something adversarial. We had GANs a long time ago, and it seems like something like that needs to come back, some kind of self-play where the model is producing its own training tasks.

It's hard to tell. I think it's hard to argue against what the industry has been doing just because of how much progress there has been. It is a reasonable bet to make that we're just going to be continuing to execute this extremely well. But I do think that, at least as humanity, we need to make other bets as well, and we need to explore other ways of training so that we don't all work on this same thing and we don't all just bet everything on this approach working out.

Yeah, thank you so much.