Chasing Real AGI: Inside ARC Prize 2025 with Chollet & Knoop

ARC is an AI benchmark that tries to measure AI fluid intelligence as opposed to skill at specific tasks. And that's a very different approach from basically any other AI benchmark. Most benchmarks, they're looking at the ability to answer specific questions, perform specific tasks. That's going to be very reliant on knowledge or skills that can be memorized in advance of trying to pass the benchmark. And ARC is not like this at all. ARC is a set of tasks that you cannot prepare for. So it's not trying to measure what you know; it's trying to measure how well you can adapt to something you've never seen before on the fly.

As it turns out, this is actually something that's extremely challenging for AI. Current models, they're really good at absorbing mountains of knowledge and specialized skills, but they're not very good at making sense of novelty on the fly, at adapting. This is something that AI is not good at, and that's what makes ARC really interesting. And one thing to note is, last year there was this big shift in the AI research world where the AI research community started to move away from this idea of just scaling up pre-training and then using the pre-trained models in a static fashion at inference time, just running them.

They're not changing, they're not learning anything new, and they're not adapting to novelty. GPT-5, Gemini, that kind of thing. And there was a shift towards instead doing test-time adaptation, in particular test-time search, which is what you see in a model like, for instance, o3 or o1 Pro. So models that have much, much longer latency, much higher latency at test time, that are also much, much more expensive. We're talking 10x, 100x more expensive, and that have much stronger generalization power. And there was really only one benchmark that provided a strong signal about the fact that this massive shift was happening.

And it was ARC, because all the other benchmarks, they were really looking at skill. So they were not able to differentiate between just brute-force scaling and memorizing more and more knowledge and more skills versus actual IQ increases. And ARC is really the only benchmark that's looking at effectively machine IQ. And that's what makes it a really interesting signal. So why ARC-2? It's because ARC-1 played its role quite well. And in particular, it was able to capture this shift towards AI systems that, for the first time, had actual fluid intelligence, which was completely new.

GPT-5 does not have fluid intelligence, for instance, but o3 does. That's a huge difference. And it was the only benchmark that caught that. But the benchmark still had a number of flaws. The ideas behind it were solid, like the idea, for instance, that every task would be completely unique so that you cannot prepare for it in advance; the idea that all the tasks would be built only on top of core knowledge priors, meaning that they only require the kind of knowledge that's extremely basic, that everybody has, that LLMs also have, of course, that any child has effectively.

But there were some issues with the execution. So in particular, not every task was 100% unique. There was not quite enough diversity in task space. And most tasks were actually very brute-forceable, meaning that if you would just look at the task, you could instantly see the answer without having to do much thinking. And that means that if you're using a brute-force program enumeration process, for instance, to try to find the program that will solve the task, you actually have a pretty good shot at finding it because the space is pretty small.

And so in ARC-2, we are fixing that completely. So now it's much, much less brute-forceable. Any task that you see is going to take you at least several seconds of deep thinking, maybe even minutes. So for instance, we actually ran all these tasks, thousands of them, by actual people in live, in-person sessions in San Diego. And so what we found is that everybody can solve these tasks. We only kept tasks that could actually be solved reliably by multiple people.

And the average time to solve one task was about five minutes. So it also tells you that they're not trivial. They're not super easy. It actually takes you quite a bit of thinking. So it's basically just an upgraded version of ARC-1. It's the same ideas, the same format, the same principles behind it, the same goal about what it's trying to measure: just machine fluid intelligence. But the execution is better.

Yeah, absolutely. And it's important to note that it's not like recently LLMs started doing well on ARC. LLMs are still doing extremely poorly on ARC. What's actually going on is that we've seen a shift from systems that were just base LLMs used statically at inference time to much more sophisticated systems that are no longer LLM systems that use LLMs as part of a search loop, but fundamentally they're not end-to-end deep learning models anymore. In general, LLM is a very overloaded term, and different people can mean different things by that.

There's of course the base LLMs, where you have a prompt and you're just doing one generation at inference time. Then you have the so-called reasoning models that are first going through this step of generating a chain of thought to improve on your prompt and try to adapt a bit better to the task at hand instead of just trying to one-shot the answer. And then you have models that actually perform test-time search, test-time adaptation. Some of them might be doing things like test-time training, test-time fine-tuning, which is not something you see in commercial models.

But it's a technique you can apply. And increasingly, you're going to see commercial models that use test-time search, where instead of just trying to generate one single CoT to adapt to the task, they're actually going to run through this search. It could be, for instance, MCTS-style search to try to synthesize the perfect chain of thought for the task. And so you really have to keep in mind that there are very different kinds of systems with different kinds of abilities. And the limitations of the base models are still the same.

And a benchmark like ARC, which was designed to highlight these limitations, has completely resisted the paradigm of scaling up these base models. Back in 2019, when I released ARC initially, the best LLM out there was GPT-2, which of course was completely useless on ARC. Then about half a year later, we had GPT-3, which was also doing 0% on ARC. And today the latest base LLMs, they're doing something like 10% on ARC-1, but on ARC-2, they're doing 0%. And to be clear, 10% on ARC-1 is very low because you would be doing like 95% plus, perhaps.

No, definitely. Maybe you should try the eval set.

It's not difficult, right? It's really straightforward, especially if you take your time, you take a few minutes per task. And even if you're just doing extremely crude brute-force program enumeration, like what we saw in the first competition in 2020, you get like 20%. If you scale up the same technique just a little bit, you get to like 50%. So if you just do 10%, that's extremely low. So since 2019, there's been this 50,000x scale-up of base models that has resulted in basically a flat curve on ARC.

So where we've made a lot of progress is with test-time adaptation. So your question initially was about what LLMs are good at, not good at. Well, if you're talking about the base models, they're kind of bad at everything, to be honest. But what you can use them for is things like non-critical information retrieval, for instance, or automating tasks where failure is not costly. Because the thing is that they may be able to do something useful for you, but there's always some probability that they will fail or they will hallucinate in some way.

So you should definitely be using them with a human in the loop and for tasks that are non-critical. And if you're using them for brainstorming, information retrieval, and so on, you should make sure that you check your outputs if you're going to use them for anything. But they're super useful for a very wide range of minor automation tasks. So I use LLMs daily for all kinds of tiny things, for coding, of course, as a sort of text-line completion mechanism, or a replacement for Stack Overflow as well, if there's a question I need to answer. For brainstorming, they're great as well, checking typos in a piece of text and so on.

Lots of things that they're really useful for. What they're bad at is everything that's more like algorithmic-style reasoning or adapting to a problem they've never seen before. And that's why they fail on ARC, for instance.

Sure. I mean, again, intelligence is a very, very overloaded term, and different people mean very different things by it. But the definition of intelligence that I'm really interested in is not intelligence as the ability to do a bunch of specific things, basically a bag of skills. I'm interested in fluid intelligence, basically the ability to approach a new problem and creatively solve it, or approach a new domain and pick up a skill that you didn't have before, and do that efficiently. And I think the notion of efficiency is really at the heart of what intelligence is, because you can always do things expensively.

You can always use brute force, for instance, to solve a problem. Well, here's a very easy algorithm: just consider the space of all possible solutions and iteratively check them one by one. Eventually, in a billion years, you'll find the right solution. You can really apply that to any problem at all, including art, by the way. You can definitely solve art via brute forcing if you have billions and billions of dollars in compute. But that's not what intelligence is.

Intelligence is the ability to take a shortcut, to basically do these very hard things, solve these very hard NP-complete problems effectively, dramatically efficiently, like in linear time, effectively. And yeah, we are intelligent because we can learn fast, pick up new skills fast, solve problems fast. Like when you solve an ARC task, it takes you maybe a few seconds, maybe a few minutes. You don't have to iterate over 100 million different possible programs that go from the input to the output. You look at maybe two or three possibilities.

And yeah, so to summarize that, I see intelligence as skill acquisition efficiency. So it's not the fact that you can acquire skills; it's how efficiently you can do it. That's a measure of your intelligence.

On the small private set, which is what we actually used for the leaderboard, it was much, much lower. Generally, base LLMs were, like, at most around 10%. But yeah, so why this sudden huge jump? It's really because you have this paradigm shift away from just scaling up base models to doing test-time adaptation. And o3, best I can tell, is the most advanced, the most successful test-time adaptation model out there at this time. And what were they doing exactly? Well, I can only speculate.

Clearly it's doing test-time adaptation. That's why you're seeing this huge jump in generalization capabilities. But how does it work exactly? So my informed speculation would be that they're probably doing some form of test-time search over the space of possible chains of thought. And every time they try to make an edit to the current chain of thought, they're checking for self-consistency, for consistency with the task, and so on. And the check is performed automatically by something that you would call a grader model, a model that's just specialized in evaluating whether your chain of thought is actually solid, whether it's self-consistent, whether it's a good match for the problem, and so on.

So it's effectively a form of test-time program search, where the program is actually a natural-language program in token space, and where, instead of verifying correctness by actually executing it against, like, a suite of unit tests, for instance, you're actually asking another model to evaluate it for you. And yeah, that's what I think is the most likely, in the sense that it is consistent with what you observe, with the information that we have. And so, in general, one way you can tell apart a model that is just a base model, or maybe a model with a single chain-of-thought generation, versus a model that does this time search, is you can look at the inference latency.

If the model does search, it's going to take minutes to answer. Maybe it's even going to take longer. And you can look at cost as well. A model that does test-time search, it's doing a lot, lot more work at test time. It might cost 10x, 100x, 1,000x more. For instance, OpenAI o3 on the highest compute settings that we tried it on for ARC, it was consuming somewhere between $10,000 to $20,000 per task, for one little puzzle, which you could normally solve with a base LLM API for a few cents.

And another thing you can look at is just generalization power. And ARC is actually a very good benchmark to probe generalization power. And so you see this huge jump from base LLMs to models that perform this adaptation, just massive. We're going from 10% to 80%.

That's a great question. And we don't have a clear answer there. So they told us that they were using a significant fraction, I think they said something like 75% of the training tasks, to adapt the model in some way. So of course, to start with, that's entirely legit. The training tasks are there to train on them. So if you're training on the training data, that's perfectly legit. That does not invalidate the score that they're getting on the semi-private test set in particular.

But what did it mean exactly? What were they doing with these tasks? There are at least two interpretations you can have. One is that, oh, they were just part of the pre-training data for the base model. That doesn't really make any sense because everything was part of the training data for the base model. In particular, that's a model that's definitely trained on GitHub. Well, our training data is definitely on GitHub, not 75% of it, but 100% of it. And plus the public eval data is also on GitHub.

So it would also be trained on that, definitely. And on top of that, it will also be trained on many, many different repos on GitHub that provide more generated ARC tasks or that provide solution programs to specific ARC tasks from the training set and the eval set and so on. So presumably that's not what they meant. They must have been doing something special with specifically 75% of the training tasks. So another interpretation is that they have a system that can do few-shot fine-tuning via reinforcement learning, for instance.

of their model. And so they created an instance of the model, of the base model, that's just specifically fine-tuned for ARC. I think that's what's most plausible here. And that would certainly explain parts of the performance jump. But still, these are capabilities that we've never seen in a model before. So whether that's a model that's just ARC-specific or it's a more general model, regardless, I think that's extremely impressive. It's definitely a breakthrough.

I think ARC Prize, last year, did a tremendous job at highlighting what are the current best approaches to create models that actually have fluid intelligence. And they're all test-time adaptation methods. So in particular, one category of method that really got big via ARC Prize is test-time fine-tuning or test-time training, where you're using an LLM that's doing transduction, meaning that it's looking at the task and it's trying to directly predict the answer. You can think of it as opposed to doing induction, which is that you look at the task and you try to predict a program that will turn the task into the answer.

So in one case, you just directly predict the answer. In the other case, you try to predict the process or program that gives you the answer. And so you're looking at these transductive models, and at test time, you're going to generate input-output pairs from the current task that you're trying to solve, specifically for fine-tuning your model to map one input to the output. And then you're going to run that model on the test input and see what it gives you.

So that's one thing. Another category of approaches that got big with ARC Prize is program synthesis. So program synthesis is this idea that you have some language that you're working with. Maybe it's a language that's specific to the problem, like a DSL, a domain-specific language. There are many ARC DSLs out there. Maybe you could also just use a generic programming language like Python. And at test time, you're going to look at the task and you're going to try to write a program that solves the task.

And then, in the process, you're going to generate a bunch of candidate programs, and you're going to try to run them on the inputs and see if they give you the right outputs. If you find a program that's correct, it actually works on the test pairs, and it's short, it's reasonably short, then it's a pretty good candidate to be a program that will generalize on the test input. And there are many different flavors of that idea. So you could use, for instance, an LLM to generate the code in Python.

That's something you would do. That would be one instance of deep learning-guided program synthesis. You can also do actual discrete program search, meaning that you have operations, like elements in the DSL, and you're trying to recombine them into graphs, which are programs, and you do this recombination just via a discrete search process. So that's actually the kind of technique that performed best in the prior editions of the competition. In fact, the very first edition in 2020 was won by a technique that was just doing this with a handcrafted DSL and a very, very cool search process.

Well, the big constraint is that the way you write your program is not very sophisticated. So the probability that the first program that you try is correct is very, very low. To find the correct program, you're going to have to try many, many different programs. For instance, if you're just doing brute-force program search, you're trying programs at random. And so to find the right program, you have to iterate with millions of potential solutions or points in the search space. And the main bottleneck is that this search process takes a very, very long time.

It's a very large search space. And to evaluate all these points, each point takes you some amount of computation to evaluate. You're going to have to evaluate millions of points. This is extremely expensive. And of course, it's not how humans do it. Humans probably do something akin to program search, but they're doing very, very little search. You look at a problem, you're quickly building a very small set of hypotheses to explain what you're seeing. Maybe you have two, maybe you have three, and you test them.

So it's a form of search. It's a form of program search with backtracking. If one of your programs doesn't work, you can just mentally debug it or just discard it and move on to the next hypothesis. But the size of the search space is tiny because humans are fundamentally not very good at these things that computers are extremely good at, like evaluating discrete search spaces with millions of points, for instance. So humans can have some shortcuts to find the right program intuitively.

Yeah, I mean, we don't know how humans do it, but clearly intuition is part of the picture. So intuition is this idea that you're going to leverage the experience you have to make fewer guesses, so that there are fewer guesses that you have to check before you find the correct guess. A very crude version of that is what LLMs are doing. With an LLM, when you generate a program conditioned on some task definition, effectively you're using a statistical prior about the shape of program space to generate fewer guesses, right?

So most of your guesses are going to be syntactically correct, for instance, which is already a nice property. Most of them are going to be related to the task in some way, and so on. And so if you're using a statistical prior like this, maybe you can find the right program with a few million programs, or even less than that, a few tens of thousands, maybe thousands of programs, instead of trying billions and billions, which is what you would have to do if you were just doing brute-force program enumeration.

But the more sophisticated your intuition is, the fewer guesses you have to make to find the right answer. And if you look at an extremely sophisticated intuition system like humans, we just see the right answer. We only make very, very few hypotheses in our mind, like just a handful, effectively.

They're usually the same thing because if you're doing any kind of program synthesis, even by using an LLM to generate the program, for instance, you still have to check that it is correct. It's probably not going to be correct on the first try. So if you're doing program synthesis, you're definitely doing some form of program search. You have to test a few guesses before you find the right one.

Yeah, it's like back when GPT-4 was released, the universal narrative, like everywhere in the tech industry, was, okay, GPT-4 is just a bigger version of GPT-3 trained on more data. So it's the same architecture, it's the same principles behind it. It's just scaled up, and look at this incredible step-function improvement. And GPT-3 itself was just a bigger version of GPT-2. And so the idea was that, well, GPT-5 is just going to be the same, but it's going to be 100x bigger.

And we're going to need these massive data centers to train it. And it's going to be truly AGI. AGI is just going to spontaneously emerge from scaling up this thing. And basically all the evidence was pointing in this direction. If you looked at benchmarks that were based on just brute-force memorization of knowledge and skills, if you didn't look at ARC, this was the takeaway you had. And so everybody at the time believed that. And maybe some more sophisticated researchers, like maybe some at OpenAI, I don't know, maybe some at DeepMind, had different beliefs, but that was really the mainstream narrative.

And, well, interestingly, one year later, like about one year and a half later, this narrative was gone. People were completely aware that no, just scaling up pre-training is not all you need, that you actually need entirely different ideas and maybe, in particular, test-time adaptation to achieve actual fluid intelligence and AGI.

Yeah, I mean, the basic idea is that there's one track for self-contained approaches with no internet access that are very efficient. So they have a very limited compute budget, and authors must open-source them at the end of the competition. So this is really designed to incentivize open sharing and get as many ideas as possible, with a big focus on efficiency. We believe efficiency is not just a good feature to have in your system. It's actually at the heart of intelligence.

And the other track is to provide continuous benchmarking of frontier models to be able to track, okay, if you look at the best currently available commercial frontier models, things like right now, for instance, o1 Pro, soon in the future it's just going to be o3 and so on, Gemini 3, whatever. How much fluid intelligence do these models actually have? Mostly independently from the efficiency consideration. We still do have this notion that we want to monitor efficiency. So we're going to be reporting results on a 2D plot.

So we're not just looking at the score as a scalar, we're looking at the score associated with the cost per task that was required to achieve this score. And of course, a model that gets you the same score, but at much lower cost per task, is a smarter model.

So, yeah, they were at the top of the leaderboard up until the end of the competition. They ended up dropping out because they did not want to open-source their solution. And of course, that meant that they were not eligible for the prize.

To be clear, an average person on their own would not score 85%, probably. Based on our own testing, an average person in our test sample would score about 60%. But if you take a small panel of people from our test sample, like about 10, and you just have them solve the task independently and then do majority voting, that panel should score collectively 100%.

We were not trying to select for people that were good at solving puzzles, for instance, that would be strong at ARC. We really hired random people. We went through a service that exists to basically just hire people for scientific testing. And we ended up with a pretty diverse crowd. We have anywhere from Uber drivers, students, people who are unemployed. So basically anyone trying to make some money on the side. So very much regular folks. And we know our tasks are solvable by regular folks.

One thing that we did with the task selection process is that we tried to eliminate tasks that could be solved by everyone, because if something is just universally feasible, it doesn't actually give you a very strong signal. It's probably something that can be brute-forced very easily. And we also eliminated tasks that were too hard. So we only kept tasks that could be solved by at least 2 people independently in under 2 attempts so that we know that it's solvable and it seems, in fact, reproducibly solvable.

It's more than just a one-off.

We can go into what each one of them actually represents, if you'd like.

The high-level takeaway is that the new ARC tasks all require some level of deliberate, deep thinking. You have to look at the data, you have to think for a while, and the reasoning chain that you're going to have to come up with is going to have a few hops. In particular, it might have a few control-flow hops, like something like an if statement, for instance. And that's something that we're not seeing a lot in ARC-AGI-1. And it really creates this big difference in terms of how challenging the dataset is for LLMs in particular.

Right. I mean, pragmatically, you have AGI when it's no longer possible to easily come up with tasks that you and I can do naturally, but no AI system can do. If it's no longer possible to come up with this kind of task, then you probably have AGI.

Yeah. The closer you are to AGI, the harder it's going to be to come up with this kind of problem. And right now, it's not difficult to come up with this kind of problem just yet. So we still have some time to run. So how long will it take for ARC-AGI 2 to be saturated? We don't know. And it's not just a question of when will, like, 85% be reached? It's also the question of how efficiently it's going to be achieved.

But we're already starting to work on version 3, which should have a brand-new format. So we don't think ARC-AGI 2 is going to be the last benchmark. I think that there's definitely going to be a next benchmark after that, and then a next benchmark after that. There's probably still a couple of important milestones on the way to AGI.

Well, we have a vision about what we want to build, and it's a vision that's very different from what the other frontier labs are actually looking into. So we have some insights, we have some beliefs that are going against the mainstream narrative in the AI research community. And we believe the best way to actually execute on that is as an independent small-scale entity. What's also going to be very important for us is to move fast, to iterate fast, and that's much easier to achieve as a small, scrappy team.

That's right. We really believe that the frontier is test-time adaptation and that the best approach to solve test-time adaptation is deep learning-guided, or intuition-guided, program search. And so that's what we are trying to achieve.

What is interesting to us personally, but it's also the fact that the kind of technology we are building, it's differentially advantaged, is that it's going to be capable of solving problems that have never been solved by humans before. That's a very different deal than LLMs, for instance. This makes sense, but effectively only in verifiable domains, like symbolic domains, things like mathematics, physics, programming, but not, for instance, creative writing. So we're not trying to help people write better emails. We think LLMs are actually a great system to achieve just that.

That's not what we're building.

Thank you, Matt.