How to Build Long-Horizon AI Agents — Mitch Troyanovsky, Basis

Yeah, I think maybe the best piece of advice—not for building agents, but for working with AI in general—is that you need to give it as much context as possible, because it, by definition, is always missing context in some way. Speaking is just so much faster than writing things down. And in fact, when you try to write things down, you are actually essentially trying to summarize all the crazy thoughts in your head. And so that's why it takes a lot of time.

And it's useful for you or I because it's rude if somebody just blabbered and sent that to you as a Slack DM. But to an agent, they don't care. They actually prefer it. So it becomes much more productive to be able to whisper your thoughts because you don't want to be shouting. And you have these microphones now that allow you to whisper very quietly and still pick up with full fidelity. So that's why we have it. And sometimes people see it and they think it's a little weird when they join the company.

But after a month or so, they can't go back.

Yeah, into whatever people use. I mean, different people use different things, but yeah, Codex or Claude or Cursor or whatever people use. And not just engineering, right? Like all the functions, if you're trying to get something done, if you're trying to describe what you want, and all these things.

Yeah, Basis builds agents to do accounting work end to end. And accounting is difficult. It's not something that is just purely text in, text out. And so it requires the ability for AIs to be able to perform lots of actions over long periods of time and actually be coherent over that period of time to get to outcomes that are good. And that's why we've always been very focused on how do you really build agents that can scale to do that work.

That's a good question. It's probably the other way around, but I do think it is actually quite interesting from an agent perspective. Accounting is interesting for a lot of reasons. It is one of, if not the largest, knowledge-work professions in the country. There are over 3 million combined accountants in the country. And what I think is so cool about accounting, actually, is that most people don't really think about accounting. They don't think, like, oh yeah, why is that even there?

Probably most listeners have never thought, like, why does it even exist? And I know we're gonna talk about agents. Maybe quickly, 30 seconds just to convince everyone how cool accounting is. If you think about the real world, so much stuff happens, economic activity, right? Like, I was just drinking a water bottle there. That bottler had to choose to buy from that factory or that supplier, or decide to open some additional store or hire a salesperson.

And these are all economic decisions that stem from understanding the real world, what's in the real world. Money moves hands, someone signs a contract, someone delivers the inventory. It's like all these events that occur. And so much of modern capitalism relies on the ability of all these actors to make decisions on these events, right? Like the CEO of that company, the IRS, obviously, to decide how much to tax, the bank to lend credit, investors, right? All these people, they care about the real world, but they can't understand it because it's gigantic and it involves all of this unstructured and difficult-to-parse information.

And accounting is actually the art of compressing all of that into something that is structured, that now people can look at and understand and make decisions. So something about accounting, you could argue in a meta way, is kind of like an intelligence over the economy because it is really a compression activity of all the information that exists. So I think it's a very cool problem to kind of think about.

Yeah. So maybe I can give my quick definition of an agent. I know probably everyone knows at this point, but I feel like that's a gotcha question I like to ask in interviews. I tend to think of an agent as an AI or some inference that occurs that has the agency to go and make decisions to do different things. And so by definition, it's a spectrum. Because you can have varying degrees of agency, right? Like, you're constrained by whatever environment you're placed in.

And I think long horizon, again, is a spectrum where you're granting the agent the agency to make decisions that allow it to be coherent for longer periods of time, right? So let's say that you were asking an agent to go and look up the weather for you. It might be an agent in the sense that it has the agency to decide what tool to call or what Google search to put in. But it doesn't need to do much work to be coherent over a period of time because you're just getting the weather.

But if you're asking an agent to, say, perform an entire feature, like implement some feature in your repo, or asking it to go and make a big Excel workbook, now suddenly it might have to operate for longer than a minute. We're talking 10 minutes, 20 minutes, 30 minutes, and potentially even much longer than that. And once you're starting to get into those scales, you start running into the fundamental limits of how LLMs work, in which I always like to say LLMs have very large working memories and, by default, no short-term or long-term memory.

And so you have to leverage these strengths of the LLM to make up for the fact that they don't have good, or actually any real, short-term or long-term memory by using harnesses and all these kinds of advancements to allow them to be coherent over a period of time. So I think once you start getting into the art of trying to get it to be coherent because you're going past the amount of working memory it has, I'd probably say that's when you're starting to get into what I'd call long horizon.

When you are really autonomous or kind of doing something over a really long horizon, say doing a tax return end-to-end, that means that you have a lot of information that is needed to do the work, and you have the tools to go and get potentially more information. So let's say, imagine you're doing a complicated 1065 and you have all of the different K-1s, W-2s, other documents, 1099s, whatever you need from the company. And then you also potentially, depending on what you're doing, might have the trial balances already.

So that tends to be what you need to actually start a tax return, or you're working with books that aren't even done yet. And the agent then has to actually go and figure out, based on all this different stuff, how it's going to tackle it and what it's going to be able to do. And that's where you start getting into some stuff about what the behavior should be that we could talk about. What does good practice look like to get to a solid set of trial balances?

What does good look like in order to properly extract out the K-1s and the K-3s so that you can be confident in the outcome? And for it to be autonomous, it means it's not going back to the user and saying, hey, is this right? Is this right? I need this. I need this. It's like, starting the job to I'm done. And I'm done does not mean I'm done, you click a button, no one looks at it.

It's actually the opposite of that. It's much closer to what you can imagine a preparer doing, or maybe like a first pass or a junior engineer or something, of, "I'm done. Here were the big decisions I made. Here were my assumptions. Here were the different things you need to look at. Let's go and review together," right? And if you think about somebody, say, in engineering, an engineer handing you a PR, nobody likes being handed a 1,000-line PR.

They're like, "It's done, I promise." It's like, you don't want to review that. But if you instead handed somebody a great stack that was properly split out and you could understand very easily, "Hey, here is exactly what this change is in this diff, and I made this big architectural assumption here, and here's why I made that change," you can optimize not just for getting the work done, but for making it easy for your reviewer to understand the decisions that you made.

And that's obviously very true in software engineering, and it's actually true in, I think, most professions, and especially accounting, which we can kind of get more into. And so, to me, that's, I think, what it means to be autonomous.

I think within the paradigm of modern-day LLMs, yes, more or less, in the sense that I would say it is actually the same, but it kind of extends out further, which I don't really remember if it was part of that paper back then, of needing to use your reasoning to regulate your own state. Like, the analogy I like to always give to people at the company when I'm learning about agents for the first time—not even just technical people, like anyone at the company, actually—is the movie Memento.

I think Memento, for those who haven't seen it, is a movie in which there's this guy who has short-term memory loss, and every day he wakes up and he knows who he is and he knows, like, he's a human. He knows some basic stuff, but he doesn't know what's happened in the last couple years. He has no idea. And for him to make progress to any particular goal, that could be something like getting groceries or getting revenge or whatever it is, he effectively needs to write stuff down for himself.

And then when he wakes up the next day, he reads his notes and essentially builds that up. That's how he builds up knowledge.

Yes.

Yes. I still think The Prestige is the best Nolan movie, but yes. So anyway, I think that, to your point, it is actually about reasoning. But I don't think they talked about it in the paper. There's reasoning in terms of what path is right to go for whatever the task is. Like, do I do tax this way or that way? For sure. But there's also reasoning about how do I make sure my next inference step has what it needs to properly interact with the world, which is easy if you're in a short horizon because you're within the context window.

But once you start getting to longer horizons where you're beyond the context window, or you're getting to context rot, you need to kind of use and sort of brute-force your reasoning to build out your environment, whether that be with subagents or compaction. We talk more about that, but that's, I think, where the reasoning becomes super important.

Yeah, I think back then, I believe, if I remember correctly, that was like GPT-4, maybe GPT-4 Turbo at the time. The models back then, for starters, if it was pre-Turbo, the context windows were very small. And so, if you're going to be coherent, you need to have at least enough stuff in your context that you can organize your own environment. So they didn't have that. And even when the context windows got larger, I don't think by the time BabyAGI came out, the models were actually good at keeping attention over, once you got past whatever it was, like 20,000 tokens, they were just not smart.

It took until maybe Claude 3 Opus for them to start getting smart at like 100,000 tokens even. And so BabyAGI didn't have that. And then also, obviously, I'm going to talk more about it, they just were not good reasoners. And so if you're not a good reasoner, then you're going to have lots of compounding errors because you're going to make one mistake that's in your kind of per-token generation. And now suddenly that just compounds, and you have no way to have the self-awareness to actually self-heal in any meaningful way.

Yeah, I would say, when I think about what were the big, like, holy shit moments for—I don't know if you can curse—but yeah, like, what were the big kind of holy shit moments for us? At least for me personally, it was probably—or at least the moments in which there was a genuine paradigm shift. There haven't been that many. I would say they were Opus 3, which I think goes underappreciated, but I think was the first model to truly be able to actually understand at long context.

Before, if you put anything in the 80,000 tokens into GPT-4 Turbo, it could not understand it. Opus 3 actually understood it, which was remarkable. I think it was that. I think it was o1. Obviously everyone talks about that. And then I think after o1, it was o3, because I think o3 helped prove that not only could you scale the amount of reasoning at inference time, but with better training, with more compute, better data, et cetera, in the post-training phase, you could make the reasoning higher quality, more efficient, and just better.

And so each incremental token that it reasoned with at inference time was higher quality, which was not obvious when O1 came out. So I think those were, to me, the big breakthroughs.

I think it's a couple of things. I think it's that because the model is able to kind of titrate the amount of compute it's throwing at any specific step in the process, you're able to—in any trajectory, there are a lot of decisions that are really hard and some that are really easy. And it's just not feasible to have some giant parameter model that's super expensive to serve that has essentially been using all of that compute at every single inference step. And so the ability to reason allowed you to titrate that.

And I think, as you pointed out, as a kind of maybe emergent behavior, if you are a really good reasoner and you can dial it up a lot—like, if you actually look at the amount of compute for an easy step versus a hard step, it's a lot different with the modern reasoning models—you could become better at self-healing because you're going to be pausing and thinking about everything and sort of using that more, that kind of thinking, versus just doing everything on instinct, which is what was happening if you're doing kind of like just per-token generation.

Yeah, so that paper came out. Back, I think, before people—I don't remember the exact history here—but essentially there were a lot of rumors back then about, if people remember, the Information article, like, "Oh, Strawberry, it solved math," or whatever it was. And so I think even before those rumors came out, there were some hints in the literature like this. And that might have been after the rumor, actually. I don't remember. But that math was a—or these kinds of problems that you could verify were maybe good ways to try to train the models to be better at different tasks.

And I think this is purely speculation because I was not in the labs, but there was this sort of question back then, and through a lot of the history of ML, around: are you going to give the reward just from whether it got the problem correct, or whether it approached it like a good mathematician would approach it? And what's interesting is that in that paper, which they published, they showed that actually, if you rewarded based on the process rather than just an outcome, you actually got better results within that sample.

But that obviously is expensive because that required mathematicians to grade the approach they took to the problem, right? Versus if you fast-forward a bit and you look at the DeepSeek R1 paper, where they effectively laid out what I think all the labs were doing at that time, or at least OpenAI was doing, in terms of RLVR, reinforcement learning from verifiable rewards, that effectively had very little process supervision and instead was essentially just saying, "Hey, did you get the outcome right?"

Yes. Okay, let me reward you, and then scaling that up, which obviously worked well.

Yeah, absolutely. I think the METR chart is somewhat inaccurate these days because it's so hard to measure.

The bar is pretty low. And I think I'm not super familiar with their dataset, but my understanding is that the sample size is kind of low. So I don't know about the specifics of that metric, but I think, from a vibes perspective, absolutely. The models were able to start being coherent for longer periods by being smarter, but now they're also being specifically trained to do that, right? So that means they're being trained on how to have good theory of mind over yourself.

Because you need to, if you're going to be outsourcing to a subagent, right? Or if you're going to be writing notes to yourself, going back to the Memento analogy, right? You need to think about, hey, I'm going to wake up tomorrow and I'm going to read these notes. What is the most information-dense way for me to write this note down so that me in the future will understand it? Well, that requires kind of a theory of mind. And so, as these models are being trained more on actually how to do that kind of work, which is very, very non-human—humans don't have to, because we're great at that, actually.

So we don't have to write for ourselves. Our brain does that for us. You're starting to see it get even farther. I think we're still actually quite early at that, six hours or so and whatnot.

It's kind of interesting. I think people get this a little bit wrong. I'm speaking with a little bit of speculation because I don't actually know exactly. But coding, yes, is verifiable in the sense that I can know whether the code passes or not. And so I could train on that. Did you get the problem right? Did you get it wrong? Et cetera. Which is useful. But the models being great at coding is partially that, but it's partially, I think, a couple other facets of coding.

So number one is that coding is verifiable at runtime. I think that's a very important point because, at the end of the day, an agent needs to get signal as to how it's performing. And so imagine if you were an engineer and you wrote code and you could never run it. Even if you were the best engineer in the world, you're going to have a syntax error in which the thing is wrong at some point. And so I think the fact that coding is so easily verifiable at runtime, or at least some parts of it are verifiable at runtime, is obviously very valuable for it being there.

And it's cheap to do, right? It's easy to do within the environment, and it's super text-based. So it's available anywhere. You can just, on your local computer, run it, right? And so I think those aspects of coding actually are a lot of the reasons the agents are getting so good at it. I think one more piece of it is that, and I think we probably saw this, if you think about how good agents were at coding maybe a year ago, a year and a half ago, they could go and implement the thing you told them, but they didn't have the level of taste or level of what is good software.

Because even if you train them with verifiable rewards on, like, hey, did this unit test pass? You could pass all the tests in the world. Doesn't mean that you set up the app correctly. Doesn't mean your database was built well. Doesn't mean that you split out the files properly. At the end of the day, coding is subjective. It's an art, and you're not going to solve an art through verifiable rewards. And so I actually think there's a large part of this, which is the amount of training data and the quality of training data that the models that the labs have clearly for coding has gotten quite good.

And they've focused a lot on making sure it's very good so that they're training on high-quality code. So I think the other part is just that maybe because the labs are obviously full of engineers, it's more top of mind, and obviously it's part of their strategy. And so it is more top of mind for them to ensure that both their pre-training and post-training sets have lots of high-quality code. And that's what makes the agents great at not just writing code, but now starting to become good at actually engineering.

So I think it's all those things together. I think the verifiable rewards are maybe just, like, the beginning that allows that stuff to happen. But I think the other stuff matters just as much, if not more.

Yeah. So I think agents struggle for a lot of reasons. I think, one, they struggle because they don't necessarily know what good looks like. I think they struggle because it may not be easy to verify yourself at runtime, as we were talking about with coding. I think another part is that—and this is maybe not an agent struggle, but maybe it's a UX thing—is that for coding, engineers are just very in the weeds of it. And so there's kind of a difference where, if you were running a long-horizon agent for coding, if the engineer was not engineering and in the weeds of the code, if instead they were more abstracted away, your maybe level of quality and how you make decisions, probably you maybe need a higher bar than you would for coding.

Coders are okay with lower bars. That's been true forever. And so I think all these things add up in making it. And even now with coding, the agents are not yet—they're not human-level at being coherent over long periods of time. That's obvious because they can't code like a junior engineer on a project for two weeks. So that's worse than a human, even to start.

So the good news is there are some things that can compile. Not all, but you can obviously test to see—

Yeah. Well, I think the answer for this is you look at what humans do and how. And one nice thing about accounting, which is true in some other professions as well, is that accounting is a profession in which you really try and have to be correct. And so, because of that, it's something in which there's so much judgment and process involved. One of the sayings we have, and I say this on the design side a bit, is that humans are already used to working with non-deterministic systems.

It's just the systems are normally their coworkers, not their computers. Humans coordinate together to solve a problem. And once you realize that, it's kind of like agent design. And so I think if you need to think about what is good agent design and what can be verified, you should look at how the humans organize. And if you look at how humans do tax returns, you have steps of verification, you have independent review, you have things that can be deterministically verified.

So you can say, hey, obviously, do the TBs add up to zero? Right? That's a very obvious check. Things like, does the Excel not have errors, is an obvious thing, right? And so there's lots of things like that that to a human are obvious, but you need to make sure are properly encoded. And there's other things that are maybe not as deterministically verifiable, but would be obvious to an accountant to look at it: oh, this is wrong. Like, oh, you deleted this tab in this Excel, or you didn't cite this thing, or whatever it is.

And so you can start to build verifiers effectively from these things that are not deterministically verifiable. But if an accountant would look at it, it'd be obvious. And so you can start to think about judges or other forms of verification to get that signal both in your evals and at runtime.

Let's say, just for the sake of argument here, you had not just synthetic, but every real tax return across the country, which you actually couldn't do for privacy reasons. Let's say you did have that, and you were just like, okay, let me use this as a way to get data that you're not training on, but data that allows you to verify how well the agents are doing. The order of magnitude there is tiny compared to standing up math problems synthetically, and you're generating whatever, hundreds of thousands, millions, et cetera.

And so even if you had all the data in the world, you would not be able to scale it. So you need to think about how do you synthetically generate it? And if you think about how to synthetically generate it, that's really hard because you're not just synthetically generating text, you're synthetically generating artifacts that have to be real and diverse. And so now you get into all the same problems about data diversity and all the different things that you need. And I have no doubt that that problem will be solved over time, but it's not solved today.

And so there's this kind of gap between maybe what is possible from a data generation perspective and what latent capabilities the models actually have, which is maybe where you get to some of the stuff we were talking about in the thread.

Yeah, I mean, that's the other thing, is the length can be very long depending on what you're doing. Performing a 1065 can take a human, like, 20-plus hours easily of actual work. I don't mean, like, it took them a day. I mean literal sitting-down work. And it could actually take much longer for very complicated returns. And so there's just no way that you, even if you had all the data in the world, which you can't have, would be able to have the feedback loop needed to do whatever improvement loop you want to do to get the agents to be really good.

Yeah, I mean, to give you a sense, you could have, for example, 500, 1,000 documents easily. You need to think about how to map those documents against each other, understand what matters. You have to potentially perform lots of different research per different document. You have to potentially think about what they all mean. You have to compile them all into, at least today, workbooks of certain types, which are like big Excels. There's a lot that you have to do. So you're talking steps definitely in the few thousands easily, if you're thinking about inference steps.

And depending on how you build your system, if you start to spawn out subagents for different things, which you kind of have to do, you're increasing that potentially. And that's before you think about other test-time compute methods. For example, one thing you could do is say, well, imagine there's a tax question you have to solve and it's insanely difficult. Only an expert would be able to solve this specific tax question. Maybe there, instead of the agent spawning a subagent, you're spawning five and you let them vote.

And so there's just lots of other things you can do to throw more compute at the problem. And so based on what you're doing, the amount of steps ends up being quite large.

Yeah, the problem with relying on outcomes: so, if you have a multi-thousand-step trajectory, or even honestly one that's like an hour long, you will have evals that will say what good looks like. And it could be entirely verifiable, like, do the numbers match? And it could be parts that have an LLM as a judge, like there's some rubric, et cetera. And let's say you have 100 evals. Great. They all pass and look good. Are you confident that that now generalizes to the real world, to production?

And our answer has been no. You actually cannot be confident of that. And so you shouldn't only rely on that, in the same way that if an engineer came to you and said, hey, all my tests pass—end-to-end, unit tests, et cetera—does that mean that they architected the database properly? Not necessarily. That doesn't actually tell you that. There are a lot of ways to pass, quote-unquote, outcomes without having done the process properly. And so, what we have found is that there are, especially going back to my point earlier on analogizing to human organizations, lots of learnings from how humans do work.

And so I think it is a mistake to throw out those learnings and say, bitter lesson, throw out those learnings. We're just going to have the agents at runtime develop an entirely new way to do a tax return that is, because bitter lesson, yada yada, better than the hundreds of years of human history that have gone into learning about the right process. Maybe that will one day be true. I'm not saying that it will never be true. And I do think it is possible, going back to the data bottlenecks that we talked about earlier.

And I think if you throw enough data and enough compute at an outcome-based process, you can eventually get there, but not soon. And so if you're not going to get there soon and you have this latent potential in the models, what you really need to do is think about, let's take the learnings from how humans do things, from what good process looks like. You can't copy that exactly because there is a lot of thinking you have to do about good agent design. It's not like the models out of the box today are incredible at being coherent on long horizons.

There's actually a lot of work there about behaviors, for sure. And you can instead put in place certain evals or potentially, in the future, reward functions that look at whether it followed the process properly or not. And maybe the example I mentioned in the thread, which I'll just say for the audience, is imagine you're doing something as basic as tax research. If you ask some tax question, the agent could definitely get it right. They could know it from their pre-training knowledge.

They could read some blog and get it correct. But a real accountant would not trust that. They would want you to cite the primary source. So even if you got it right 100 out of 100 times, if a person is just getting it right because they're going to Wikipedia, the accounting firm wouldn't hire them, and so they shouldn't hire us either. And so we think it's really important that, no, actually, our agents are not learning from the pre-training knowledge or reading from a blog.

They are going to the actual code and verifying the information with the primary source, which is what you would ideally want a real tax accountant to do as well.

Yeah. So the original idea, actually, my co-founder Matt came up with the idea literally about two years ago. We were talking about AutoAGI, even back when we had agents that weren't fully, I guess, agentic as you think of them today, and they had restricted choices. Even back then, you still wanted to think about, okay, what kind of choice do you want it to make at this fork in the road? And so Matt, we actually used to call it internally meta behaviors.

Because the idea was that you're defining a behavior, but it's at a meta level because it's across all the behaviors the agent will have in all the different trajectories. And so the idea was that instead of trying to write the prompt, you have to first align on what the meta behavior is. And so that was actually the first purpose of this concept. I swear to God, literally two years ago. And over time, that kind of evolved, and we ended up calling it behaviors just because it's a bit simpler.

And the idea is that you have a Markdown file in which you write down, how do you want an agent to behave? Simple as that. It could be at varying degrees of granularity. So let's say you have something that's very specific, like you need to go look at the primary sources. Maybe you want to be more specific. Maybe you're like, no, you should always go look at specifically the IRS website. Or another example could be, imagine you're making PowerPoints.

And a behavior is, well, before you return the PowerPoint, you should render it as an image so you know if you made any formatting issues. And so you put that in a Markdown file, and ideally that is a Markdown file that can be self-contained. One that humans can look at and align on: yes, these are the behaviors that we want, in part because behaviors are actually subjective exercises. We can talk more about that. It's just as much of a product thing as it is an intelligence thing.

And then it's something that a judge can look at, where the judge can look at a trajectory and say, hey, did the agent exhibit this behavior, or did the condition occur that it would need to exhibit this behavior? And if so, did it actually exhibit that behavior? And then you can grade it accordingly.

Humans.

That's a good question. It depends a bit. I think it is a combined effort between accountants and ML researchers at the applied level, because you're not just saying, for example, hey, the behavior is you should go to the website. You might be saying, hey, the behavior is that you should be, for this type of research, spawning a subagent with full history, because you need to build up that context. There's a lot of agent machinery that comes into play.

And so you kind of have this thing where there is what a good process looks like for a human, but then you need to translate it into agent language and then decide. And there's a combination there where you're also talking about what are the agent mechanics. And so I'd say it's a dual effort between accountants and ML researchers. And we have a whole team, actually. It's called Accounting Product Operations, where it's accountants who essentially work very closely with the research teams to build out rubrics, both outcome-based rubrics and then also behaviors.

No, that's a very good point, which is exactly why the behaviors are not actually shown to the agent. So the behavior could say, hey, it should go to the IRS website. That doesn't mean the agent is told to go to the IRS website, right? It might be, it might not be. It sort of depends. But the point is that the way I think about it is, as the agent engineer, the systems engineer, you are making the decision as to how specific do you want to be with the situation.

Obviously, you prefer to be less specific, right? I think, as people who are good at building agents know, it's much better at the margins to be able to give principles and the whys and more context and let them figure it out. And so with behaviors, you want to actually not define every possible thing that can happen, but just say, no, no, no, we know that. For example, let's say you're making PowerPoints. Taking a picture of the PowerPoint before you give it to the user is going to catch issues.

We know that for a fact, right? And so we, as the agent engineers, are going to take a stand and say, out of all the different things you, as the agent, are going to do, this is the thing I'm going to grade you on. And then maybe you're being a little bit more specific there. If you want, you can be more specific, like use this exact tool, but ideally you don't have to. It depends on kind of how your system is built.

So I think your level of specificity depends on maybe what specific outcome you're sort of trying to drive and how much that outcome generalizes to the universe of situations. If you're producing PowerPoints, of course taking a picture makes it better. But let's say instead your agent actually is a super-fast agent. Well, taking a picture takes time. So maybe for a super-fast agent, you don't want it to take a picture of the PowerPoint because now you went from it taking 30 seconds to taking one minute.

But if you have an async agent that's taking 20 minutes, you went from 20 minutes to 20 minutes, 30 seconds. Do it every time. And so this goes back to my point on the product aspect, where it's not just an intelligence thing. It is a subjective exercise about how do you want the agents to behave in production broadly. And actually, I think it's why it's so critical to product building.

Yeah, good question. Today, at least, we're not actively rewarding the underlying model. So we're not currently post-training our own models by rewarding them on this process. I do think that's a very interesting area of research. We can talk about that later, but that is not actively what we're doing right now. It is something we're actually researching separately, but that's more in the future. And so if you maybe take a step back, it's useful to sort of analogize the work of agent building and context engineering to the work of training a model directly.

I think when people think about context, people say prompts, context. I think the mental model people have usually is like, oh, I wrote some English to tell the model what to do. I think it's the wrong mental framework. I think the framework I like is thinking about it as training data, except you are just training the model at runtime. It is training data. And because the model's learning at inference time, the total amount of training data is far lower, right?

Like, the total amount of context in your system that the agent would progressively learn or discover throughout its trajectory is obviously orders of magnitude lower than the amount of data that you're post-training a model on. And so what you're doing is you're essentially taking this data and you're trying to ensure that it is of the highest quality to get the agent to exhibit the behaviors that you want it to exhibit. And obviously the data is only one part of it. There's the harness.

What are the tools? What are the capabilities, the environment? There's all that stuff, obviously—which model you're using, things like that. And so what's different here is that we're taking this signal and using it to improve the entire agent system, which requires far less data scale than if we were trying to take the signal and more literally reward it in an RL capacity to the underlying model.

Yeah, I think that's true not just for behavior evals, but also for outcome-based evals in general. It's a great question. I wish we had more time to go deeper on this. The reality is we just don't have the time or resources to spend a huge amount of effort perfecting every judge. But I'll give you the high level. Generally, what you do is you need to build an intuition for if the judge's taste is correct. And I do think an interesting area of potential post-training research is on judges and potentially taste there.

Today, you need to set up the judge so that it has the information it needs to make the decision, and it has the right kind of framework to do that, and that it has the data and mentality to do that. So I'll give you an example with behaviors that you can imagine being pretty complicated. So right now, at least, the behavior judging is relatively expensive because it's a pretty advanced judge in that it is also an agent. It's not a judge in the traditional sense.

It's literally an agent because it has to look at the trajectory. So it's quite expensive. You can imagine in the future—and I think folks like Braintrust and others are starting to think about this—how can you label trajectories better so you can potentially more easily filter the trajectory to only the potentially relevant parts to give to a judge instead of having it look at the whole trajectory in some ways? But if you have a trajectory, especially if you have a long-horizon one that might have a lot of subagents, you as the judge need to think about, well, where do I go in this trajectory?

How do I understand it? You probably want to have a map of it in some form. Who am I even judging? Imagine you have an agent system with depth of seven. You could have literally seven layers of subagents. Am I judging whether the root behaved properly? Am I judging one of these other agents? And so you need to actually properly prompt and potentially tune the judge so it has a good understanding of where to go judge, understanding the behavior itself—like, what is the condition?—and understanding how to judge whether the behavior occurred.

It's a good question. Well, I think, for starters, and again, one thing I want to make sure is clear, I think, to everyone listening, is that writing a behavior down is expensive because it is something where you are now keeping state, right? You need to keep it up to date. If the way you are making the model or the agent exhibit this behavior is through context, now you have potentially duplicate state, depending on how you're thinking about it. And because of that maintenance burden, you ideally want to have as few behaviors as possible.

So it's not that you look at what it takes to do a tax return and you say, hey, let's write all the best practices down and see if it's doing it. It's that you take the couple that you think are most important, that have the largest amount of generalization to production and are the most impactful, and you care about those, not everything. So I think that that's one important part. So ideally, if you do it properly, you still have room for the Move 37s, in theory.

But I think there's another part here, which is: what are you selling to someone? If you are trying to—just like if you were to hire an engineer at your job, at the company, you have a process, and the process is like, I'm just making this up, but you write a quick architecture diagram, and you chat with the CTO, and you get it approved, and you make a PR, and you split up the PR in like 10 different smaller PRs into a stack.

And then you merge it in, and you make sure you have your end-to-end tests, and you deploy it. Like, that's the process. Imagine an engineer came to you and they're like, hey, here's my 1,000-line PR. I'm going to merge into production right now. What if it's better than what you—what if it's a Move 37? It could be better than what the CTO would have come up with, but that doesn't mean it's good. Like, that doesn't—good in the sense that doesn't mean the CTO or the company is happy about that result just because it's better, right?

Because at the end of the day, the reason you perform work is not because any individual unit of work is incredible, but because you can scale it to a company, to a system, whatever it is, to an organization. And so the thing that someone is buying from us is not, this will be the best ever tax return. They're buying the confidence that—

Yeah, the best-looking one. Or like, it Move 37s the TVs over here or something, right? They're buying that it's going to be consistent and reliable and something that they can trust, that actually will—they will, just like with a human, learn to trust more and more and then grant it more agency over time. Right. And I think that level of trust and reliability, that's what you need to deploy into the real world. You don't need the Move 37s.

You need that maybe at the Olympiad math competitions, but not doing work in the real economy. Yeah.

Because I do think one thing that's really important when you're building agents, but definitely building a company around it, is you shouldn't be that surprised. You should have a model of the world, and as things change, you should update your model. But to be successful, you can't just update your model all the time. You need to be right a little bit. And I do think that if you really internalize some concepts about this, right, that, like, you now have this—forget even about the internals for a second, forget about this LLM—you have this magic box, or this alien, I like to call it sometimes.

And you could send in huge amounts of data into this alien, and it will be able to reason and learn at inference time within that magic box and then come back to you with output that, now that tool calling obviously works and whatnot, you can plug into the rest of the system. That's kind of all you really need to know. And I think once you really appreciate what that means and then you take it to its logical conclusion, a lot of stuff starts to fall out of that because you start to understand.

It's like, okay, well, wait, like, if I have this magic box that can do this, does that mean that it could decide to call another magic box? Does that mean that it could potentially string together, like, multiple of them in a row? Right. Does that mean it could leverage—obviously, at that moment, it has some—it's a magic box, but it has some state. We know this. It has an activation state. That's how the caches work. So there's some activation state that, by definition, is going to be biased to that current trajectory.

And so maybe for review, you want an uncorrelated trajectory, right? Where it's like a new box and it's just as smart. And I think if you build these kind of LLM intuitions and you combine them with maybe basic principles of organizational design and management, I think you start to get to maybe what is, like, the frontier of agent building.

Yeah, I think it's none of that, actually. I think reading Twitter and whatnot, just language understanding, is good, but I actually think a lot of people overindex on that. I think a lot of people think, like, "Oh yeah, I saw this tweet, I saw this, like, it's the next cool thing." I think the problem is, without a fundamental grounding in how things work and what is possible, it's easy to feel like things are moving around a lot when they're actually not.

Things have really not changed since o3. I would say almost everything since o3 has been relatively on—I don't want to say on trend in that I knew this exact trend, but I would say it's all within the same paradigm. Nothing paradigm-shifting has changed since o3. And I think the best way to understand it and learn about it is to just use them in your own work a lot. I think especially in coding and just trying to understand things.

A good mental model is, let's say I tried to have an agent implement some feature for me and it didn't do it the way I wanted to. Why? What is actually the limiting factor? It's kind of like the famous Elon mindset. It's like, okay, you go to the main limiting factor and you figure that out. I think if you apply a similar mindset to agents and you understand, why could it not automate this? Was it actually not smart enough? Probably no.

They're pretty smart. They've been pretty smart for a while. And so if you apply that mindset to your own work, I find that is quite useful for building intuition. And I see that actually with people I interview. A lot of the people with the best agent intuition actually—yes, a lot of people come from ML backgrounds, but people who don't, a lot of them are ones who are just really good at automating their own work, really good at thinking about it, really good at understanding what is the system to build.

I did a talk at Data Driven about ontologies, whatever, a year and a half ago or something. And I think there are a lot of people who think about ontologies in their own data, in their own repo. And those people who are actively thinking not just, how do I prompt a model, but how do I build a system? They start to build really good intuitions.

If we go back to the idea of behaviors, right, the idea is that you can actually write down in Markdown what is a— it is actually both a spec and a rubric. We call these specs, and there were some people who asked, isn't this a rubric? And it is. It's both. The reason it's both is because it is not just used to grade or potentially reward the agent. It's also used to align the humans. I think that is an underrated point, in that how you want the agent to behave, as we talked about earlier, is actually a subjective question.

And so internally, you need to build processes to all agree on, hey, this is the product, right? How do you want the agent to behave? Going back to the example about the fast PowerPoint verification. And so you want a standard to kind of write that down. And the project kind of came about because I was actually having coffee with the CEO of Braintrust, Ankur. And I forget why, honestly, but I started telling him, I was explaining this concept to him.

I was talking about this because we were doing this internally, and I thought it was very cool. And he got pretty excited about it. And one thing that I had internally that I'm trying to think about is we, I think, do a lot of really cutting-edge work, but it's not something we talk about much because, to be honest, we're working all the time.

It was prompt refactors.

There's no 9-9-6. For Matt and me, it's 24/7. For the rest of the company, people work hard, but it's definitely not a 9-9-6. And so we wanted to talk about it more and just share what we're doing. And we don't have a lot of resources to blast out to people. And so we were talking. I was like, well, I actually think this could be really useful for Braintrust and honestly the whole industry, because if you have a standard that could be something that people define, it can get automatically slurped up into observability platforms, monitoring platforms.

And for people maybe who are less advanced, it could also have out-of-the-box judges or ways to define, hey, here are the behaviors, and you don't have to configure your own judge. You can actually get it to judge it for you and see the results. And so he got pretty excited about that. And so that's where the collaboration came from. So that's what the open-source repo has. It has a couple small examples. It has an example judge that you can use.

It has examples of actually written behaviors that you can leverage and sort of build your own. And I think it is useful to think about how to adopt this standard, but I think it is also, maybe more importantly, thinking about how to adopt the mindset of not thinking that an agent operating over 10 hours is a black box. It's not. It has a lot of data, and you're probably doing a disservice to your customers if you don't understand how it's going about the work.

Yeah, it's a great question. I don't actually know. I think the coolest thing would be for people to contribute ideas to it. I think there's a lot of work left to do. I think it's just the beginning. I mentioned a couple things earlier, but there's so much to do around, one, how you make good judges. Two, how do you properly label and dissect trajectories to make it easier for judges to understand? Because at the current level of expense, you couldn't run this in all of production, for example, because you're running judges on every single trajectory.

But there's a lot that can be done, I think, with building out the work that sits on top of the behaviors. And I think also just seeing, we purposely tried to make the standard relatively flexible, similar to skills, where it is just Markdown. There's not an overfit, hey, you need to have these exact five words. You can make it very broad and you can make it very specific, as long as it is still self-contained to the point that a judge could look at the behavior and actually know, was the condition for it to be exhibited met?

And if so, did it get exhibited or did it not get exhibited? And as long as it has that, there's a lot of leeway there. And so I think we wanted that to be flexible.

Yeah, they're super important. A lot of people, when they think about agents, their mental model always goes to coding agents because that's everyone's experience with, at least the people who probably listen to this podcast. People think about coding agents a lot. Coding agents are interesting because they obviously have a harness that they get shipped in. They have certain tools, they have certain behaviors encoded in them in the context, right? Codex, by the way, is open source.

I highly recommend people go look at the open-source repo, but they don't control their runtime training data, right? Because their runtime training data, which goes back to my analogy earlier, which is your context, is actually the repo they're working on. And so you could have Codex, the same agent, quote-unquote, on one codebase perform somewhat well, and then another codebase perform spectacularly because that codebase has much better runtime training data, right? It has potentially good skills or good context about how to operate in the codebase because there aren't contradictions and confusions, whatever it is.

And so the ontology of your codebase, it always mattered for engineers. It matters just as much, if not more, for really good agents over time. That actually, I think, goes up another level if you're thinking about non-coding agents, because in coding, you don't own the runtime training data. In non-coding, you do, right? Most of the data, quote-unquote, that an agent sees when it's a Basis agent, Basis owns, right? It's training data that we have to ensure works really well.

And again, when I say training data, I mean, like, effectively handwritten context, right? Or things that are part of your broader progressive disclosure. Some could be handwritten, some could not be, whatever it is. And because the agent is always starting from scratch, designing that ontology in a way that is ergonomic for the agent is super key to building something that's long-running. That's both, like, the data that is maybe static, like skills and whatnot that all the agents have. But also, once you get to really long horizons, if you're talking about a stateful agent that's maybe operating over days to months, now suddenly you have an ontology of, going back to the Memento example, information the agent's left for itself, which, if you're operating for maybe a couple hours, could be a couple notes.

If you're operating for months, you're talking folders, right? And you have so much knowledge and context that describes the lived experiences of the agent that suddenly this new agent—well, new—that is standing up with effectively very compacted context has to get it back into the state of mind of its entire lived experience. And so the ontology designed to make it easy for it to do that, and the behaviors you encode that properly ensure it's doing that well, are sort of the key to making it work really well.

Yeah, there are different formats. I think the simplest way to think about it is honestly just a file system in which you have some structure. Obviously, most people do virtual file systems, and so you have a lot of flexibility there. So there could be other types of metadata associated with the files and the folders, right? There could be connectors and nodes in some graph DB if you wanted. Obviously, there could also be embeddings.

There's so many different sources of data that you can get to help enrich. And now, by the way, as models are getting cheaper and cheaper, more and more of that actually can just be done using inference instead of using things like graphs or things like embeddings. If LLM costs are free, then suddenly you could run LLMs across your entire ontology and summarize stuff for the agent, or things like that. And so that's kind of how I would think about it. And maybe also one more piece.

Ontology doesn't just mean the structure of the folders. An ontology, traditionally, is also like the language, right? What are the objects and the concepts? Because at the end of the day, if you're being trained at runtime, you need to ensure you're not confusing concepts together and that things kind of generally make sense. And so that's what I mean by defining the world, right? You're defining what the agent can expect to see as it goes and explores the kind of world around it.

Yeah, that's a good question. So, just to quickly separate, when I was talking about the ontologies and the historical, I was referring more to inside of the Basis product and the production product. In terms of internal use of agents, let's say coding agents or maybe other internal agents that we might make, there it's kind of interesting because, depending on what you've set up, your environment might be less controlled by your ontology. Because, to your point, in the real world, you have to go and access Linear and Gong and Pylon and all these different things.

And so I think one of the keys for internal agents is having a very keen understanding of what is canonical versus what is not canonical. And so, just like if I'm a human who joins an organization, I could go and read all the Gongs, but what is our sales strategy today? If you watch the Gongs from two years ago, you'll get a lot of context, but you won't know what your current sales strategy is. There must be some canonical piece of documentation. In practice, a lot of times humans learn this by just talking to people, and you kind of learn stuff.

But with agents, it's hard to get that. And more importantly, if you want real organizational intelligence, you don't want an agent hearing one thing from one person, another thing from another person, or having different written records of, like, what is the current sales pitch? Or how do we make our decks? Or how do we make our emails? You need one canonical source. And that's why I think for true agent-native companies, especially in the future, today I think it's still quite early, but especially in the future, having a clear understanding of what your company canon is and organizing that in an ontology that makes sense, and ensuring that that is kept up to date, just like code in some form, I think ends up becoming one of the most important parts of a human inside of a company.

Yeah, we need a lot of them. So if you're listening and you want to join, please hit me up. Great question. We're still figuring that out. It's not easy. Here's what I know: I think one skill that really matters is good systems thinking. And where does good systems thinking come from? It comes from people who have had to think about some abstraction, some system, something, and design it in such a way that it performs in a plethora of situations.

So obviously, if you're a really good engineer, engineering is systems thinking. Now, I think the majority of engineering historically has not been really systems-thinking-based. It's been a little bit more execution-oriented. But if you think about the hardest engineering, like, hey, I'm trying to design what the systems are going to look like, or I'm trying to create the right abstraction, that really is high on systems thinking. But it's not the only profession that's like that. I think law actually is kind of like that in many ways.

I think maybe the Founding Fathers would have been really good context engineers or agent managers because you had to write a piece of English that was going to be interpreted at runtime, literally millions of times by lawyers and judges and whatnot. And so if you're writing a law, and I don't mean some politician, but if you're actually trying to write a law and trying to write it well, you're trying to somehow write something in English that will abstract at just the right level across the universe of situations.

And you have to have theory of mind over the judicial system to think about how they'll interpret it. It's funny, you'll look in certain airports and sometimes they'll have these signs. It's like, don't bring a gun, don't bring a sword, don't bring blah, blah, blah. And it's like you can list out 30 things, to your point on brittle rules. And so, same thing: you can list out like 50 brittle rules, or you can write the right abstraction that somehow covers it just perfectly.

So I think anything where you need to think in abstractions in that way, I think is good practice for being a good systems engineer. I think people who've had to manage the most complex Excel models in the world, honestly, not that dissimilar either. So I think there's a lot of potential backgrounds for it.

No, of course. The DI team is awesome. I think, to this day, I need to go Google it, but we definitely came up with the term Deployed Intelligence. I know because if you Google it, we are the first company that comes up. It's a cool name. I don't know if other people have taken on the name. I don't think it's actually caught on yet. But the idea is, it's not FDEs, so it's not engineers who are coming in and building something custom for you.

It's also not these agent PMs you see now at some companies that I won't name, where these PMs, quote-unquote, are coming in and building an agent for you using an agent builder. It's actually neither of those. It's people who have a deep empathy and understanding of the profession, of what it takes to put in place good process, and what it takes to be successful when suddenly you can start to offload certain things to agents.

And so the DI team's job is to come work with our accounting firms to help them transition into this new era. We're giving them magic. But if we don't teach them how to leverage the magic, not just in the day-to-day, but how it changes the nature of the firm, how does it change what kind of business they can take on, who they hire, how can they scale to be that firm of the future that everyone wants to be?

That's what the DI team works really closely with people to do because it wouldn't be fair to ask them to go and learn that themselves or do that themselves. Instead, they bring a lot of the knowledge about their firm, about how things have worked, obviously their people. And we can combine that with our knowledge of how to deploy agents. And I think together get to something where it can be a really frontier accounting firm.

Yeah, we actually internally, in the thread I talked about some of our research directions. One of our big research directions is: how do you actually close the loop, as we like to call it, going from, hey, an agent made a mistake or is not performing well or whatever, to we've gone and improved the system to do that? And I think that closing the loop is going to happen pretty fast. I don't know about the entire loop being closed, but I think you'll be relatively close by end of year.

I think that as agents are getting better theory of mind over themselves and therefore other agents, you have two things happening. One is, at runtime, they're being better at orchestrating subagents and also regulating their own environment for themselves. But it also means they're becoming better context engineers, right? They're becoming better harness engineers. Right now, they are far, far, far worse at engineering agent systems than they are at engineering most software. Far worse. Because by definition, that kind of work, which is so novel, has not seen a large amount in their training data.

And so they have very bad intuition. Actually, I think a lot of the mistakes a lot of agent builders make is they have this weird intuition that slop in your context or your agent is somehow more acceptable than slop in your code. And you'll see people freaking out over a code file that isn't abstracted properly, and yet their context is, like, total shit, which is hilarious because the context actually affects the performance at runtime. The organization of the code does not affect the performance at runtime.

Time. And so I think, obviously because of built-up behaviors, a lot of engineers treat the code as more precious than the English, when actually the English is more precious because the English affects the performance. The code does not affect the performance, right? Assuming the logic's the same, it does not affect the performance. And so I think as agents get better and they get better at this type of engineering of how do you build agent systems, you'll start to actually be able to close the loop.

Because in order to close the loop, you need to take the signal that you are getting, and you need a lot of signal. One of the pieces of behavior, one of the points of behaviors, is a way to get more signal. Whereas if you only have outcome-based evals, your signal is pretty sparse. So how can you take all of that signal and actually now use it to improve your system? You proliferate it throughout the system, and that could be done by an agent that is, like, updating the context, changing the nature of the tool, updating the harness, et cetera.

And I think you'll probably, in a way that is generalizable, that doesn't overfit—you can do it today if you wanted to overfit to some signal—but if you wanted to do it in a way that generalizes, the agents need to get a bit better still, and there's some more work to do there.

Yeah, it's a great question. The way I think about it is that the hard part of reinforcement learning is deciding what your reward function is and then deciding how you are going to allocate that reward over whatever occurred. Right? So that is the active research that we are doing, because that's what behavior is—a source of signal, right, that you can use to craft into a reward function. Same with some of the other production monitoring work and some of the evals that you build.

And so deciding how to build good signal and doing the research and understanding what it means, especially in a non-perfectly verifiable domain, that's the work we're doing. Whether you take that signal and then proliferate it through the weights, through formal RL, or you proliferate it through the harness, through whatever you want to call it, like informal RL or harness engineering, I think is a separate question from the development of the signal. But I think the development of the signal is really the hard part, and that's the part that we're really focused on.

And today we don't go directly into the weights. And primarily, the reason we don't do that is because a lot of the advancements the models are having when it comes to orchestrating themselves yield far more performance gains than the benefits you would have of updating models directly. I think that might asymptote—we'll see. But I think that's one of the reasons that you don't go to the weights yet. I do think that is a very interesting research direction as well that we'll start pursuing.

And so for us, at our level, at the kind of applied application level, that's the kind of research that we're doing. But I suspect that in order for the models to actually do real work in the economy, that's the only way you can get there. I don't believe that if you were to train a model and scale up the amount of pre-training compute and scale up the amount of post-training from perfectly verifiable rewards, that suddenly will output a model that will do a tax return reliably.

It might do one that's really good. But the question is not, does it do really good? It's, does it do it at the level of quality, reliability, scalability that would be expected from somebody operating with it?

Oh, I assume it will be swallowed up. Yeah, I'm not worried about it. That is absolutely the future, no question. And in fact, if you look at my thread, as a hint to this, I think I said the behaviors don't get shown to the agents yet. And the reason is because if you are truly bitter-lesson-pilled, then in the future, this whole idea of trying to context engineer, it will just go away, and you'll just specify, hey, I want these behaviors.

It'll just work. And so I definitely think it'll get swallowed up, no question. I think that how long it'll take to get swallowed up, I don't know exactly. I think it's probably sub-five years. I don't think it's sub-two years. I think it's probably sub-five years. And so for us, we're in hyperscale mode. We can't wait for the bitter lesson to arrive to perform tax returns accurately. And so that's why I think that'll be one of the keys to doing it.

I do think that—I don't know, this is now total speculation—but I do suspect that maybe some types of process rewarding for this type of work might end up being pretty important for optimizing the compute that the labs even use over time. Because I don't know if you want to move 37—like, every single profession in the economy. We have a lot of learnings already, and there's kind of no reason to do that, would be my guess.

So generally, and if anyone's currently trying to found a company, I recommend thinking this way: technical moats are not real moats. There's no portion of Basis's long-term terminal value that stems from some secret RL trick we found that nobody else found. So that doesn't matter. What matters is that right now we are obviously very good at building long-horizon agents that can be reliable and deployed in production, and we'll continue to be the best at that. And that allows us to win market share and get deeply embedded.

And that's why we move really fast, because the work we're trying to do is to go and proliferate maybe before you get to AGI, whatever you want to call that. So I think that most of the moats that will exist will be business moats. That is true in the AGI era. I would argue that's also been true in the pre-AGI era. I don't think that Salesforce can write a better SQL query than I can. The moat that Salesforce has is not related to their technology.

Right? It's related to their business position. It's the powers. It's the workflows that they own. It's all these different things that come with being embedded in. And that's what matters, not the technology. The technology is a temporary dislodgement that allows someone like us, who obviously didn't exist three and a half years ago, to now suddenly be able to do all this kind of work.

I do think, though, for a long time, maybe to your point, that it's not like it'll be a total commodity. Because just like you could have a bunch of genius humans, doesn't mean that all the genius humans are equivalent in being able to do a tax return. Because at the end of the day, there are certain things that you care about for that that are subjective. And I do think that building up the competency of the work we're trying to do is actually really important for us to be able to deliver really good quality for, I think, a foreseeable future.

I think maybe the biggest lesson I would say is it's easy to look at the world and how fast things are changing and say, like, oh, things are just one thing here, one thing here. It's like ADHD on Twitter. It's crazy. The Chinese labs are releasing something every day, and feel powerless to make first-principles decisions. Whereas I actually think if you treat the new world as an underlying paradigm shift, in the same way that a shift to cloud or something is some underlying paradigm shift, and you try to extrapolate, okay, you have these things, what does it mean if the intelligence became X better or if it didn't?

I think you'll build a lot more coherent systems and make a lot better strategic bets, both at a technical and at a business level. Things are changing, but it's not like the paradigm is changing, at least not that dramatically. And so I think really understanding it is very, very key for being able to build in this world.

Thank you so much for having me.