The Agent Harness: Building Secure Sandboxes for Autonomous AI Workloads

Great to be here.

Well, when I think about agents, I think of them as digital knowledge workers. And to do anything as a knowledge worker, you do need a computer, or I should say, anything sophisticated. So you and I can be in a conversation and we can get something done, but we usually need some sort of tools. In our world, it's usually a computer to get higher productivity and to be able to do things. And so I think of that through the same lens. So for me, agents are literally just digital knowledge workers.

Absolutely. Well, the term sandbox first comes from isolation, so making sure that there's a secure place for, in this case, an agent to run and do things. But on the other side, it is essentially a computer. It's a full-on computer that an agent has the ability to install tools, access the web, run scripts, run code, whatever it needs to get its job done. So the shortest answer is a sandbox is essentially—we call them composable computers for AI agents.

Exactly. So I think the OpenClaw Mac Mini thing helped a lot of people understand what we actually do. It's like, oh, I get it now. I get it, right? So it needs a computer. In this case, it was a Mac Mini to be able to do different things. So, yes, that helped a lot with awareness, for sure.

Is that kind of— So there's a couple of things that I personally thought about. And so my OpenClaw runs on not a physical Mac Mini, essentially, but a sandboxed virtual Mac Mini. And the reason why I didn't— so the way people usually run these things, and Claude Code or OpenClaw, is usually on their own computer because it helps them organize emails, search whatever documents they have, and whatnot. One, there's a high security risk there because at one point we were doing our board meeting presentation, and I would ask Claude, "Can you go fetch the data from our bank?"

And then it was like, "Oh yeah, just log in and give me access." I'm like, "Log in and give me access? No, I will not give you access." Right. And so right away, fundamentally, for me, that broke the entire thesis of it. So you give it its own machine. I personally gave it its own Daytona account, gave it its own phone number. And the reason I had to give it its own phone number is because it has to do 2FA to get into the bank.

There's no other way except for 2FA with a phone number for this particular bank. And so it has to have all these things like an employee, like a digital employee, to be able to go and access these things. And so the risk now that it has its own computer, it has its own account, these accounts have the limitations. So it can only look at the data from my bank. It can't spend the money from my bank, except for the credit card that we did give it, which has $100 a day or whatever it may be.

And so the only risk you essentially have at that point, if it's in the sandbox, is: will it take this data and now leak it somewhere? And that's something we can talk about later. But essentially, the worst thing that can happen is that, and to your point, you can kill the whole machine if you need to kill the machine.

So that basically comes when we talk to people about what we are building. They're like, "Oh, doesn't this already exist in any of the hyperscalers?" Right? And so the answer is no. And the answer is no because everything that they were built for, which was deploying apps, they were stateless. Like, if you have—let's pick any of your websites or whatever you want, web apps—you do not want that to change on the fly. If you are the company, we'll pick eBay because they're in this building.

So they came to mind. And so if you're eBay and you're an engineer there, you have this new update, a button is here or does this thing. You want that state; you don't want that to be changed on the fly, right? The database might change, information might change, but you don't want the app to change, right? And so with those things in mind, that is how people had built the hyperscalers. Like, that is the fundamental architecture you came into and built on top of that.

And the simplest analogy I give people is, let's say you're building a truck, right? You are a factory for a truck. There's, like, the weight, the type of the engine, the chassis. All of that is made to very slowly but surely, securely transport some sort of goods, right? On the other hand, you can have a sports car, which still has four wheels, it has an engine, whatever, but it's made for different things. It's made to go very fast.

So the way the chassis is built, the way the engine, the weight balances, is completely different. And so you can do a lot of things with both, but they're not fundamentally the same thing. And as a company trying to build out both of those, they're separate platforms completely.

Did sandbox exist before? So, I would argue, I would say that the person that kind of nudged—although he said he didn't—someone else did. But anyway, there was a company called CodeSandbox way back in the day when we used to compete with our company, Codeanywhere, which also was in the realm of, like, Replit. It's all these cloud-based IDEs. And so they called it CodeSandbox because I believe it sounded cute. It's like a box where your code for IDE lived, and they were actually one of the first. So the team there actually used microVMs, did snapshotting, forking, all these things that we do use today.

And so that is sort of—and this is maybe a decade ago—but the utilization or the usage from, or the value that it was giving human developers was not there. There's this whole article about, like, the end of localhost, and people have been talking about this for—I’ve been talking about this for 20 years—and basically developers would say, “You'll take my localhost out of my cold, dead body.” But now that agents are here, localhost no longer actually—one, you don't want that for a number of reasons.

And so we're finally getting to that. So that technology and that thesis around sandboxes originally now seems to be coming to fruition.

That's the thing. Yeah, absolutely. When we think about agents, so one, even if you do run an agent on your computer—like, you want to do that, up to you, go ahead and do that—there's a bunch of problems. One is, let's say you're on your laptop. There's this whole thing on Twitter now where people are holding their laptops open. Yeah, yeah, yeah. There are people—

Oh, that's because I always hold my laptop open. That's just, like, my vibe. But I've been doing that for a long time. But people do that because they want Claude Code or OpenCode to finish.

Because if you close it, it's—

You terminate. Yeah. Or pause or stop, whatever. So, like, the problem is, like, the continual ability to work nonstop is not there as long as it's on your laptop. So that's one thing. The other thing is you can't do concurrency. So you can do multiple to the amount of compute that you have on your laptop, which will be quite limited. And so you might want to spin up 10 or 20 or 50 or 100 or 100,000, whatever that number might be.

And so that's very hard. So ideally, you actually really want to run it somewhere remotely, so you can start on your laptop, you continue on your phone. It's the same computer and same agent that is doing their thing, right? And so that's a takeoff of: we have now decided that it's absolutely okay that it's no longer on our localhost. But again, it is still a localhost to that agent. So it's still a computer to that agent, what a laptop is to us.

My argument is that every agent will need at least one sandbox, sometimes more. And we can get to that. Again, there are places where you don't need—and I analogize agents with humans for the most part. And again, we can do productive work without computers. And so there is—and that was the original sort of chatbot, where basically you would just talk to an agent and it would infer. So it would just think and give you value. So I don't know, a lot of people use it for, like, emotional support or whatever.

For the most part, it doesn't need a computer. It has enough data from you going back and forth, and then it can sort of understand and give you feedback. But that's a smaller subset. If we think of where all productivity gains are among the biggest verticals—healthcare, financial services, whatever—most of that is done via a computer. And so if you want agents to do all these things, then agents will need these computers to do it.

If you just chat, you probably don't need it. Yeah. So even if you have a chat, we can get into detail, but if you chat and it has to search the web, it still has to open a browser or it has to use something like Parallel or Exa or whatever to get to that. That would be a tool call. So it depends on the use case. But the thing that's quite interesting about the world that we live in is, one, let's take a step back.

One, I firmly believe that in due time, all the tools will be headless. Now, will it be inside of a sandbox or outside? That's another thing. All of them will be. And that's the most efficient way for an agent to work, but most of knowledge work is still locked into legacy apps inside of Windows for the vast majority, like the absolute vast majority. So if you want an agent today to do a job end to end, you literally have to give it a computer.

And so an example of this is, again, the board report, which is like, oh, can you pull out this report? And our bank has an API and it can pull it out, but it only has spend on the API. It doesn't have incoming revenue on the API. It's just not exposed, or through the MCP tool. Actually, it's not exposed. And the agent was like, oh, I can't get to it. I'm like, dude, log in, download it.

Okay, I'll go to it. And then you can see it opens up a browser, it goes down or whatever. Same thing with, I don't know, any other data source that we have. If it can pull it from the headless, it'll do it headless. If it cannot, then it sort of logs in and does that. And so if we want to give them power today and get value, we have to enable them to have these tools.

Basically, I think I've structured it better since that tweet, which is right now we have two major use cases, or two types of customers consuming Daytona in different use cases. And one is on the researcher side. So it'll be RL eval benchmarks, and the other will be on what we call background agents or long-running agents. And so when you think of background agents or long-running agents, the most popular, those are where a human is the end consumer.

The human talks to a, let's call it, app-layer service that has an agent in that, and then the agent will call on the sandbox. So things will be, think of Harvey or Perplexity or Lovable as these types of background long-running agents. And so they can both be sort of headless, so code and command execution and/or computer browser use, depending on what they need to do. And the same thing is on the researcher side, where it's like RL and evals and whatnot.

You can do RL and evals for coding, and for that, it's basically just headless, like commands and command execution in there, or you can actually teach it to do things in the real world. And then it does have to fire up a Windows, a Mac, a Linux sort of desktop, or a browser to go through the thing. So code and command execution and browser computer use are like two ways an agent can work. And then basically, the consumption of the sandboxes to do those two different things is different.

I try to think about everything. To me, it's actually quite interesting where—and we'll get to this a bit later as well—a lot of this all exists in real life today. And so people are overthinking this. I'm not saying that there's not going to be new products and solutions and technology to solve it, but it's not a new fundamental way of work. And so when you think about the agent stack itself, it's like, okay, you first have the models, and the models are essentially the brain. That is sort of equivalent to what a human's brain is.

So you tell it something, it replies and understands and whatnot. And then under that, it's like, oh, what are the tools it can use to get things done, right? And so that can be anything from any of the MCP or tool calls it can do. It could be the sandbox, the computer, whatever. We as humans also have a bunch of tools that we use, everything from a hammer to a computer and everything around that, right? So all of those things exist as well.

Then there is memory that exists. Can an agent remember these things? Similarly, do you remember these things that are there? People talk about orchestration of agents. It is like management. You manage people. I manage people. Managing agents is not dissimilar to managing humans. Now, what tools will you use to manage them? Will you manage them the same way, just send a Slack message, or you do Linear, or whatever it is? We will see.

Or it's going to be a net new app. We'll figure that out. But it is all of that. And, of course, it's like observability. Can you see what your teammate or colleague has done and verify that that job is good? Right. And so that's sort of how I think about that on a very high level, and then you can break down the different types of solutions people are creating for this.

The things that I think about, there's also things that are more solved and less solved, where the models—I'm not saying they're solved, that they won't continue to progress. There might be different versions of models, but it's very clear that there is some sort of brain that is there. And we have the leaders in the frontier, and we have people trying to do different things. And that is all said and done. The tooling, I don't know if I would spend too much time.

That's all we're talking about now, which is like the sandbox, the computer, the MCP tools. There's a bunch of them being built there. The thing that's not solved very well is, one, memory itself. And so memory itself, like, how do you do this right now? The best solution is just dumping things into MD files and then sort of having access to that and then compressing that and how you're going to get that. We talked about this the other day, which was the book Why We Sleep, right?

And in that book, Why We Sleep, it's basically a lot of how you do data retention and things like that. And that's sort of what we're trying to solve for agents, agents themselves into that segment. And so memory is there. The thing that is actually quite interesting for me is actually the learning of the model. One thing that I didn't understand, internalize, is that models actually don't learn right now. So you use a model, and even if you solve memory, it has memory of things.

So it has context of these things. And so it can, oh, here's the context, and so it can have a better answer because it has the context, but it doesn't actually learn on the job that it has done yesterday. Right. And so who is solving, or how do we solve that? Does that require something like constant RL learning, post-training for this? Or is there something fundamentally that we change in the model so the models can do that?

I don't know, but it's definitely something that's hindering progression today. And it's not absolutely clear how we get to that point because it is sort of weird to work with someone that actually doesn't get smarter, right? Like, you get a new model, but it's like, oh, it's like a new one that comes out of it. But you do something five times and it can mess up. It literally screws up the sixth time, right? So, like, every single time, really?

And so you can try to prompt it better, you can have more context so it doesn't do it, but it doesn't actually learn. I think that is something that's going to be quite interesting to see, sort of how that progresses.

I mean, not from a personal—but I might change this from a personal perspective. I also would tend to agree with you that Markdown seems quite trivial for the problem that's there and the compression and how you get to all these things inside of that. So the thing that we try to do as a company providing infrastructure for these things is to make sure that we can expose the ability for the agent to access these things in a very simple way and add to them.

But solving it itself is outside of our, let's call it, mandate.

Two things. So if you look, there's, like, the sandbox itself. And again, restating, I'm saying it so many times in this conversation, but if we take, like, an average worker at Goldman, for example, right, you have the worker, the person, which, let's call it the model in this sense. You have the computer which it logs into, and the harness is a set of the way it shapes that model, that it can and cannot do things, interact with things. So it's almost like hands, so to speak, of that a bit deeper.

But basically, the sandbox does support it. And so if you think of a computer, again, I'll pick on Goldman, for example. I've never worked at Goldman, but it's my assumption. I've worked at bigger companies. So my assumption is that when you log into that computer, there is so much software in that computer that logs what you do, restricts what you do, makes sure that you don't leak data and do these things. Again, no prior knowledge of Goldman, just assumptions on these things.

And so those are the types of things that we, as a sandbox provider, will most certainly incorporate into that. But that harness still has its function in that, which it guides the model, like, oh, I now know how to interact with this machine. Oh, I know how to do a tool call. Oh, I know how to—this is how I work with memory. This is how I chain with a model. And so there is value in that. And so we don't take on the harness.

That is something we're pretty sure that we never do. But there are things that we do in the sandbox that help or support that entire system.

Yeah, so, I mean, they have a big push into this space where Claude Code and their Agent SDK have been. And so it's a focus for them to essentially catch up on that segment of the market there. So there's two things, right?

They're different. Yeah, they're different. Yeah.

The Anthropic managed agents, it is a managed service where you essentially have the model, the harness, and the sandbox all wrapped into one, basically. And so you have that all managed for you as a service, that entire stack. So that is there. Whereas if you just have an Agent SDK, you can use that and run that into any sandbox provider or your own or any machine that you want to run it. And it can connect to, depending on the licensing or whatever, maybe you might connect it to that model provider that gave you that harness, or you might be able to interchange those things.

So basically, one is just the harness; the other is the model, the harness, and the sandbox altogether.

Exactly.

Yeah. So we started the cloud IDE space basically in 2009, both me and my co-founder. And so in 2009, for those of you that were around, there was no Docker, no Kubernetes, no VS Code. And so we had to build the entire stack, which was something that we learned how to do and something that we apply today. So it was actually very, very, very early. The only—I say that we started it because the company that started before us was Heroku, which became Heroku and killed the IDE, which we probably should have done sooner as well.

So, but we were the only one that sort of started and finished, finished in that sense. Later on, you had other ones like CodeSandbox, like Replit, and like others that had joined, and StackBlitz, which is now Bolt, and whatnot. And so Replit also now changed, and Bolt into their other directions, and Replit is doing really well. We decided to go in a completely different direction when we decided to do our next company. So we had learned a lot of things on how to build this entire stack, but when we decided to kick off Daytona V1, which is different than it is today, we decided not to go app layer but just be infrastructure.

Probably on the teachings of Heroku, which definitely went in that direction. It's like, oh, we learned how to do all these things on the orchestration level underneath. That seems to be very, very valuable. So let's push on that. And that is how we kicked that off.

We learned a lot of things. One thing we learned is timing. I never understood the timing of the market. Like, you have to pick. It's very hard to time the market, but if you're in the right time, like fairly soon, ideally you want to be just before the time, but that's very, very hard. You definitely don't want to be completely wrong. We were like two decades wrong, or a decade and a half, so definitely wrong on that one.

The other thing is, I did not know the difference between a user and a customer. And so selling Daytona now is a PLG motion, 100%. The Codeanywhere product was also a PLG motion, but the Codeanywhere product had no enterprise use-case value. It was all single-developer value, and single developers do not want to pay for these things. Whereas a product like what we are, it is a PLG motion. It gives value to a single developer, but a single developer is probably working inside of a company.

And so they will pay with their company's card, which is very, very, very different. But you don't know when you start out. So those are definitely things that we understood when we were creating this company. And the difference—we didn't get into detail, but the difference between Daytona V1, which was managing—it was an infrastructure product for human engineers in very large enterprises. It was an enterprise play, whereas Daytona V2, the sandbox, is a—we can call it a neo-cloud, like it's a new cloud, although neo-clouds are usually for GPU clouds.

We did understand that there was an enterprise play to be had. So everything that we had learned of, like on-prem, multi-cloud, different ways of managing observability, audit logs, all these things that you need for enterprises, we had already either baked in or understood that that would be needed. So we prepared the product for that originally.

I mean, it's all about, like, I probably sucked at all of that. Terrible, terrible. I was the worst. So let's take—there's so many ways to say how we did it. So one thing is, I think I very well understand humans at scale, like what the market wants and/or needs and/or feels. And so when you do that, it's like I did not know that inherently, but you sort of start learning that you understand humans. It's very hard for single humans, not so much, but more like larger masses.

Aggregate humans much better. So that's one thing. But the other thing is, I was super shy, geeky, whatever, as a child. And the stage fright, terrible, terrible. Like pitching my first startup, I would be so nervous. Like, if I had to talk on stage, five minutes before stage, I'd go to the restroom like 10 times, just under pressure of these things. And the thing that happened quasi-randomly is, with our first venture, Codeanywhere, we did pitch around all these conferences around the world and whatnot.

And it was such an interesting thing that I decided, like, with one of the founders of one of these conferences at the after-party, quite intoxicated probably. He's like, "Oh, do you want to do one in Croatia," where I was living at the time? And I'm like, "Fuck yeah, let's go do this." And so I go set this up, this conference. It's in Croatia. Two hundred and fifty people come, which was really big for us at the time. And I had hired an emcee that had bailed last minute.

And so there's no emcee. I have stage fright.

I have to go. And so I was literally the emcee for two days in a row, and you break stage fright. Like, you break it. You just have to do it. So what I'm trying to say, this is very different from the go-to-market, but it builds on—it's like, okay, now that I no longer have this stage fright, you start teaching, start understanding what interactions excite people, less excite people, how to bring people, and whatnot. And it's also a problem because when we said we're going to do a conference, I didn't understand what a conference was.

And so it's like, how do you break down what is a conference? Okay, a conference is entertainment first and foremost. And so you have the show. The show is the speakers. How do you get the speakers? And for the speakers, you have to sell the audience. Who is the audience? Who is coming there? How do you get them there? And then someone has to pay for everything. The audience does pay for tickets, but the vast majority is under sponsorships.

And then how do you sell that to sponsors? And so you break those things down, and then you start understanding incentives of all these different parties and you start putting that together. And so our go-to-market motion when we decided to do Daytona—and we did conferences for a decade, more or less. So you have iteration cycles of a year. Then we started doing two a year, then we started doing three a year, and then every conference got better because the iteration cycle was much faster. The feedback loop was much faster.

And then when we decided to do Daytona, our whole go-to-market strategy was originally around in-real-life events. And so we did dinners, meetups, drink-ups, hackathons, but all of them were simpler versions of conferences. And mind you, the conference that we had was like 4,000 people. So it's fairly big. It's not 100,000, but still fairly, fairly big. And I remember telling one of my teammates that works on the conference business with me now at Daytona, I'm like, "There's no way in hell I'm ever doing a conference, ever."

Not doing it. That's the most stressful job in the world. Never doing it again. Never. But we'll do these little things because it's easy. And then people would come up to us, "How do you guys do these events? How do you do these meetups?" Well, when you understand how to do a big conference, like what are the interested parties, then you know how to do a very small meetup. The same things apply. It's just quite a lot smaller, a lot easier to do.

And we ended up doing a bunch of these. They were all—we didn't sell, obviously; these events don't sell—but sold out in the sense of, like, there's no more space in the rooms that we were doing. We now attract partners that co-sponsor these events for us. Just a great motion for us. And then at some point, my colleague was like, "We have to do a conference." And I'm like, "No, we have to do it."

And so we ended up doing that in the Chase Center. We ended up doing the Chase Center in San Francisco two months ago. I think you were there as well. You helped out. And the whole—

Thank you. But all of it, the entire GTM that we have, we have no other things that we do. Like Twitter is a go-to-market. We have zero salespeople and whatnot, but it is definitely—so the way I think about this, and I stole this from, or am paraphrasing it from, David from Sentry, is like there's basically three ways that people pick your product. And one is awareness. Do they even know you exist? If that doesn't happen, there's no way they can pick you.

Two is preference, which is the pricing, the brand, the person, the different features, whatever. It's preference. And the third thing is, is there a deterministic thing that you offer that no one else has? So the easiest example for the third one is, do you have FedRAMP, right? Do you have that certificate? If you have that, the customer can only use you and no one else. But the two above are quite interesting, which is, how do you make sure that everyone knows who you are?

And then you work on the preference. And so for me, I don't—it's not all I think about. A large part of what I think about is if all things were equal. So if our product is equal to everyone else's product, what is the differentiator to that? And so, one, can more people know about you than others? Is the branding there? Is the experience there? Experience is a nuance which some people don't think about, some people do think about, but it's like, do you prefer the feel of this product versus that product?

And these are all things that have nothing to do with the actual technical capabilities of the product. And so my co-founder Vedran, our CTO, his job is mostly to make sure the product is better than anything else in the market. And my mandate is to make sure that even if our product is equal, that we can supersede that. And so that's how I think about go-to-market in general.

All of that? It's all go-to-market. All of that is go-to-market. And we've seen this, we've chatted about this, where we've been getting users and customers just because we are so good at that. And so all of it is an experience. So if you think of any experience as a human, you go to, like, a store, restaurant, whatever. It is the entire experience. What is a brand? It's the perception of that brand itself.

It's just a perception, and the perception is, like, if you go into whatever store, pick your brand you want, the smell, the music, the people, the smile, whatever you get, all of that together is the perception of that brand. And so if you think of that through the lens of a product, which might sound counterintuitive, I don't know, or non-obvious to people, I think about that altogether. Let's say we were selling sneakers, right? We don't sell sneakers, but the sneaker—we won't say other brands, but if you do, you can always pivot to GPU.

Yeah, exactly. Like, if you go to the store here in New York, it's a beautiful store. The people are very nice. Everything's aesthetically pleasing, and you just enjoy that entire experience, and it's a good sneaker, right? And so you have to have that all together. And so that's how I think about this as well, which is, you have to have a good product, you have to have all these things, but all these other things have to collide with that.

Now, the risk is, and we've seen this with a bunch of startups, like there's no substance, so the product is not good, and you have everything else. And that is sort of—that's when you get into trouble. But if the product is good, or at least as good—if it's better, it's amazing—and you have all these things, for me, my belief is that that is a key driver to continue growth.

It's hard to measure. I've worked in bigger companies, and if I was working for my former company, they'd be like, "How do we measure that?" I have no idea how we can measure that. But what I can say is that I was never very active on Twitter—I have had an account since 2009, but never very active—until this holiday season, where there was this semi-viral tweet that went out. What did you say again? So the tweet was, "If you're taking a break these holidays, you're NGMI. You're not going to make it."

Which I honestly believed.

Like, the tweet was basically—I wasn't even thinking about—there's like a typo in the tweet. Like, it's not—people are like, "Oh, you're rage-baiting." It's like, no. My thought is, we are a part of this supercycle right now, and the supercycle does not last forever. And so if you're going to pause during the supercycle, you are ceding market. Like, that is what you are doing, right? And also, it's very much towards founders and executives and tech leaders.

It's less about the individual person that can't have an impact on that. People have to take their breaks, and so people took that in all different ways. There's like two camps, two boats. Like, "You should die, you're a capitalist," whatever, all these things. And then I said, basically, "That's why you have to keep working," and that was perfect. And again, I was like instant reply on that. And so, basically, not to talk too much about that, from then I had noticed that people, even if they like your tweets or don't, or are completely opposed or not, has no bearing on—I should say it has a positive bearing on—their thoughts of your company.

So we have found that people that even are completely opposed to the posts end up being customers, assuming that they need the product. Not everyone is there. So it was quite interesting to me that, oh, just because someone doesn't agree does not mean that they won't like that, because it's all generally awareness. And so that has been something that I've spent a bunch of time trying to catch up to you on, on the Twitterverse.

We have no salespeople now. Yeah. So I think, like, how do you experience the product, right? So once you've seen the product, how do you experience the product? It's like your door to product—the website, the login, the whatever. We can fix a lot of these things. To be very clear, I'm not saying we're the best at this, but there's that. And then what are the feature sets that are in there? Can I get things easily?

Our SDKs are really, really, really good. People really like the ergonomics of them. So it's like really good. That part is great. And then the thing that—and this is even public on Twitter—all our case studies that we've done, and we outsource the case studies to third parties, so we're not part of this, is that we reply very, very fast. And so this is a very public thing, and it's something just core to who I am and how I learned to work.

Because one of the, let's call it, first real jobs I had was a system admin. So my job was to fix the printer and computer and whatever. And so one thing that I learned, and I try to talk to my entire team about, is one thing you have to do is: the first response, very fast. Just the first response very fast. People are then calm. They know they have transferred their problem to someone else, and someone has acknowledged that.

And so when they do that, they feel relieved, right? And then you just have to promise that you will solve it or get back to them in X amount of time, whatever that is. And they will be calm until that moment. And the thing that you have to do is either solve the problem by the given time or call them, message them, whatever, and contact them before that time and state a new time. That is the solution to support. That is it.

There's nothing more than that. There's obviously key things if you're actually down and don't work and the person can't work—that's a completely different thing. But every other non-critical problem, and you have the happiest customers in the world because they don't have to think about their problem anymore. You think about their problem. And so you can keep that on until you get it done. But the key part is if I said that something is going to be fixed or deployed or done or whatever in two days, a day and a half later, if it's not going to be in two days, a day and a half later, I say, "Hey, this is going to be delayed two more days," and they're fine.

You thought about it before they thought about it. And it's not magical. I mean, it's a magical experience, but it's not a complex thing. But I found that people don't understand that intuitively. And so that is what I learned back in the day, and that is what we do in the company today. And that's why we have very, very happy users and customers.

Yeah, there's a lot of things there we can talk about. The way I think about a sandbox is the ergonomics of consumption of compute. And so, because there's a lot of different layers on that, I basically put it into three different layers, which is the infrastructure, the primitive, and the tooling. Those three things together. And what I mean by infrastructure is, does it spin up very, very fast? Can you spin it? So, like, we spin up in 60 milliseconds.

Can you spin up a lot of them at once? So, the rate of creating them. And so we can spin up 50,000 in 70 seconds, so a little less than a minute and a half. And then, once they're up, how many can you keep running? We have customers that have billions a day. So those are the infrastructure parts that are non-trivial.

Again, it depends on the user and the use case. So if you're like a long-running background agent, again, everyone prefers it to be fast. No one wants to wait. You don't want to wait for a reply. Everyone wants it to be fast, to be very clear. So the faster, the better. But generally, there's an actual reason why you want it very, very, very fast. And that is especially for a background agent. A background agent might work for like 10 minutes or an hour or whatever.

So the incremental millisecond might not matter. But I still believe that, from a user perspective, even a second of waiting is kind of uncomfortable. You don't want that. And so, is it 60 or 90? Maybe less so. But there, you want that one or two seconds, for sure, under that. But the more interesting part, where that is really, really important, is for the researchers, where, when you're doing reinforcement learning, you basically have an allotment of GPUs, and the GPU is more expensive than the CPU.

So the vast majority of sandboxes are CPU boxes, to be clear. So they're the computers that we all work on. There might be a graphics card in there, but basically it's the compute, the RAM, the CPU, and the hard disk that's in there. And they are cheaper, less expensive, and easier to get, at least for now, than GPUs. We'll see how long that lasts, which means you want your GPUs always to be at maximum utilization, and the CPUs can then idle if they need to idle.

But you don't want the GPUs to idle. And so what that means is you want to make sure that the CPU machines, the sandboxes, spin up so fast because you don't spin them all up in a training run. You spin them up, like, depending on how many you have, it's like 1,000 at once. They do the task, then do the next one, the next one, the next one. And so, between turning off and on the CPUs, you want that time to be as short as it can so that the GPU utilization does not go down.

Right. And so that's why that's very, very, very important. And obviously, the number that you have concurrent depends on the background agents. Let's say, just for the sake of people who know Lovable, the amount of users that they have is astonishing. And so each user, for every task—and they can have multiple tasks, multiple agents—they need a sandbox. And so imagine, I don't know, their user number in the millions, whatever. So you have to have millions of sandboxes up and running.

On the RL side, you can have smaller labs that will have 5,000 or 10,000 concurrent. But we have a request today for 5 million concurrent. Look, so 5 million at one point in time, right? So being able to handle those types of things is part of that infrastructure segment. The other two are the primitive and the tooling. And so the primitive is essentially, let's call it the computer itself: the isolation, the VM, the container, the microVM, the isolate, the whatever.

We can get into details of what those are, but there's different shapes of them. And also, what feature set do these things have, right? And so, for the most part, a VM that you would find in AWS or whatnot is usually much slower to start, much harder to get those spikes. But also, when you get a certain size, they're usually that size forever and forever. And then, just one example: whereas in Daytona, you can define a size, the CPU, the RAM, the disk, and then while it's running, you can resize that.

So if an agent gets to use the entire memory or 100% CPU, the sandbox or VM would die. In our case, we can expand that. And there's other things. For us, it's not just a Linux CPU box. It could be a Windows, a Mac, an Android. They can have a GPU in there. It can not have a GPU, depending on what you need. And so that's the primitive. And the last thing is the tooling.

The tooling is either there to support the agent to be better at its job or to be guardrails on the agent to stop it from doing dumb things, basically. So you can think of it: we have a bunch of tools, so headless terminal and file operations and other things that enable it to use fewer tokens, get the job done faster, but also guardrails like Secrets Manager and a firewall and all these other things. And so the combination of those three—the infrastructure, primitive, and tooling—essentially creates what we call a sandbox.

All the things, all the joy. So most sandbox providers are Firecracker VMs, microVMs, mostly.

Firecracker, it's an isolation provider. So it is essentially a type of VM, or virtual machine, that's very stripped down. It was made by the AWS team. It's used inside of Lambda, which are like functions. And so the idea was to have something very, very fast that can spin up and stay stateless. That was the entire thing. And they were very, very ephemeral. And so they were used for when your website has a large—if it's Black Friday, a bunch of people hitting your website—you can spin up a lot of these very, very fast, handle the load, and enable all these humans to look at the website and buy whatever they were buying.

Right. And so that technology is there. It's a very stripped-down version of a full-on VM. So it has less features there, but the things that it does, it does very, very, very, very well. So you can spin them up fast. It does that because it can sort of lazy-load things into memory. You can do point-in-time snapshots. You can do all these nice things. The things that it can't do is it can't run a, let's call it, sandbox now.

It can't run a sandbox with a GPU. It just doesn't work. So you can't have a Firecracker. So if you want a GPU, you have to do something which is a cloud hypervisor or a QEMU, spelled Q-E-M-U, which are two different types. QEMU is almost a full VM, so it has all these different things inside of there. So you can't do that. But let's, for example, if you need to run—we were solving this for one customer earlier today—they need a machine that also has an Android device inside of it.

And so the only way we could solve that was inside one of these QEMUs. It can't work in a Firecracker, can't work in a container, can't work in anything else. And so there's different types there. These are like the microVMs of the world. There's also containers. And so containers, most people know Docker, which is there. Daytona originally started running Docker containers. We do have them. The problem with Docker is that they're much less secure than a VM.

The thing that we do in Daytona is we harden that with Sysbox, which is a VM-like isolation that wraps around that. It's really good for handling density of these machines. It's very fast. It allows us to run Docker on Docker. There's a bunch of features that it's very, very good for. And so there's differences there. There's also isolates, which we've talked about, or other app containers, which are abstractions of these containers; you're just running a single app.

And so these different things all exist in the world, and they can all be theoretically sandboxes with these other two things that are there. And our original idea is to get back to the original conversation. Most sandbox providers started as Firecrackers, but as we start seeing that agents have different needs and different use cases, they are going to need all these different shapes consumed through one interface. And so we at Daytona now support containers and all the microVMs depending on your use case.

The user doesn't know. It's the same ergonomics. It's like, oh, I need Windows, it'll spin up this one. I need Linux, it'll spin up that one. And so we will continue to add these things inside of the same infrastructure. Sometimes it'll be slightly faster or slower, but the concurrency will be there. All the tooling will be there. And so everything that you would come to know and like about Daytona will be there, but it'll be different sizes and speeds and configurations depending on your use case.

There's a lot of reasons. Let's get into the commercial one, which is probably the one that people think about most, which is pausing sandboxes. So if you are an app-layer company and you have 10 million users, and your 10 million users spin up, let's just say, 10 million sandboxes just to make the math easier, these things cost, right? Because they're running, they're using CPU, RAM, and hard disk the entire time. The most expensive thing is the CPU. Second, the RAM.

This is almost free. It's very inexpensive. And the user sends the agent to do something, the agent does something, and now it's waiting for a reply. It could be waiting for the human, or it could be waiting for a service, depending on what it's trying to do. You ideally don't want that sandbox to run idle because you are now, regardless if you're a customer of Daytona or running your own, there is a cost to running these things. And so what you want to do is pause that and wait for a reply, either from a service or from a human, and then resume. And the feeling and experience is that it never turned off.

So you feel like it never turned off, but it actually did turn off. And so that one is, from a compute management perspective, if not from a cost perspective, probably the first reason. The second thing is you can enable your agent to take multiple paths. So your agent can say, at this point in time, I'll take a snapshot and then I will either continue and be able to roll back to this point, like a point in time, or at this point in time I will replicate the sandbox and have two sandboxes, and I can try two at the same time.

Obviously, it can be more, but right. Those are the reasons why you would have that there.

Every cloud that exists today, neocloud, hyperscaler, whatever, they are all built on servers, right? Like metal machines. And historically, way back in the day, we actually stacked these data centers, me and my co-founder, a long, long time ago. But they're all just servers, like CPU, RAM, disk. They're computers, basically. And on top of these computers, there is a software stack that everyone has built for their own reasons. So AWS has their software stack, Cloudflare has theirs, which is very different.

Everyone has their own on top of that. And so basically, what you're trying to do when you think of these machines, these servers, basically what we do is cut them up into small little machines and then give you that sandbox. And so, on these big servers, you have these small little machines that can run for a minute, a second, three hours, whatever. And you don't have to worry about this. And so basically, the scheduler or the orchestrator is the one that says, after you send me a request, I send you to this server.

And turn on that sandbox, a little computer there, and then I turn it off or I kill it or I snapshot it. It's the management of all these things there. And most of the other companies in the space basically have off-the-shelf schedulers. So it can be like Kubernetes or Nomad or whatever. And when we decided to build Daytona, the sandbox product, we inadvertently, with a lot of naivete, were like, oh, this stuff doesn't work because we had built our own, then we had used Kubernetes.

Like, we've seen all these different things and we knew what was good and what was bad, or what was use case or not. None of these were made for these super-fast, stateful, long-running machines. And so we're like, we have to do this again. We have to build it again. Because the way we thought about it, which was not known at the time, is because at the time sandboxes were very ephemeral, similar to a Lambda function. And we're like, no, why would your sandbox be ephemeral by default? Like your laptop, you don't want it to die.

Like, you want it to work until it's done, and it has to be very fast. It has all these things. And so my co-founder mostly built this sort of initial version of our scheduler, and that scheduler of ours is the basis of everything that we do. And it gives us a lot of things outside of just the performance. It also enables us to do things like have four different isolation products. Like, we are the only company right now that you as a user don't know.

But, like, we have, depending on your use case, we will spin up any one of these: Firecracker, Cloud Hypervisor, or whatever, to get the job done. So whatever is needed, we'll get there. The other benefit is we built this, we have colocation providers, so we have bare metal machines. So you can think of us as our own cloud, but we're more and more akin now to the neoclouds, like the Base10s and Fireworks, because we got this question yesterday because we have a large request for a number of CPUs. It's like, we can't get enough.

And so how do we solve that problem? Well, we've already solved it, where our scheduler can be attached to any CPU machine, any server that we think is good enough, and it becomes part of our cloud, essentially. And so why I say Base10 and Fireworks is that both of them run on more than a dozen compute providers, depending on what you want to call them. And so we can do something similar because we've created this in such a fashion. So that has given us, I believe that's given us, sort of an advantage there.

The reason most sandbox environments do not run forever is because it's a technical problem. If you think about servers underneath, these servers also have to be managed and maintained. And so if your sandbox can run forever, that means that you can never reset, you can never reboot the underlying server, you can't update it, you can't patch it, you can't do all these things without turning off all the sandboxes. And so the easiest way to solve that is having sandboxes that have a termination time.

It's like they will only last an hour, 24 hours. It doesn't matter. Pick your time. And then if you decide that you have to do something with the underlying machine, you just flag that machine as non-schedulable. And so, at some point in time, there's nothing else on that machine. You can fix it. You can reboot it. You can do whatever you want. Easy peasy, done, right? And so because historically most of the workloads were ephemeral, you didn't have to try to solve that problem because you didn't care.

Like, most workloads, like a Lambda function, usually runs, what, five minutes? And, like, whatever, it's not a problem, right? You never had that restraint or constraint. Now that you do, to have something that can run forever, you have to be able to live-migrate the sandboxes themselves between the machines so that you can reboot and manage these machines. And I understand that most people—it's interesting, even technical consumers are like, "Oh, all compute is elastic." Well, yes, your consumption of it from us or from AWS is elastic, but someone actually has to screw together a server, turn it on, boot it, make sure it works right, and have enough of that there.

So that is the difference from our perspective and why most start off ephemeral. It's an easier way to do it. Five percent of our sandboxes run longer than 24 hours, but it's a large percentage, like 20-ish percent of revenue. So it's not an insignificant part to be there just because it can run for a much, much, much longer time. On the user perspective, just to finish on that, sometimes users actually want it to be ephemeral. One is they don't want any data retention, so they want it to run and die. Like, whatever was run inside of that, they got the data they wanted and they want that data dead, dead.

And so they flag it with us as ephemeral, which means that we won't force it to die. But when they kill it, it goes into the ether. It does not exist anymore.

No, so we've seen this multiple times, and I'm sure you've seen this across other companies as well, where everyone's like, "Oh, I can spin up a Firecracker on a machine." Yes, you can spin up one, but spinning up a million is a problem. Adding all these features is a problem. Having throughput for different use cases, they're all different problems. And what we already see is companies that were like, "Oh, we'll do it ourselves," come back three months later, six months later, eight months later.

It's like, "Oh, actually, now we can't do X and Y." And so there's a bunch of things on the technical side which we can address. And I think where we could finish with that is that the thing that Daytona does is, if you think of most sandboxes, this is a performance thing. Most sandboxes and most VMs actually use the CPU and RAM from the server, and then they use the hard disk from a network drive. You do that for a bunch of reasons.

Your life is so much easier if you manage that. Why is it easier? Well, it's easier because if a server dies, then all the data is on a shared drive, and I can reboot that server really fast without maybe even the user noticing. And there's no data loss, maybe something in the memory, but nothing there is lost. So it's very, very easy to manage. Whereas what we do is we actually use the CPU, RAM, and hard disk of the machine.

Which means we have so much more overhead of, if that machine dies, how can we reboot it back? You have to do backups that users don't see. You have to do all these things that happen there. But what that means is that the IOPS, the speed with which you can move data to the hard disk and which the CPU and RAM interact with, from a network drive, just to give people numbers, is in the hundreds of thousands.

If it's a local drive, it's in the tens of millions. So it doesn't matter if you know what the numbers are, it's just extraordinarily large, right? It's very, very different. And so you have use cases that no one really cares about that speed, which is fine. We have new customers that have a use case. They're like, "It has to be that fast," right? And so when you get to the point, it's like, "Oh, I can use this."

But wait, now I need this performance. Oh, now I can't do that. Right. And so, to the complexity of sandboxes, if people are starting out, their use cases are like, "Oh, I just need to run code," you can run that on anything. It's very easy to do, except at scale. And then when you get to scale and you need performance because now agents can do a lot of things, then it starts breaking that sort of mold, and security.

And so we didn't even touch the security part, but yeah.

Yeah, it might be happening. So one of your former guests, Dylan Patel from SemiAnalysis, was at our conference. They wrote the report, and I believe it's somewhere in October: we have no more CPUs. And people hadn't thought about it, like, it's all GPUs, all GPU-intensive, then memory. But basically, now that agents need all these computers, it is very much needed. Now that RL is the way that we've gotten most of the progress in models over the last 18, 20 months, you need all the CPU machines to do this.

Now that the GPUs are more powerful, they can do more of these CPU boxes in parallel. And then you see this. There was a tweet that I put out a while back, and it was like, you should definitely invest—not financial advice—you should invest in Intel and look at their stock price, right? And so there's definitely a need for that. And it's something that people didn't understand intuitively, that you would need there. So it's something that we think about quite a bit: can we, alluding also to how our product is created, make sure that we can match the demand of customers for the CPU that they have? Because I don't know, it goes to the extreme of where GPUs are, because that is very, very, very extreme.

But there is quite a high probability that there will be shortages of CPUs going forward.

So we talked about the stack there and things that have to be done. The identity part does need to be solved. Again, it's basically solved, but not solved. So there's a lot of things to do there. I'm actually also wary. I think people think that the models that we have, the technology for the models that we have, is the end state. And I really, really don't think that is true. I think that would probably be the most surprising thing: we have a new type of model that is just better or different than what we have now, that either needs less compute or doesn't use GPUs or doesn't, like, I don't know.

There's things that we've seen with wetware where people can have fake human brains and it learns how to play Doom. You have the, what was it, the fly brain that was replicated inside of a computer that then works like a fly? Can you replicate a human brain? And I say this not to replicate myself, but can you replicate a generic human brain? And then it's like an AI that is essentially equal to a human.

And so what does that mean for all of technology? I mean, I'm now saying extremes, for people not to get me wrong. I don't think that happens tomorrow or that I'm completely crazy, but I don't think that this is the end-all state. The question is, does that happen sooner or later? Do we keep our transformers, the vast majority of intelligence going forward? And if it is, then it continues probably directionally where it does. But is there something that happens that changes that fundamentally?

Something that I think about, because then all our bets, yours and ours, are very, very different, right?

Thank you for having me. Great.

Bye.