Can AI Infrastructure Work Like Magic? | Erik Bernhardsson, CEO, Modal

Thank you. It's exciting to be here.

Yeah, so Modal makes it easy to build, scale, and deploy applications in the data, AI, machine learning realm. So basically, you can think of us like: you write a little bit of Python code, and we take that code, we stick it in a container, we execute that in the cloud in a way where you don't have to think about infrastructure. You don't have to think about GPUs, you don't have to think about the cloud vendors. We just magically make that work and scale it to very large—thousands of GPUs, potentially.

Yeah. And by the way, I think it's funny because it's like the typical VC, like, what's the market map? And my role is like, there is no—we defy it. We do all of it.

No, I don't think it was the nonsense. I think I was just pointing out how much stuff there is. I love your market map, by the way. There used to be like—

People, for the record, we will get back to that later. I built this thing called Luigi many years ago, and it used to be kind of—you could see it, but every year it's getting smaller and smaller. I don't know if it's still, but every time it comes out, I zoom in and I try to find it.

It's still in there. Nice. I love it even more now. Anyway, going back to what does the market map look like? There's obviously the clouds, right? Like AWS, GCP, Azure, et cetera. Oracle. I should shout out to Oracle. I actually like them a lot. We use them. Big fan. So they, I think, have been there for a long time and have been doing a phenomenal job building primitives for running stuff in the cloud. More recently, there's been a bunch of sort of newfangled, I call them like the alt clouds.

CoreWeave, Lambda, what are the other ones? There's like FluidStack, there's Vultr, there's a lot of them, right? As it turns out with GPUs, what people really care about is like, just want a box with the GPU in many cases. And so that's something that's like a little bit more of a commodity, and I think a little bit easier to build in, arguably. They would probably disagree with this. So I think we've seen a sort of renaissance, or not a renaissance, but like a bunch of new sort of cloud entrants, like sort of competing with the hyperscalers, focusing more on just like only GPU availability.

It's like they provision a box with a GPU and you can SSH into it, right? So it's sort of like you get a GPU on your desk.

Yeah, it's sort of almost like real estate, right? They own sort of real estate and they rent it out, arguably sort of WeWork, you could say, if you want to be a little cynical, but not a ton of value higher up in the stack, which is, I think, where we play, is slightly higher up in the stack. I think if you go up in the stack, start looking at more like applications or infrastructure as a service, there's like a few different providers. First of all, there's a whole bucket of all the LLM inference providers.

So there's OpenAI, Anthropic, then there's a few ones focusing on open-source models. So Fireworks, Together, I think Anyscale to some extent, Replicate, et cetera. So those I would put in the LLM bucket. And then there's like a slightly adjacent bucket, I would say, of like more like AI as an API. And actually, that's probably more like where I would put Replicate. I also put fal in there. I think we occupy a slightly different bucket, which is more like you're not using a model through an API, you write code.

We're all about running, taking users' arbitrary code and running it in the cloud, right? Like, we're a serverless compute platform. So we take your code, we execute it in the cloud. You can run whatever code you want. In many cases, that involves GPU, in many cases not. In that space, I think there's less of an obvious—I would say Base10 maybe to some extent is somewhat similar to us. They focus only on model inference. We do model inference.

Model inference is probably the majority of our use case, but we also have many other use cases. We have people using us for batch, we have people using us for large-scale video encodings or building web services or all kinds of other stuff. And then I don't know, what other boxes should I talk about that we didn't cover?

Yeah. And you mentioned Groq. It's not like an area we're playing in, but I do find it super fascinating.

Traditionally, all these LLM inference providers have obviously been building on H100s or NVIDIA GPUs, but I do think it's kind of fascinating how it's kind of going in the same direction as crypto mining towards more specialized hardware. Cerebras is another one. I think it's sort of interesting to see for LLMs. I think the reason why it's happening for LLMs is it's like a kind of well-defined API. It's like text in, text out, and it works really well. And it's sort of a few big models dominate everything.

The spaces where we tend to compete and where we have most of our customers is more like very custom models, like people training their own proprietary video models, image models, like audio, things like that, or having very custom workflows. And there's less of sort of like power law there. There's thousands of models. Everyone has their own slightly different models. And that's where we focus, is like, if you have your own custom model, you're probably really good at training models and building applications, but we handle the infrastructure.

So all the auto-scaling and all the making sure utilization is maximized and all these things.

I mean, I don't know if I necessarily believe in it, but I certainly bet on it. I think there's some sort of existential risk that there's such a good multimodal model that people just use it for everything, right? It's like a model to rule it all. I don't know if that's true. I tend to think it's probably less likely to happen than other people just because having seen so many custom, all the idiosyncrasies around building custom models, let's say you want to build a voice-cloning thing, right?

Probably, like, a multimodal model is not going to care about all the specific things around that, right? I don't know. I'm a big believer that, in the long run, people will train a lot of different custom models, even though the multimodal models may handle even more stuff. I mean, the pie is growing. I guess it's like a general statement of software engineering in the last 30 years, but the pie always gets bigger.

Yeah, so I've done a bunch of different things, but in particular, I was at Spotify for seven years, and I built a thing called Luigi, which was a workflow scheduler. So this is sort of a problem I tried to solve at Spotify: we had a lot of different batch jobs, and you ended up with a very complex graph. You need to run hundreds of different jobs in a certain sequence in sort of very parametric ways. And so I built Luigi to sort of help us figure that out, like how to execute all of it in the correct order. Open-sourced it.

A bunch of people started using it. I think this was like 2011. Then I stopped really caring about it in 2015. Airflow kind of took over, and then later now there's Dagster and Prefect and Flyte and a bunch of other ones. I mean, I still think it's a good idea, and kind of related to how I ended up starting Modal. Part of how Modal came to be is I actually started looking again at workflow scheduling in late 2020, when I realized I wanted to start something or build something, and started thinking a lot about what makes a good workflow scheduler.

I even built one. I have some janky code on my laptop. I realized at some point a workflow scheduler is only as good as the underlying compute layer, because at the end of the day, you can't just describe the work you need to do. You also need to describe the code. You also need to write the code. And if that integration point is bad, if you have kind of a system where you have to define the workflow over here and the application code over here, you're never going to have a great experience.

So I realized, actually, I'm just going to stop working on the workflow scheduling. I'm going to focus all on the compute provider instead. And what makes a good compute provider? I realized, what I want is I just want something that lets me define application code in Python, because I was very focused on data, AI, machine learning applications, and let this infrastructure handle all the rest, like all the scaling, all the provisioning, all the resource management, all that stuff. And I just got obsessed and carried away.

And three, four years later now, I have a company around it. But that's sort of the genesis of Modal. So it kind of started with Luigi in a way.

Is that what I told you a long time ago?

I feel like we had a conversation many years ago, and I probably said something like that.

Yeah, early 2021, really.

Pre-GenAI.

So it's a little bit different. And yeah, I think that's always been the ethos of the company. At the end of the day, I just want to build better tools for data, AI, machine learning teams, and what we actually build, I'm actually somewhat agnostic. At the end of the day, I want to be a platform that helps people build those types of applications.

I mean, a little bit. I don't know, AI has its ups and downs. I think we had, whatever, four AI winters or whatever. Now AI is like a great term. I feel like 2020 AI was like a bad term. I used to say only, like, data. I'm just going to focus on data. There's been so many other terms coming and going. To me, it's all the same. It's like data, right? You're working with data, and sometimes you're training models, sometimes you're doing, I don't know, data pipelines and batch compute.

But at the end of the day, it's data.

Totally.

So I think the core of it is really just compute and running containers. That's where we started. We always had a vision of building a platform, but the core of it is: we want to make it easy to run code in the cloud. And it turns out that inference is such a killer use case for that, that we effectively spent the last three, four years now entirely focused on just how do we take user code and execute it in the cloud in a way where we handle the scaling and all the infrastructure.

We sort of let everything else wait until we really nail that experience. How that breaks down is under the hood, and we realized in order to deliver that experience, we had to kind of go deep in the layer of the infrastructure and throw out a lot of existing stuff. In particular, one of the things I always cared about is developer experience. I wanted to build a tool I always wanted to have through my own experience building these types of things.

I didn't mention it, but at Spotify I built a music recommendation system. So a big part of my life used to be iterating and shipping and deploying things, or just running various types of batch jobs and debugging things. And at the core of that, one thing I realized, for instance, was if you don't have a good, super-fast feedback loop, you can never make it. So much of developer experience comes from having a super-fast feedback loop where you can take code and execute it in the cloud in a way where it almost feels like it's local.

And if you solve that problem, you solve a lot of different problems. You make engineers more productive because they can iterate and run things quickly. You basically get rid of what I think was the gap between local development and cloud development, which I think with AI and machine learning has always been a big challenge, because you end up having different environments, different toolchains, and then you end up having to take research code and repackage it and rerun it in order to get it out there in the cloud.

So if you get rid of that distinction between local and cloud and just turn it into one environment, you always run things in the cloud, you solve a lot of different problems. But in order to do that, you have to make it fast. You have to make it feel like you're developing locally when you're running things in the cloud. So what we realized, in order to do that, is you have to solve a number of core technical challenges around container cold starting.

So what I mean is you have to take code and you have to ship it to the cloud and you have to start that container. And ideally you want to do that in a couple of seconds at most, because that's sort of a human reaction speed. It kind of feels snappy. And then we started looking at, okay, well, what actually happens typically in a system when you do that is you deploy something to Kubernetes, and then you have to pull down a Docker container image, and that then has to start up.

And all of those steps are super inefficient. So we built our own. We threw Kubernetes out the window, we threw Docker out the window, we built our own file system in order to optimize for how container images are distributed. We built our own scheduler in order to maintain this pool of workers and make it possible to start tasks very quickly. We built our own container image builder because we had our own container image format.

I mean, it was too long. And this was during the ZIRP era. So I feel like VCs were like, yeah, cool. People were asking, why do you really have to go so deep? But fundamentally, I think we felt like we're going to solve this problem well. So yeah, we spent basically a year and a half, two years, two and a half years, almost like in a cave, just building a product that no one had used, which is maybe a bad idea.

And then when we actually kind of emerged out of that cave, it turned out we didn't really have an obvious use case. So we struggled for six months or a year or so to try to find a use case. And then luckily all this Stable Diffusion came out in summer of 2022 or something like that, and a lot of people started coming to us and were like, actually, this kind of makes sense. You have this serverless GPU thing. You can sort of realize this makes a lot of sense for us to focus on.

And then we started seeing more meaningful traction with users.

They're all over. And this is not a secret, by the way. We use AWS, we use GCP, we use Oracle. We have a number of alternative cloud providers now that we're also using. And we run a lot of different regions. We have a bunch of GPUs in the US, we have a bunch in Europe.

Yeah, totally. And for a few different reasons. One is just capacity management. From time to time, capacity varies, right? There's this sort of cyclicality. If you look at Europe versus the US, it's actually kind of interesting. You see that when it's early morning, the US is not high utilization, but Europe has high utilization, and vice versa. So you can play all these games kind of just aggregating capacity across different regions. Pricing is the other obvious dimension, right?

We actually have a system that continuously looks at cloud pricing and solves an optimization problem to sort of figure out how do we allocate capacity across 100 different regions in order to deliver the capacity we need to our customers. Because we use a combination of reservations, but also a lot of spot and on-demand capacity, which means we rely on the cloud's ability to scale up and down, because it's a sort of supply-demand matching problem, right? We have volatile demand coming from our customers, which means ideally we also have the ability to sort of match that with on-demand capacity on the other side.

Which is also why I think we found a good product-market fit with inference, because inference is inherently sort of unpredictable, right? You don't know necessarily when you're deploying something what the usage is going to be. There's going to be a lot of daily variations, going to be a lot of weird spikes when something goes viral, et cetera. And a big part of the pitch of Modal is you can use us, we'll just scale automatically. You don't have to worry about making a cloud commitment, especially not a three-year or one-year reservation.

You can just deploy things on Modal, and then if you one day need 1,000 GPUs, we can typically get you 1,000 GPUs pretty quickly, like, talking minutes.

Yeah, you can definitely take a Hugging Face model and deploy it. That's what a lot of people use Modal for. I think where Modal really shines is also people training their own models, like having custom models. One example I always bring up as an amazing use case—I love the product—is Suno, which is AI-generated music. And they have a big cluster; they train their own models outside of Modal, and then they use Modal for the inference side, which means all this sort of generation of AI-generated music happens on Modal at very large scale, right?

Millions and millions of pieces of music generated on Modal.

Yeah, it's actually funny because sometimes when I talk to them, I end up talking a lot about licensing and stuff like that, music licensing, because I happen to know a lot about that. But yeah, so what I love about that is they can do what they're good at, which is to build an amazing user experience and a user application and train models. And for all this sort of infrastructure mess, they can effectively outsource that to Modal. And that happens to be something we love and we can do really well.

So I think that's sort of the comparative advantage of specialization in that sense.

We are interested in training. We're looking at it. I think training at very large scale, people are very price-sensitive; it's somewhat transactional. I think you have very different sort of requirements, typically, like InfiniBand and very high interconnect. It puts a lot of different constraints on the infrastructure that we don't support today. It's something we're interested in, somewhat looking at down the road, but it's not a super high priority for the business right now. I think another use case that we do support is actually a lot of preprocessing for training.

So a lot of users use this for, let's say you have millions and millions of audio files or video files and you want to extract some training data from that. Typically, that requires a sort of batch job that extracts features from it. And that's something that Modal does really well too. It's like very high, bursty batch parallelism, just like fan out, parallelize, run this over millions and millions of audio files or something like that.

Totally, exactly. It's sort of pay-as-you-go because everything in Modal is usage-based. So you just pay for the exact time your GPUs or CPUs run.

I think fine-tuning is like—a year ago, I would have said fine-tuning is absolutely critical. I think now the models have gotten so good that this is sort of a question of: is fine-tuning even needed? I think fine-tuning has a lot of different use cases. I think in particular, if you have very domain-specific models, let's say you're focused on financial data or document extraction or something like that, I think fine-tuning absolutely makes a ton of sense. And so we have a lot of users using Modal for that.

I think beyond that, I don't know, fine-tuning is still, to me, a little bit of an unproven thing, but clearly there are certain places where it does make sense.

Exactly. Yeah. And also, a couple of years ago, a lot of companies tried to train their own LLMs. To me, it makes almost no sense, right? Like MosaicML and stuff like that. So, yeah, I agree with you, right? It's obvious that in the long run, inference spend will dominate just for the obvious reason that that's where you make the money, right? When you're training a big model, you're spending a lot of money training that model. How do you recoup that?

It's typically through inference, right? If you look at OpenAI, any of these businesses. So to me, it's very clear that maybe in the past we've had like 80/20 spend, GPUs training versus inference. I think in the future it might be the other way around. Much more dollars will be spent on inference than training.

Yeah, there's a bunch of people using us for data parallelism and pipelines and stuff. There's weird sort of unexpected use cases. I shouldn't say weird—it sounds like I'm dismissing that—but actually fascinating use cases. We've seen in biotech, for instance, companies having—I don't know bio super well, so this might be a gross, generalized, bad sort of characterization—but scanning millions and millions of compounds for certain chemical properties, or medical imaging where customers are running computer vision on millions of medical images.

So there is a lot of sort of batch processing also in those use cases I find fascinating. Not just sort of data pipelines, which is what I think people think about with batch processing, but also all kinds of other interesting applications around biotech or video transcoding or feature extraction and things like that.

Yeah, we started seeing a lot of customers who build LLMs needing, especially people doing LLMs for code, needing safe code execution. As it happens, we had great primitives for that because that's effectively what we built, right? We built a system for containing user code and executing it in a safe way. So we started thinking about, can we expose that as a service? So we also have primitives to take sort of untrusted code and execute that in a safe way. And I would put that in the sort of category of unproven.

It's still sort of early days. We're trying to figure out exactly what that market opportunity looks like. But there's so many startups building LLMs for code, whether it's code generation or migrations or things like that, or debugging or things like that. So to me, that remains an unproven but potentially very large upside market opportunity for us that I'm pretty excited about.

I think it's funny because I feel like serverless was all that hype. People were really excited about it when Lambda came, and then it didn't really catch on. And 10 years ago, when Lambda came—I actually don't remember what year Lambda came, but I think it's about 10 years ago—I probably would have expected in the future 90% of web development will obviously just be—people don't want to think about containers and processes and things like that. And that didn't really happen.

And I've thought a lot about why. And one of the reasons I've kind of realized maybe the reason it didn't happen is actually the developer ergonomics around building backend services is actually kind of fine. If you're, like, a backend developer just deploying with Kubernetes or running things locally, it's actually not so bad. It turns out moving things to Lambda is actually kind of worse in many ways. If you look at data teams, though, like data, AI, machine learning teams, I think we never figured out developer ergonomics for those companies, for those teams.

And 10 years ago, I was the only data engineer, the only data scientist at Spotify. It was kind of fine because it was such a small fringe part of software engineering that probably didn't really matter. But now there's a lot of people building data, AI, machine learning applications. And I think developer ergonomics and developer experience matter so much more. And as it turns out, actually, serverless kind of makes sense, I think, for a lot of the applications that those people are building.

So serverless always felt to me like, in hindsight, it looks like the right idea but applied to the wrong problems. And I always wondered. Now, looking at what we built at Modal, I'm kind of convinced that serverless actually makes a lot more sense for data, AI, machine learning applications for a lot of different reasons. You have very bursty things, you have a lot of different containers, you don't necessarily care about hyper-low latency. I think there's a lot of different reasons why it actually makes a lot of sense for those types of use cases.

Yeah. And it's something we're still debating to a large extent.

And by the way, I think of self-hosting not just as a binary thing. It's kind of a spectrum, right? There's on-prem on one side, there's fully multi-tenant on the other side, and there's a bunch of stuff in the middle. There's VPC peering, PrivateLink, cloud-prem, BYOC, bring your own cloud. So it's actually more of a spectrum, but we've been cloud maximalists, and we've been all in on, like, let's just build a multi-tenant service where we just run everyone's code in our cloud environment.

And to be clear, a lot of potential customers are just not comfortable with that model. I tend to think about that as like the cloud itself. I'm old enough to have started my career pre-cloud. And I remember the first time I heard about AWS in 2007, 2008, or something, when they launched, my first reaction was like, how could anyone ever run their code on someone else's computer? That's nuts, right? And then just a few years later, I was doing it myself, and I was like, this is awesome.

And then I think there's sort of a similar thinking. I met with the Snowflake team when they were still building the product super early, and I was like, why would anyone put their own data in Snowflake's data warehouse running in the cloud? But now they have very large enterprise customers that are very comfortable with that. And so I think the general wind is blowing in the direction of multi-tenancy and security moving away from the network layer into the app layer and a bunch of different other things.

And I think one trap that a lot of startups fall into is that you go out and talk to a lot of customers and they sort of insist on self-hosting. In my opinion, sometimes that's actually true, hard concerns. But in many cases, it's actually more of a soft concern, and it really just means you haven't built a product that they're desperate enough to use. And so maybe one day we will open up and offer other deployment models, but I'm actually more interested in the other option.

Like, how come we haven't built a product that they're desperate enough to use us? Right. Can we push more in that angle? Can we make it so good that they're willing to waive the typical concerns around deployment model and embrace this fully multi-tenant model where we run everything in the cloud? Because the benefits are tremendous. I really think resource pooling is one of the free lunches that exists in cloud economics. You can get much faster capacity, much more liquid capacity, et cetera.

If you have a lot of different people and they all have very volatile demand, if you just put those pools together, you can actually scale it much more efficiently and drive much higher utilization than if you separate it and say each one of you has to run your own pool of compute. Right.

I think it's just easier to grow your business that way, right? First of all, for a lot of products, it's not necessarily a core impact on revenue, right? If you're building a SaaS business and self-hosting versus multi-tenancy, if you fundamentally charge roughly the same thing, or maybe even more for self-hosting, I don't know, then it doesn't really make a big difference. And if the quality of the service is kind of similar, then why even bother insisting, right?

Let's just go and meet the customers where their demand is. For us, I think the product is much better with the multi-tenant service, and the revenue potential is much larger because we have full control over the underlying COGS, like the underlying cost, and we can do all these optimizations. So for us, I think there's more of a case to be made for pushing for this thing. For many other types of business, I don't think it matters that much. And so I think they're better off just meeting the customers where they are.

But I do think there's a lot of customers, also a lot of vendors out there, that maybe should push more for a cloud-maximalist approach, because I do think that's where things are going. Who knows? Maybe I'm wrong. Maybe in a year or two, if we do this podcast again, I've been giving up and just banging my head against all these late-stage enterprise customers, and I've just given in and said, actually, you know what?

There's a big business in on-prem. And maybe that's fine, I don't know. But right now, that's just not where I want to focus my energy.

Yeah, I think fundamentally, through our software, we have the ability to charge more than the underlying GPU costs. Now, how much more is the question, right? Customers would probably freak out if we charged 10x more, because then they're just going to go, actually, as much as I like the developer experience of Modal, I'm not going to pay 10x. I'm just going to go deploy it directly on whatever, Lambda Labs or something like that. And so the question is, how much more can we charge?

And it's probably like 2 to 3x. I think one of the things I always think about is, are we a WeWork in the sense that do we make long-term GPU reservations and then we have short-term sort of income, and then we have this massive sort of exposure to the underlying costs?

Yeah, I think probably the answer to that is we need to be a little bit more conscious about not spending a lot of capital on very long-term reservations, especially in this climate right now where no one can predict the GPU prices. GPU prices themselves are kind of strange, right? For a while, H100 prices were almost going up, and then they were kind of still for a while, and then now they're actually going down quite a lot, from what I'm seeing in the market, right?

And so we have to be cognizant of that and lower the price. Does that make us a commodity? I don't know. I look at AWS. You can argue, is AWS a commodity? Are they not? But at the end of the day, EC2 is like, what, like 50%, 60%, 70% margins? It's actually not as much of a commodity as people think. So I think the value of software and the value of creating this pool of liquid capacity lets you actually have a decent amount of pricing power.

And in the long run, I also think there's many other add-on services we can charge for as we go up the stack that let us retain decent margins. I also think a lot about GPU prices. What happens if they crash? What happens if they go up? Are we long or short GPU prices? Fundamentally, I actually think that since we're sort of—the value we create is like the software on top of the GPUs—that in a way, if GPU prices were to crash, the relative value of that software actually goes up.

So I'm not necessarily sure that a crash in GPU prices would be bad for us. I think it actually could be great for us because if GPU prices go down, there's going to be more applications using GPUs, and then people are going to pay more for software, just like they do for CPUs, by the way. Margins on Lambda are much higher than margins on GPU serverless products. So I spend a lot of time thinking about this, but I do think there's a lot of pricing power in building infrastructure as a service in this space.

I think we did a few things right, which is, like, one was just me building kind of the product I always wanted to have and just being kind of opinionated about what that looks like. And then not being afraid to actually go deep into sort of layers of infrastructure and change things that sort of got in the way of delivering that experience. To me, it's like we live and die by the quality of the ergonomics of the tool. And so, to me, that's like the core competitive advantage.

That's the sort of core ethos of the company. We cannot absolutely lose that. That's why people pick Modal. Everything else is sort of secondary. I also have a long background in building consumer products my whole career, so I think that's maybe helped kind of thinking about what is the onboarding experience like? What does it feel like? Sort of unboxing, when you get started, download a client, install it—what does that feel like? I think that's kind of often like a missed opportunity to create the sort of set the expectation that this is a magic tool.

So, spending a lot of time thinking about that, I think it also sort of inherently led to more of a bottoms-up go-to-market approach. We're so focused on developer experience, but yeah, just being obsessed and maniacal, thinking about what the SDK looks like.

Yeah. So what I wrote about is generally just like, how do you write code when you're writing code for humans? When the actual end user of that code is a human? I'm talking specifically about packages, frameworks, even programming languages. How do you think about code when the end consumer of that code is a human? So how do you think about their mental model, their onboarding experience, making it easy for those people to take your code as building blocks and compose larger things?

And a lot of those things are just kind of things I've been thinking a lot about for the last 10, 15 years, building things like Luigi and other things, and Modal, of course, now. And to me, it's actually not like rocket science. There's a bunch of very core principles of just, how do you craft this experience that makes a great experience? And for Modal, it's so important. We focused everything we can on that. But I actually kind of feel like it's somewhat repeatable for many other people too.

Yeah, exactly. I think there's so many products where, like, so many dev tools, or you go check out some code on GitHub and it's like, maybe this is interesting, maybe not. And you're like, all it says in the README is how to configure it and how to install it, but not actually showing an example of how to run it. And I'm the least patient person in the world, and there's like 50 billion gazillion dev tools out there and so many GitHub packages and whatever.

So when I encounter something like that, I just don't have the energy to sort of visualize, how does this fit into my codebase? And so I think it's such a lost opportunity for all these dev tools. Show me some examples. Show me how to build something. I don't know, maybe my mind is different than other people. I just don't understand when I look at core concepts, stuff like that. When I look at a dev tool and it starts out introducing a bunch of weird things, I just don't have the energy to kind of visualize how those would fit together to solve the problem that I have.

I'd rather just look at an example that sort of reminds me of the problem I'm trying to solve. A lot of this actually goes back to my time at Spotify and kind of what Spotify optimized for. It's kind of a weird, very different product, obviously, but it has a lot of analogs. And one of the things I think Spotify did incredibly well was the sort of first experience of, like, you download Spotify and you search for music and you double-click on it and it just plays music.

And you do that within a couple of seconds, right? It's so obvious, it's so visceral almost. It's funny, it actually goes beyond that. You're like, is this music streaming in the cloud? But it feels like it's local, right? And that's, in a way, exactly the sort of experience I wanted to have with Modal, right? Is this code running in the cloud? No, it's actually running—or it is running in the cloud, right? And nailing that sort of first experience and setting that expectation for this is how the entire product is going to be, to me, that's a big part of how do you sort of make a magic developer experience.

As well as many other consumer products. It goes well beyond developer tools, but I build dev tools, so that's why I write blog posts about that. So to me, that's a lost opportunity in many dev tools, and that's why we spend so much—I mean, to the point of having a bunch of frivolous stuff. We have silly confetti stuff in the web browser and stuff like that, which maybe went too far, but I'd rather go too far, like making the onboarding experience magical.

Because it really sets that expectation.

Yeah, we probably should do that even more. But yeah, we definitely cut out a bunch of features that are rarely used in order to simplify the SDK. If something is not really used by more than some small minority of customers, I'd rather just get rid of it and merge it into some larger concept and make it a special case or something else in order to reduce the number of concepts you need to learn.

Yeah, honestly, it's like Twitter shitposting.

Yeah, exactly. Reinforcing it. I don't know, I'm trivializing a little bit. I think I have a background in consumer products, and I had always been tweeting and blogging and stuff like that. So I think every company has to find their own channel mix and their own go-to-market, and everything is idiosyncratic, and there's no thing that really translates. For us, it just turned out that targeting early data influencers and talking about the product and giving away hats online on Twitter, that's always worked for us.

And I think now, more recently, we hired our first salesperson six months ago, and I'm trying to figure that out. And to me, that's a whole different world. Maybe that's a different conversation we should have in a year or two, what that looks like. I don't know, but I think that's the next step. So almost everything is inbound so far.

Yeah, there's a fair bit of that. I also think fundamentally, when you look at later-stage companies, I do think that a more traditional sales model will make sense for those companies. If you're selling to, an extreme case would be Bank of America. You can't expect them to sign up using GitHub and just start writing code and deploying things and swipe a credit card, right? There's going to be some procurement process, there's going to be some proof of concept, there's going to be a bunch of other stuff.

So we're trying to figure out exactly what that looks like and how it fits into our model. We've been very happy with the inbound and just kind of upselling, and that's been enough to get to the point where we are. But I also think layering on some sort of top-down will help us get to the next layer. To me, actually, the best companies that I look at for inspiration would be Datadog or Mongo or something like that, or AWS, actually, frankly, started with a product that customers loved and nailed the bottoms-up, self-service process.

But then also, once that got to a certain point, started layering on more of a top-down, traditional sales process.

I think that in the early days, actually mid-stage too, you can get very far with almost no management. And I look at a lot of the lessons from early Spotify days. Spotify probably went too far. No one told me who my manager was for the first year at Spotify, which is probably bad. But I think it's sort of an extreme example of the freedom under responsibility that existed in the early days of Spotify. At the end of the day, people were super commercial in building the stuff that they thought was the best for the company.

And kind of self-organized around that. There's a little bit of Scrum and stuff like that, but actually I think that was kind of bad. So, to me, I'm not naive about it, but I think aspirationally, you can get very close to self-organization if you just hire fairly smart people, very entrepreneurial people, competitive people, and just give them the context. You just tell them, this is what we're trying to build, and then you go figure it out. And then obviously, as you grow, you're going to have to kind of put things in different swim lanes and have a little bit more project management.

But you can actually get much further with a lot less of that than people think, is my experience. And you're actually better off doing that because you sort of retain people's autonomy. You retain people's ability to sort of come up with solutions themselves, especially when you're hiring smart people. They want to do that, and you should do that. You should harness the creativity of the sort of the largest surface area, which is the individual contributors.

Yeah, I mean, I think you can look at the final product. You can talk about what they're working on. I don't necessarily believe in those sort of estimates or deadlines or things like that. But I do think there's an element of looking across the portfolio of your current projects and continuously adjusting. Are we spending too much time on this? Are we spending not enough time on this other thing? And sort of continuously adjusting that. Every month, we do a lightweight sort of set-the-priorities-for-the-month, and here are the things we're trying to accomplish, communicate with all teams, make sure there's nothing that's missing.

Sort of the, let me see, we have everything covered and there's nothing that's missing. But that's like a Notion doc. That's like super, very basic. And then we do track things in Linear, but for most of the sort of day-to-day management, it's like people just come in and write a lot of code.

Yeah, it was just a kind of reflection of how I've always built functions at startups, is you can skip steps, but it's generally kind of a bad idea to, let's say, you're not doing sales and suddenly you go from that to, let's hire a VP of Sales. Now you're skipping three steps, or maybe more, right? The correct way, I think, to build up sales—and by the way, this is probably the thing I have the least experience with—but, or like HR or finance or whatever, you start doing it yourself, and then eventually you hit a wall and it's taking up way too much time.

Then you can delegate to someone, or you can sort of find a part-time contractor or something like that. And then eventually that takes up too much time, so you convert that into a full-time person. And then eventually you realize you need more people, and then you bring in someone else, or you bring in additional people. You should manage that team initially too. And then eventually you promote one of those people into a leader, and then you can manage fewer people. But I think skipping steps in that sort of growing teams is kind of dangerous because how are you going to manage, say, a sales team unless you know how to run sales and how to do sales yourself?

To me, that seems very hard. I think you kind of need to do it yourself first. Not for 10 years, but do it for a little bit. And then eventually you hire people and promote people, and then you can manage the people.

I mean, I only built a company in New York, so I don't really know. I'm sure there's many reasons to build things in San Francisco or elsewhere. I just happen to live in New York, and I think New York has an amazing talent pool. If I lived in, whatever, Jacksonville, I probably would have moved. But I think New York has such a good talent pool that you can totally build a company here and build a very large one and be very successful, as evidenced by the companies you mentioned.

So I'm very happy to—I just love the city. I have a wife and kids here. I'm not going to move in order to start a company. And luckily, New York is a great place to build these types of companies.

I think tech in New York is amazing. I actually think, on that European thing, I grew up in Sweden, right? And I still have a network in Sweden. So one of the things we've done at Modal is to hire a bunch of people in Sweden. That's actually something I think New York is very good for, right? If you're a European founder or have access to European talent, we have many examples of that. I think Datadog has a bunch of people in France, for instance, because they're French founders.

And with the time zones, it's much easier than SF, right? Because it's like you have a six-hour difference, you can actually have a decent overlap and coordinate across the time. And there's such amazing technical talent in Europe, especially low-level stuff, Linux containers, whatever, the kind of stuff that we need. So for us, that's been a great sort of competitive advantage, is having a team in Stockholm. It's still pretty small. There's six people, but they're very core contributors to the infrastructure.

I hope to build this for the next 20 years. So I think there's so much stuff we want to build right now. We're focusing a lot on just taking the existing compute platform and improving it in various ways. One thing, for instance, I'm very focused on is how can we get into more real-time, low-latency use cases, which is kind of an annoyingly complex technical problem. We've sort of relied on a centralized control plane running in one single region, us-east-1, but we sort of recognized, in order to get to latencies below 150, 200 milliseconds, we probably need to split that up and run a decentralized control plane.

So that's going to take a long time too.

Speech-to-speech, I think, is really exciting, like real-time translations or voice manipulations. There's a lot of interesting use cases around that, or video, actually, is another use case. So low-latency stuff is something we're really interested in focusing on. We're interested in kind of pushing the boundaries of very large-scale batch processing. Modal right now, when you start doing millions of function calls or billions, you start to run up against limits. But we're very interested in how can we push that into millions and billions in a reliable way so you can truly run this very large-scale batch.

That's another thing we're very focused on over the next six months. What else? I think as we go up this, as we sort of mature the compute layer, I think there are many opportunities to go up the stack and build more enterprise features, build more observability into the tool. We're very interested in deepening our enterprise security compliance offerings and the less technical side, like SOC 2 Type 2, and we're going to be available on the AWS Marketplace pretty soon.

Things like that, I think, is kind of exciting too.

I don't know. It's, like, weird, right? To me, the best analog would be either, like, fiber optics or, like, mobile or something like that. I still think they're kind of, like, bad analogs. But one thing I, for instance, thought a lot about is the applications that, in the long run, tend to be sticky are the type of things that really truly take advantage of things that couldn't be built previously. And that's why I'm really excited about Suno, for instance. Suno is a product that clearly couldn't have existed a few years ago.

And I also think music is kind of always, like, a good sort of place where you tend to see new applications pretty early. And similar to mobile, I've been asking myself, what's the sort of Uber or Spotify or Instagram of AI? What's the sort of consumer application that is AI-native in a certain way, right? And I don't know if it's, like, ChatGPT. I think ChatGPT is fantastic in many ways, but it feels like, I don't know if it's, like, long-term, it feels like I'm looking for something that's a little bit more consumer-y and something that's a little bit more vertical.

And is truly AI-native. And I don't know if we've seen that quite yet. I think something like Suno is probably the closest I've seen in that vein. But that would be what I would be looking forward to.

It's great. Yeah.