AI Engineering Revolution: Winners, Chaos & What’s Next | FirstMark

Yeah, well, thanks so much for having me on. So, quick context: I was asked to pull this presentation together for the CTO summit that we threw last week. I would say it is one of the most interesting times in engineering, in this broader ecosystem of technology. And so maybe I'll just kick off the presentation by talking about a quick summary of how we've arrived at this generative AI moment in our engineering ecosystem, by giving a quick highlight of, call it, the six trends or so that have demarcated the last two decades in the world of developers.

So I'll start with pre-cloud, when we used to just deploy code directly onto servers.

Correct. And then around, call it 2010, along with cloud, DevOps finally got coined, which was really the merging of dev and operations into this whole subsequent toolset and set of workflows that emerged around it, like Git, CI/CD, et cetera, that we all know today. And that very much spurred this generation of SaaS explosion. All of the small-, mid-, large-cap companies today that we think of as being venture-backed very much belong in that category of company. And then two sort of things between then and now that I think are worth noting.

One is the proliferation of APIs, which essentially gave all of those SaaS applications the ability to very quickly expand their capabilities by almost outsourcing things like payments and messaging and search to companies like Stripe and Algolia and MessageBird. And then two is abstraction. So you think about companies like HashiCorp and Snyk and dbt and Vercel very much catering to an audience of developers who are using code interfaces to all of a sudden do things that historically have been very much outside the purview of developer workflows.

And so think about infrastructure provisioning with Terraform, or think about dbt when all of a sudden we're now data engineering through a code interface. And over that time, so from, call it, 2006 all the way to now, estimated about 7x growth in the number of global software developers, which I think of as sort of an index to this tech ecosystem. And that's where we've arrived today. So generative AI, this massive shift in delivery model, in the way that developers work, and one that I, in this presentation, really talk about being probably the most impactful thing on this page.

Yeah, I think you nailed it. The first point you made is really interesting, which is this corpus of open-source code that all of these models could use as training data, which is very much true. And actually, I think later in the presentation, we'll talk a little bit about some of the bad or somewhat negative effects of that corpus of data that those models have been trained on and how that's created specifically security concerns. The empirical nature of code has certainly made it a great early use case for AI.

And then the last thing I'd say is just developers have always been the type of persona that loves to try new things. And so, as a buyer and a user, they always made a ton of sense for AI. And obviously, across the broader world of consumers, it's been well documented that ChatGPT has made its way into millions of people's hands. So it's not to say that AI is not a mass-market phenomenon already, but I think from an enterprise perspective, they've just been a great buyer for tools like this, and the results have been very much measurable.

Yeah.

Yeah, absolutely. And so maybe to kind of exemplify a lot of what we've been talking about, we're about 24 months into this wave, give or take. It's changed a ton of behavior. It's taken our community by storm, but it's also just created a ton of really, really interesting, special companies that have grown especially quickly. And so Cursor, maybe three weeks ago, announced they're at $500 million of ARR.

I believe so. Lovable, which allows people to create prototypes and web apps through a series of prompts, has gone from $0 to $60 million of ARR in the last two quarters. GitHub Copilot, which I know you just had Thomas on the podcast not too long ago, $400 million of ARR, 15 million developers, very much the steward of this category, first mover.

And recently has had a large resurgence in a new form. A couple other examples. So v0, again, you just had Guillermo on the podcast. This is for Salesforce.

Windsurf, which is rumored to have sold to OpenAI, hit $100 million of ARR quite recently. And then Replit, which went from $10 to $100 million of ARR in just the last six months.

Yeah, it's fascinating. Historically, they were known for being very much in the hobbyist, indie developer, educational setting. And what's interesting is actually that is the setting where people are now growing up in an AI-native way from an engineering standpoint. And so, in many ways, doubling down and betting on that distribution channel over the long run has actually probably created the moment that they're having. Yep. Well said. So anyways, point is just to say, this is a very special group of companies that is ramping quickly and has no sign of stopping.

And there really is no plateau in sight for innovation. I mean, it's just, for all of your listeners who spend their time on Twitter and TechCrunch and VentureBeat every day, it can feel quite overwhelming.

Never. My argument in this presentation really is that behind all of the marketing gloss, there really is more substance than there ever has been before. And if you—I hate to say this—but if you go offline and you aren't keeping up with the updates for a week at a time, you really might miss something spectacular and industry-shifting. And so far, we've seen really remarkable, tangible results across our network of engineering organizations. So hopefully this can just put some numbers to, at an organizational level, how code generation is really changing engineering process.

We've seen 30% to 50% faster throughput. We've seen a 12% increase in PR merges. This is a really important compounding stat here on the bottom left, which is a 17% increase in the amount of time folks are spending on roadmap versus maintenance and keeping the lights on and support. And then, in my mind, most impressively, 82% of people we've spoken to are already using AI to write code. And so that is just, again, an adoption curve that is pretty much unprecedented.

And I think we can talk more about why that is, but I think predominantly it lends itself, again, to the audience and the arena of distribution: IDEs, where people live today. These aren't net-new interfaces for the most part, very much embedded in people's style of working, and again, with very similar distribution mechanisms and keystrokes and delivery models. And so it's just been a fascinating couple of years.

Yeah, I would say today what we're seeing is this spectrum of people and use cases that vary by experience and complexity. And for now, many of the agentic solutions that are truly end-to-end, complete something, are very much being pointed at lower-level tasks that I'd say are, quote-unquote, more mindless, that have less dependency complexity, that have less of a need for context and knowledge. And that's worked very well. So yeah, there really is a diverse array of ways that you can apply AI to this problem.

And I think many of the best organizations and engineering leaders I talk to deploy all sorts of things across their stack, depending again on the use case and the people. I would say, despite all of these amazing numbers and stats, I want to pose sort of a historical analogy here to draw to this space, basically to say that with production surges, it tends to be the case that actually a lot of problems emerge, and in their wake, markets follow.

Or maybe a punchier way to say that would be: with every surge in production, there's just a cleanup crew that naturally comes, and a new market and industry that follows in its wake. Yeah, so if you'll allow me, we're going to go back in time as far back as the 15th century and talk about a couple examples where we've seen this happen and how we might be able to draw some analogies to today.

So, starting with the Gutenberg press, this really was the advent of our ability to print books at scale: 3.6K pages being printed per day. I think over the next 60 years or so, there were 20 million books in circulation in Europe, up from 3,000. Big disruption, right?

Yes, they turned to making Chartreuse. So anyway, this sounds like a great thing, right? I would hope we all love books. But actually, like many of the content issues that we see today, there were a lot of challenges that emerged, namely misinformation, mass reproduction, quality issues, a bunch of informational overload. And so we saw a bunch of new industries emerge in the wake of the press, which were all the things that we think about when we think about books: printing, publishing houses, editors, libraries, now for consumers to deal with the abundance of options that they have, almost serving as physical indexes of the many books that they could access.

And then a bunch of regulation that came around licensing and censorship. And again, all the things that we associate with content today. Another example, almost 500 years later, was the Ford assembly line. So in the early 1900s, Henry Ford invented the first continuously moving assembly line, and we dropped the time per car, assembly-wise, by almost 90%. And so over the next 20 or so years, the output of Model Ts soared to about 10 million in the U.S. And we went from a society that was mostly horse-and-carriage, railroad, and trolley-driven to all of a sudden being a car country.

So we still have car challenges today, but if you can imagine then, they felt much more extreme. Things like infrastructure strain, safety issues, the need to build all of the roads and infrastructure to support this new economy, factory workers' rights, environmental issues. And so again, we saw these industries emerge. I'll highlight the ones that feel very much endemic to auto: quality inspectors, mechanics, dealerships, gas station attendants. And then on the regulation side, all the licensing, our driver's licenses, our license plates, traffic police, and again, all of the build-out over the following 50 years or so of the freeways, highways, roads that we all use and drive on today.

And so there are a bunch of other examples of this in history. I sort of arbitrarily chose those two, but railroads, mapmaking, the postal system. I think smartphones is probably the most recent example, where all of a sudden we have tons of compute in our hand and the ability to take as many pictures as we'd like in a day. And in its wake, you've seen a ton of industry emerge, mostly social media, influencer marketing, et cetera.

And then all of the regulation around privacy and biometrics that still feels very top of mind today and that has been evolving over the last decade or so. And so again, I would just posit: as production surges, you see all of these problems come in the wake of that production, and then new industries come around. And so we didn't bury the lead here. If you think about code, I would argue we're very much seeing a similar trend where, on the canonical DevOps cycle diagram, you're seeing code commit just accelerate so much.

And today, at least, many of the processes that are downstream of that—our CI/CD pipelines, our testing suites, our build infrastructure, the way that we think about observability and monitoring—have very much stayed the same, but it's kind of breaking. And so I'll talk a little bit more about that.

To put some numbers to how these pressure points are starting to begin to explode, we're just seeing a lot more time spent debugging. We're seeing a ton more security vulnerabilities, which we alluded to a little bit earlier when we were talking about how these models have actually been built and on what data. We're seeing a ton of performance issues emerge as a result. I know you did your podcast with Guillermo. I actually did an event with Malte Ubl, who's the CTO at Vercel.

So this was a great event for our CTO Guild. And Malte, among many interesting things, I think one of the most fascinating things that he said was that most of his great engineers who have been with the company for a while, as they've dogfooded v0 and they've used things like Cursor and Windsurf in-house, is that many of his great engineers are actually becoming predominantly professional code reviewers. And this notion that as AI codegen has just totally increased the rate at which they're producing and outputting code commits, you need people who almost act as people sitting in a toll booth, letting cars pass or not pass, very much doing the same thing with code, but people in those seats that have great taste and know what bad or good or great looks like.

And so I thought that was sort of the tip of the iceberg. As you think about, you have these large engineering organizations that have all been trained somewhat similarly on how they should all think about working together and what sorts of jobs that certain people should be doing and not doing and how teams should work together. And we've seen this really, really fast shift where individual ICs who used to be green dots everywhere on their GitHub repo are all of a sudden becoming code reviewers.

And so, yeah, I thought that was a really interesting analogy, and I'll expand more on it.

We did a study at FirstMark across, I think it was a little over 300 engineering organizations, and we asked them just sentiment basically on codegen and how it's made them feel about certain metrics that they use to measure their engineering output.

Yeah, so I would say everything in purple, unsurprisingly, which is positive, is very much linked to speed and efficiency and velocity. And so most of these stats are measuring, in some way or another, time spent to develop something, the number of times in a certain period that we commit, how often we're deploying. And so all of those things have been very much sped up in this age of AI. But what's interesting is many of those things don't take into account, like, what happens afterwards.

And so again, we could make as many commits as we want, but that's very much just the first step in the DevOps process as it exists today. In blue are most of the negatively impacted areas across the survey that we did, which are mostly reliability- and process-oriented. And so, take mean time to restore, for example. If something goes down, how long does it take to restore it? Well, all of a sudden, if we don't have provenance over the piece of code that's relevant to something that happened—an incident—it's much harder for us to go back and say, "Oh, actually there was a bug over there."

And so that's been what's generally interesting. I think this slide really just goes to say, again, everybody's getting sped up. The amount of commits is skyrocketing. But all the stuff that we measure when we think about efficiency, and all the downstream things that we want to happen post-code commit, seem to be faltering.

Yeah. So I can be a little more specific here in terms of what's breaking. Yeah, let's do it. We can start with security. Put simply, we're just seeing more security vulnerabilities than ever before.

Yes, I would say new types of security vulnerabilities, especially given—and this is sort of separate from the conversation about coding—but given deepfakes and all the stuff that's happening in the arena of email right now, certainly on that side. And then also in the world of codegen, to our earlier conversation, many of these models have been trained on really large bodies of open-source code. And many of those pieces of code share vulnerabilities and bugs that you can't see coming in the same way because, again, you're not operating with the same level of decisiveness and meticulousness.

And so as a result, you're seeing things like CI/CD pipelines and build systems completely breaking. We're seeing—I mean, folks have had brittle CI/CD pipelines for a while now. We've seen a lot of companies emerge over the last decade to ameliorate that and change the way that we think about CI and CD. But the flake rate is just much higher. And that's true of testing too. So flakes basically are just your ability to either trust or not trust a pass/fail result.

And if you have a flake, it essentially means something passed and then it failed, but the inputs were the same. And so we're just seeing flakiness generally across code skyrocketing. I would say build has come under a lot of duress. It's just much harder to cache code, which a lot of build tools try to do, when codegen is relatively stochastic. And so again, the notion that, given the same set of inputs or similar inputs, you're just not going to get the same set of outputs.

And then QA and code review are areas that we've talked about, but we've seen QA processes just completely overwhelmed. And on the code review side, we're just seeing a new focus on code review that is pretty unrelenting, that I'd say has gone from a step in a process to very much a job in and of itself.

Unfortunately, for those people viewing the podcast who are engineers or practitioners, all of the problems that I'm talking about are very much opportunities for us as investors.

And founders. And we see a ton of them across all of these spaces. So I won't go into every single one, but I'll call out a few on the security side. So there are a lot of companies, older companies, Fortune 1000 companies, that use EOL software. So maybe they're on a very old version of CentOS, or maybe they're using an old package, an old version of an open-source library. And the notion that they could just upgrade is actually quite a difficult thing to do.

And so we're seeing a handful of companies that have actually pioneered what's called auto-patching. And so, the ability just to, instead of forcing folks to make a potentially breaking change, patch a project and let things stay status quo and smooth. That's a capability that—you'd be shocked—there are thousands of engineers that have spent days, weeks, months trying to just keep something afloat.

Because of an old dependency or an old package.

Yeah. Across QA, we're seeing a lot of interesting agentic solutions that are reasoning based on essentially semantic requirements. Better to almost tell the solution what the solution needs to be than to try to do any sort of diff analysis or anything else that, again, given the stochasticity of AI, actually might take you down a dark place. And then in code review, we're just seeing a lot of tools that are getting really fast adoption that are doing essentially agentic first-pass checks for code.

So yeah, the space is moving quickly, and I would imagine if we did another check-in in six months, we'd probably be talking about a whole new slew of problems, but also a bunch of solutions that have addressed many of the things that we've talked about today. It's moving fast.

I'll use this a bit to shine some light on a handful of the companies across these different categories.

We'll find a better name for this at some point: the MAD Landscape.

You just throw this in the MAD Landscape, right? But yeah, it's been fascinating. And maybe I'll use this slide just to say this is one small sliver of the total companies in this market. And again, back to the point about us being on Twitter and reading blogs and news articles, there are probably a dozen companies that come out every day that sit in this space. And so it's made this job both exhilarating and fun to spend time in this category.

And then most interestingly, I would say, we've talked mostly about engineers themselves, but I think for CTOs, this is going to be sort of a hallmark moment for them over the next five years or so, where they have a lot of important decisions to make across talent, architecture, team structure, governance principles, security. I wanted to take the opportunity, especially given we were talking to a room of CTOs, just to sort of talk through those things and see what's different.

So I'll start with hiring, and I'll share an anecdote that I had read recently, which is that computer science grads are actually among the top five or six majors graduating from college right now with the highest unemployment rate, which is—I mean, it is shocking. And I think it speaks in many ways to the fact that despite the huge demand for software engineering as a concept, the people who are trained in that practice are actually not in high demand.

And so I very much think, whereas we used to focus on hiring these canonical 10x engineers and developers who can write code, we're seeing CTOs very much focused on hiring great editors and reviewers and prompt engineers who can almost shape and validate and curate that AI-generated output.

The idea of prompting, too, is just fascinating. So I was having an interesting conversation with an engineer the other day who was talking about the ability to one-shot Salesforce. And by one-shot, they mean, like, could I recreate Salesforce in just one sitting with one of these tools? I mean, obviously the answer is no, but if you ask why, it's mostly because actually, like, if you're trying to recreate something through a series of prompts, you actually have to really understand what it is that you're creating.

And most of us, when we do that, we're starting visually. We think we understand maybe the database logic sitting underneath an app. But Salesforce is a fantastic example of a very, very complex, deep app with a ton of different use cases and integrations and reporting structures. And so it's not just the reviewing side, I think, that will evolve, but it's also just how we think about prompting and that being a skill set.

So that is fascinating. I think on the architecture side too, we're seeing a similarly sized shift where we used to sort of define these systems via design docs and then a bunch of human-enforced conventions that might have sat in a Confluence doc somewhere to follow. Whereas today we're seeing just this huge push to define systems in a very machine-enforceable way where you're almost setting these IaC-esque guardrails that AI can just conform to and see and know in a way that is much more stress-free for a CTO who's putting, I mean, let's say, like, 10 Devin engineers to work.

So that is also an equally large shift. We alluded to this too. I'd say the third thing is team structure. So we used to, to your point about frontend engineers and backend engineers, we used to just organize teams in a very structured way. And one of the fascinating things I think that's come out of AI, generally speaking, is we're just seeing companies and teams being able to do a lot more with a lot less. And I think that's especially true in the engineering context, which, again, speaks to this college grad stat.

But we're seeing smaller teams where some people just review and manage AI-written code, and that's very much okay. And it spans from front end to back end to systems and everything in between.

Yeah, I mean, historically, EPD—engineering, product, design—has been a super complicated hierarchy system in companies where you have a ton of different handoffs between engineers and designers and product people. And company to company, it always looked very different, whether product people interface directly with engineers at all, whether they were really there to ideate and then hand off entirely. And so, to your point, as the technical barrier to create something has dropped, it is straining and changing the way that those teams operate together quite a bit.

And your story from Canva is exactly right. This notion that we would do a peer review of that much code—there needs to be a change in the way that folks think about governance, and this culture of reviews in general is changing very quickly.

Yeah. Enforcing constraints.

A big thing we've seen come out of this is this notion of provenance, which is, in layman's terms, essentially the lineage of code. So who owns it? What are the dependencies? Where did it come from? And increasingly, again, with just the amount of volume that we're seeing across code in general, and then the new sort of sources of code that aren't coming from specific human identities in an organization, this notion of provenance has actually been pushed to its outer limit and remains, from a governance standpoint and for a bunch of downstream processes, just super important.

The last two things I'll talk about are velocity and security. So, on velocity, a lot of engineering folks have historically talked about eliminating bottlenecks. I mean, it could be any number of things that cause a bottleneck in that entire DevOps flow. Whereas now, there's very much a focus on containing this idea of entropy and then managing, from a governance standpoint, the coherence of AI systems to system architecture. That's been another really interesting theme, where you have these broad-based products that are being used by everyone, but everyone's stack looks very different.

And how can you conform and give context to an AI product, especially in this world, for what the constraints need to look like? And so we've seen, from a velocity standpoint, these notions of entropy and coherence become much more important today than they were two years ago. And then we've talked about security, but I think we'll see a lot of things that come out of this space that look very much like products that are actually catching bugs and problems at the time of write.

And that will be a very interesting shift in the ASPM world. So maybe to bring it full circle, again, to this analogy about productivity surges and booms, we see a ton of challenges that come from those historically, and we see new industries come in the wake of those challenges. And I think we're very much, for now, seeing that in codegen. And so I'm as excited as I've ever been to be an investor in this space.

I think it's rare to see problems happening as fast as the companies that are solving those problems grow. Everything is just moving so fast. And so, again, I think in six months' time, maybe we'll get back together and this will look either like something that's been completely solved or something where the problems have shifted in form factor completely. But yeah, it's been fascinating.

Yeah, look, like many spaces in AI at the application layer, many of these products started out with what I would say are very simple delivery models of technology that wasn't quite their own, with really smart distribution strategies. And I mean, in many ways, I think the IDE companies are pushing the outer limits of what zero switching costs could really look like. But what's interesting is now they all have a war chest of money, usage, and talent, and they are going after much harder problems that are very much differentiated and proprietary.

And so it will be fascinating to see how the space changes over time. But I would say the parameters within which we've operated thus far have been very much unique, right? I mean, we're talking about forking VS Code, which is what Cursor did. That is not something that I would have seen coming if you had asked me in 2018 to imagine what generative AI would look like, right? But that has been, thus far, the most impressive business that's been built in this space.

And they've done a phenomenal job.

I think it's always incredibly easy to say the alpha's left the room, especially if you're the person who wants to start a company and you don't know where to begin. Again, we talk about being overwhelmed by headlines. I couldn't imagine being a founder without an idea right now because, at once, there's a million things to go build. And, on the other hand, there's also probably a million people trying to do it. Look, I think you'll get very different perspectives on this.

Developers have always been a very opinionated, picky buyer. And that's created a lot of opportunity for a lot of different companies that start with very specific frameworks or ways of doing a certain task or a delivery model. And they get uptake if they are right, at least among a subset of people. And so I think it's rare. And GitHub is a great counterexample of this. I mean, it's almost like a consumer top-1,000 Alexa domain. It's probably much higher than 1,000.

But in general, it's rare that you see ubiquity, is what I'm trying to say, in the world of developers. I still believe, just on that alone, there's a ton of opportunity left to go build something really interesting. And two, I would say, again, it's a market that's young, fast, but also huge. And to everything I've talked about in this presentation, a lot of the early opportunity, you could say the alpha is gone, so to speak.

But all of the companies that have absorbed that and captured that have been moving at such breakneck speed that there's just a ton of derivative stuff to go do now.

And so it would be like saying, well, AWS, Azure, and GCP came around and they ate up the whole cloud opportunity. So I guess there's no more money in cloud anymore. Of course, that wasn't the case, right? It just meant that there were going to be all these new things that we needed around the ability to be cloud-based. And I mean, we could talk for hours about what those things were, but I think, loosely, that analogy holds, where we have a new way of writing code and there are going to be a lot of products that exist to serve the new needs that come along with that.

And this is the magic of the Twitterverse and everything else. You see more reviews and love and hate for software tools in the B2B universe than you do sometimes for mass-market consumer phenomenons. And so, yeah, to your point, the ability to become part of the de facto stack for building a company in this era, whether it's on the database side with a Postgres database like Supabase or Neon, or whether it's with the actual tool like Cursor that you're using to then write and deploy that code, that is a huge opportunity right now.

And I think with this surge in people that are deciding to build something all at once, there is a good social-proof element to what people want to be doing and what's been working and what's not. And I think that's a—we talk a lot here at FirstMark about what are the new ways that you can have advantages as a company, especially in a world where it's never been easier to build product. And in many ways, it feels like we are at risk of being like a copycat world where you see some success online and then you go copy it the next day.

But it feels like marketing and distribution and the ability to communicate directly one-to-one with your audience has never been more important. And a lot of the companies that we're talking about today do an exceptional job of that.

So everybody wants to buy things that were bought by the most discerning people. And the notion of who is the most discerning person just seems to ping-pong around and change over time. So, to your point, most people will care, at least in our world, what the smartest buyer at Cursor thought about a given tool than what the smartest person at the $80 billion software business that IPO'd in 2012 might think.

And I mean, even in my 10 or so years of investing, it's been wild to watch the shift, almost socially, of, like, who are those companies that everybody is looking at for guidance on the right way to do things or the right tool to buy for a certain space. And yeah, you're 100% right. It's a very powerful thing to say some of these companies that we talked about today, and many others, are your customers or are your partners or are people that are willing to even put their name next to yours on an infographic.

And so, yeah, it moves quickly and it changes often.

Yes, I think that sentiment has been shared widely. What's interesting to think about, too, is just over time, how the definition of developer tool has changed. And so, take HashiCorp, for example. Terraform is a tool that very much developers use. Super successful outcome, recently sold to IBM for over $5 billion. I think it was $6 billion, $7 billion. And what's been interesting to watch, and why I use that example, is just, again, developers, while they have been cheap, so to speak, and picky and opinionated, their relative importance across organizations has seemed to just go up and to the right over the last decade or so.

Their ability to make influential decisions on stack products, et cetera, really has seemed to change over time. And it's interesting to say that now, given we're talking about maybe their skill set is less needed now more than ever. But I think developer tools in and of itself is sort of this loose category that very much used to be tools that developers use, and maybe that was limited to DevOps. And now it feels like it's tools that have either development implications or interfaces that developers use, whether or not they're for the development process.

And that has yielded a much bigger set of companies. And so Stripe is a great example. Stripe is not a developer product in the typical sense. Historically, nobody has thought of payments as a developer problem. But the magic of what that company was was that they were able to cater it to developers and make it really easy to use and adopt and play around with in a sandbox. And we all know how that story played out.

And so I think, yes, while they've been a hard group to sell into, if you can do it right and kind of capture that taste and that feeling of capturing the moment, it has yielded some of the larger outcomes that we've seen across our world.

Yeah, well, thanks so much for having me. It was a treat.