Big Data

Full Steam Ahead: The 2024 MAD (Machine Learning, AI & Data) Landscape

This is our tenth annual landscape and “state of the union” of the data, analytics, machine learning and AI ecosystem.

In 10+ years covering the space, things have never been as exciting and promising as they are today. All trends and subtrends we described over the years are coalescing: data has been digitized, in massive amounts; it can be stored, processed and analyzed fast and cheaply with modern tools; and most importantly, it can be fed to ever-more performing ML/AI models which can make sense of it, recognize patterns, make predictions based on it, and now generate text, code, images, sounds and videos.

The MAD (ML, AI & Data) ecosystem has gone from niche and technical, to mainstream. The paradigm shift seems to be accelerating with implications that go far beyond technical or even business matters, and impact society, geopolitics and perhaps the human condition.

There are still many chapters to write in the multi-decade megatrend, however. As every year, this post is an attempt at making sense of where we are currently, across products, companies and industry trends.

Here are the prior versions: 2012, 2014, 2016, 2017, 2018, 2019 (Part I and Part II), 2020, 2021 and 2023 (Part I, Part II, Part III, Part IV).

Our team this year was Aman Kabeer and Katie Mills (FirstMark), Jonathan Grana (Go Fractional) and Paolo Campos, major thanks to all. And a big thank you as well to CB Insights for providing the card data appearing in the interactive version.

This annual state of the union post is organized in three parts:

Part: I: The landscape (PDF, Interactive version)
Part II: 24 themes we’re thinking about in 2024
Part III: Financings, M&A and IPOs

MAD 2023: Top 10 Trends

Every year, as part of our MAD project, we do a presentation at Data Driven NYC about the top trends we see across data and ML/AI. (here’s the 2022 version for reference).

The presentation, done this year with my FirstMark colleague Kevin Zhang, is a whirlwind tour of top trends, as opposed to anything particularly in-depth, as we tried to keep it short. But hopefully it should provide a good overview of what’s been happening in those spaces, for anyone interested in a recap.

See below for:

the video (20’53”)
the list of top trends for easy perusal
the slides

Interview on the Software Daily Podcast

Software Daily (aka Software Engineering Daily) has been on my podcast rotation for a while, so it was fun to get a chance to be a part of it – thanks to Jocelyn Houle who moonlights as podcast host on top of her day job at Securiti. While this was done in connection with the publication of the MAD 2023, we ended up talking a lot of about venture capital and entrepreneurship in general, including some personal stories.

The video is below, and here’s the audio-only podcast: Apple, Spotify.

MAD 2023 Interview with Ternary Data

One of the cool parts of publishing the MAD landscape every year is the conversations that come with it. Here’s a fun chat I did recently with Joe Reis and Matthew Housley, co-founders of data consulting company Ternary Data and co-authors of the O’Reilly book, Fundamentals of Data Engineering (see their recent talk at Data Driven NYC). We covered a lot of things, check it out!

The 2023 MAD (Machine Learning, Artificial Intelligence & Data) Landscape

It has been less than 18 months since we published our last MAD landscape, and it has been full of drama.

When we left, the data world was booming in the wake of the gigantic Snowflake IPO, with a whole ecosystem of startups organizing around it.

Since then, of course, public markets crashed, a recessionary economy appeared and VC funding dried up. A whole generation of data/AI startups has had to adapt to a new reality.

Meanwhile, the last few months saw the unmistakable, exponential acceleration of Generative AI, with arguably the formation of a new mini-bubble. Beyond technological progress, it feels that AI has gone mainstream, with a broad group of non-technical people around the world now getting to experience its power firsthand.

The rise of data, ML and AI is one of the most fundamental trends in our generation. Its importance goes well beyond the purely technical, with a deep impact on society, politics, geopolitics and ethics.

MAD 2023, PART II: FINANCINGS, M&A AND IPOs

*(note: this is part II of the 2023 MAD Landscape. The landscape PDF is here, and the interactive version is here)*

“It’s been crazy out there. Venture capital has been deployed at unprecedented pace, surging 157% year-on-year globally […]. Ever higher valuations led to the creation of 136 newly-minted unicorns […] and the IPO window has been wide open, with public financings up +687%”

Well, that was…last year. Or more precisely, 15 months ago, in the MAD 2021 post, written pretty much at the top of the market, in September 2021.

Since then, of course, the long-anticipated market downturn did occur, driven by geopolitical shocks and rising inflation. Central banks started increasing interest rates, which sucked the air out of an entire world of over-inflated assets, from speculative crypto to tech stocks. Public markets tanked, the IPO window shut down, and bit by bit, the malaise trickled down to private markets – first at the growth stage, then progressively to the venture and seed markets.

We’ll talk about this new 2023 reality in the following order:

MAD 2023, PART III: TRENDS IN DATA INFRASTRUCTURE

(note: this is part III of the 2023 MAD Landscape. The landscape PDF is here, and the interactive version is here)

In the hyper-frothy environment of 2019-2021, the world of data infrastructure (nee Big Data) was one of the hottest areas for both founders and VCs.

It was dizzying and fun at the same time, and perhaps a little weird to see so much market enthusiasm for products and companies that are ultimately very technical in nature.

Regardless, as the market has cooled down, that moment is over. While good companies will continue to be created in any market cycle, and “hot” market segments will continue to pop up, the bar has certainly escalated dramatically in terms of differentiation and quality for any new data infrastructure startup to get real interest from potential customers and investors.

Here is our take on some of the key trends in the data infra market in 2023.

MAD 2023, PART IV: TRENDS IN ML/AI

(note: this is part IV of the 2023 MAD Landscape. The landscape PDF is here, and the interactive version is here)

The excitement! The drama! The action!

Everybody is talking breathlessly about AI all of a sudden. OpenAI gets a $10B investment. Google is in Code Red. Sergey is coding again. Bill Gates says what’s been happening in AI in the last 12 months is “every bit as important as the PC or the internet” (here). Brand new startups are popping up (20 Generative AI companies just in the Winter ’23 YC batch). VCs are back to chasing pre-revenue startups at billions of valuation.

So what does it all mean? Is this one of those breakthrough moments that only happen every few decades? Or just the logical continuation of work that has been happening for many years? Are we in the early days of a true exponential acceleration? Or in the early days of a hype cycle and mini financing bubble, as many in tech are desperate for the next big platform shift, after social and mobile, and the crypto headfake?

The answer to all those questions is… yes.

2021 MAD Landscape: The Top 10 Trends

For anyone interested in a quick overview of our long-form 2021 Machine Learning, AI and Data (MAD) Landscape, here are the Cliffs Notes! My co-author John and I did a presentation at our most recent Data Driven NYC, focused on top 10 trends in this year’s landscape.

As a preview, here they are:

Every company is a data company
The big unlock: data warehouses and lakehouses
Consolidation vs data mesh: the future is hybrid
An explosive funding environment
A busy year in DataOps
It’s time for real time
The action moves to the right side of the warehouse
The rise of AI generated content
From MLOps to ModelOps
The continued emergence of a separate Chinese AI stack

Below is the video from the event, and below that, the transcript.

Red Hot: The 2021 Machine Learning, AI and Data (MAD) Landscape

Full resolution version of the landscape image here

It’s been a hot, hot year in the world of data, machine learning and AI.

Just when you thought it couldn’t grow any more explosively, the data/AI landscape just did: rapid pace of company creation, exciting new product and project launches, a deluge of VC financings, unicorn creation, IPOs, etc.

It has also been a year of multiple threads and stories intertwining.

One story has been the maturation of the ecosystem, with market leaders reaching large scale and ramping up their ambitions for global market domination, in particular through increasingly broad product offerings. Some of those companies, such as Snowflake, have been thriving in public markets (see our MAD Public Company Index), and a number of others (Databricks, Dataiku, Datarobot, etc.) have raised very large (or in the case of Databricks, gigantic) rounds at multi-billion valuations and are knocking on the IPO door (see our Emerging MAD company Index – both indexes will be updated soon).

But at the other end of the spectrum, this year has also seen the rapid emergence of a whole new generation of data and ML startups. Whether they were founded a few years or a few months ago, many experienced a growth spurt in the last year or so. As we will discuss, part of it is due to a rabid VC funding environment and part of it, more fundamentally, is due to inflection points in the market.

In the last year, there’s been less headline-grabbing discussion of futuristic applications of AI (self-driving vehicle, etc.), and a bit less AI hype as a result. Regardless, data and ML/AI-driven application companies have continued to thrive, particularly those focused on enterprise use cases. Meanwhile, a lot of the action has been happening behind the scenes on the data and ML infrastructure side, with entire new categories (data observability, reverse ETL, metrics stores, etc.) appearing and/or drastically accelerating.

To keep track of this evolution, this is our eighth annual landscape and “state of the union” of the data and AI ecosystem – co-authored this year with my FirstMark colleague John Wu. (For anyone interested, here are the prior versions: 2012, 2014, 2016, 2017, 2018, 2019 (Part I and Part II) and 2020.)

For those who have remarked over the years how insanely busy the chart is, you’ll love our new acronym – Machine learning, Artificial intelligence and Data (MAD) – this is now officially the MAD landscape!

Dataiku’s Series E: Ushering the Era of Everyday AI

Today, Dataiku is announcing a major new financing – a total of $400m at a $4.6B valuation, led by Tiger Global (which had also invested in the company’s Series D), alongside a great group of existing and new investors.

While financings are ultimately just milestones, this is certainly a testament to the remarkable progress the company has been making towards becoming a major global software player, as it has scaled to hundreds of customers around the world and some 750 employees (and yes, hiring a lot more).

Beyond the headlines and high-fives, what is the story? Here’s a quick industry backgrounder and reminder for anyone new to the company.

A huge part of the data world has been historically focused on business intelligence, with both historical players (Tableau, Microsoft’s Power BI, Google Looker) and newer players (SiSense, Mode, etc.). Business intelligence tools enable you to analyze the past and the present of your business: “which region performed best last quarter?”, “who are our best salespeople?” etc. This is sometimes referred to as descriptive analytics.

Dataiku is a leader in another part of the data world, which different people call different names: data science, enterprise AI (for artificial intelligence), enterprise machine learning. Beyond the semantics, the core idea is to make it possible to asnwer questions about the future of your business, based on the analysis of historical data: “which customers are most likely to buy this product?”, “which customers are most likely to churn?”, “which transaction is most likely to be fraudulent?”, “which region is most likely to show strong demand this month?”. This area is sometimes referred to as predictive analytics.

Introducing the MAD (ML, AI, Data) Public Company Index

Today, we are previewing a new public market index – the MAD (for machine learning, AI and data) index.

Readers of this blog know that we have been tracking the data ecosystem since 2012, through annual landscapes (see the 2020 Data & AI Landscape).

Over the last few years, a funny thing happened – some of the small startups we had started tracking grew up, did an IPO and became large public companies.

Not so long ago, public market investors used to say there’s was no good way of “playing” the Big Data and AI trends, due to the lack of public companies in the space. This is less true today.

However, there isn’t much out there in terms of looking at those public companies as a group. For example, see this Seeking Alpha piece, Top 3 Artificial Intelligence ETFs To Consider, where none of the companies listed are actually AI companies.

Hence the idea of the MAD Index. It’s still a small group of companies, but my colleague John Wu and I were curious to see how they fared in public markets, now and going forward.

This is just a start. We anticipate that a number of companies will join this group in the next year or two, and we’re excited to see how this index matures.

Data Observability and Pipelines: OpenLineage and Marquez

There’s an inherent tension at the heart of modern data infrastructure. On the one hand, it’s becoming more mission-critical every day, as companies around the world rely on it to run their business. On the other hand, it’s more complex, and potentially brittle, than ever, an “assembly chain” involving multiple tools and repositories.

This tension has led to the emergence of DataOps as a distinct and very active segment. One particularly important area is known as “data lineage“. The concept is basically to monitor data pipelines and understand the journey of data through its various transformations and usages. This makes it possible to fix any issues that happen along the way, and go to the root of data quality, and potentially fairness, issues.

Because data lineage involves many different tools, platforms and companies, it makes sense for those different parts of the ecosystem to collaborate around standard definitions. This is the concept behind OpenLineage, a cross-industry effort involving creators and contributors from key data projects (DBT, Spark, Pandas, etc.), gathered together at the initiative of the founders of Datakin, an SF startup beyond the open source data lineage project Marquez (originally started at WeWork).

At our most recent Data Driven NYC, we had the pleasure of hosting Julien Le Dem, CTO of Datakin. His talk (video below) is very approachable and educational.

Resilience and Vibrancy: The 2020 Data & AI Landscape

In a year like no other in recent memory, the data ecosystem is showing not just remarkable resilience but exciting vibrancy.

When COVID hit the world a few months ago, an extended period of gloom seemed all but inevitable. Yet, as per Satya Nadella, “two years of digital transformation [occurred] in two months”. Cloud and data technologies (data infrastructure, machine learning / artificial intelligence, data driven applications) are at the heart of digital transformation. As a result, many companies in the data ecosystem have not just survived, but in fact thrived, in an otherwise overall challenging political and economic context.

Perhaps most emblematic of this is the blockbuster IPO of Snowflake, a data warehouse provider, which took place a couple of weeks ago and catapulted Snowflake to a $69B market cap company, at the time of writing – the biggest software IPO ever (see our S-1 teardown). And Palantir, an often controversial data analytics platform focused on the financial and government sector, became a public company via direct listing, reaching a market cap of $22B, at the time of writing (see our S-1 teardown).

Building a $12B Public Company: In Conversation with Olivier Pomel, CEO, Datadog

By any measure, Datadog is an incredible entrepreneurial success story. The company went from a tiny startup in 2010 that had trouble raising money, to a public company that, at the time of writing, has a market capitalization of $12.5B. It was a pioneer in the category of DevOps and observability, and it’s now a clear leader. With revenues hovering around $350M, it has 1,300 employees across 31 locations around the world.

Perhaps improbably, the founders built the company out of New York, which many people over the years have thought of as a hub for adtech, media and commerce startups only. Along the way, they faced a lot of skepticism: “Whenever we pitched West Coast investors it was sort of seen as a form of mental deficiency to be based in New York and doing infrastructure“, says Olivier. I wrote a few months ago about the significance of the Datadog IPO for the ecosystem and beyond. Ironically, out of the three top public tech companies in New York today, two are infrastructure software companies (Datadog and MongoDB).

Not one for gratuitous self-aggrandizing, Olivier has given surprisingly few interviews over the years, and it was a real treat to sit down with him for a fireside chat in front of a packed house of 350 attendees at our most recent Data Driven NYC.

We had an in-depth conversations and covered a lot of topics.

The first half of our conversation was focused on Datadog itself, starting with a high level overview of the observability and DevOps space to make the discussion approachable by people who don’t know the space.

The second half of the conversation was focused on all sorts of lessons learned along the way of building a major company- sales, marketing, fundraising, etc.

Below is the video. We have also provided a full written transcript to make the content easy to scan through (many thanks to Karissa Domondon for her help with this).