Dashboards Are Dead: Sigma’s BI Revolution for Trillion-Row Data

We talk about the fact that the world did not need another BI product.

We didn't come up with that line. It was actually someone from JPMorgan that told us that, and I think he was right. BI as an industry is boring. The idea that—I know I'm in an audience of data people. Some of you have done this job, so no offense intended. But when we were living in this world of client-server, you had storage and compute on premises, and you had to figure out ways to connect to those things. And that was difficult from a security point of view and a networking point of view.

And then you had these special skills, like writing SQL, and then you could create dashboards for people who really wanted the data. That just didn't seem like a great model, combined with the fact that, increasingly, strangely, over time, the chart became the proxy for the data. It's funny because if you look at older BI tools like Business Objects and even MicroStrategy and Qlik, they were very tabular in terms of their interface, and they gave you row-level data. And then we sort of, in my opinion, devolved into Tableau, and we sort of treated people like they were idiots and all they could really understand was a colorful pie chart.

I tell everybody this: I'm very kind about my metaphor. If you've ever traveled to Japan and bought anything, they wrap things to within an inch of their lives. It could be a toothbrush, but you're going to get that thing perfectly wrapped. And I think that Tableau is wrapping paper in Japan. You really wanted the toothbrush, but somehow what you really got was wrapping paper. And one of the things that we aspire to is just realizing that you don't need the wrapping paper if you could really get a great toothbrush.

We want people to interact with the data directly. We wanted to recognize that the vast majority of people actually do their jobs in spreadsheets. The most common feature in all BI products was the button that said, "Download to Excel," and that's how we did our jobs. So we wanted to change that, not be another BI company. It's boring. We wanted to be something very different, and I'm sure we'll talk about that here.

I think the most commonly understood definition of BI is a dashboard that shows you historical data trending. I think that's the way most people use it. They look at it and they get a visualization that shows them some period of time, and they may be given some filters or controls that give them the ability to adjust some of those parameters. But that's generally how they use BI. I think that's a very constrained definition of BI. We look at it more in terms of what does the user want?

They want to not just look at historical data. They want to find anomalous data. They want to do forecasting. They want to reconcile data against things that they know. They want to add their own data to the enterprise's data. And then ultimately, what I think the definition has to include is the fact that none of us really get paid anywhere for understanding historical data. And we don't really even get paid for forecasting things. We get paid for doing things.

And to the extent that you can unify the idea of historical BI being a trend and a spreadsheet being a forecast, you really want to pull that through with automation, and you want to give the end user the ability to write that automation themselves. I think that's the proper definition of business intelligence. I think that's always what it was, but it just was broken up in terms of dashboards here, spreadsheets there, and then applications over here.

It's been around for so long. I think the general thing that a BI person gets asked is one of two things. Can you please update this dashboard? By which they mean, I want another data source included, or I want a different type of visualization, or something like that. Or the second request they get is, can you extract this dataset to me and just email it to me?

That's what BI people, I think, typically do. I think what they want to do is figure out for the business what is actually going on in the data. What's happening to our marketing performance? Is it getting more or less efficient? What's happening to our inventory? But they end up servicing a bunch of very rudimentary, relentless requests from users who are one step removed from the data that they want. So I think both sides are very unhappy with this process.

But unfortunately, I think that's probably a reasonably good description of what we see people doing in classic BI jobs today.

So if you're familiar with Sutter Hill Ventures, Sutter Hill Ventures is a pretty well-known VC in the Valley and behind companies like NVIDIA, for example, but certainly Snowflake. And where Sutter Hill is a little bit unique is that Sutter Hill doesn't invest in preexisting companies. All the companies are incubated directly out of Sutter Hill. So they, I think, would tell you that their expertise is finding people, and then they give those people the funding necessary to go start a company. Now, the other thing that they tend to do is invest in areas.

So they'll say, "I think this is a really big category. How do I go find the best people in this category and get them started?" Snowflake started that way, I believe, in 2010-ish. Sigma was founded by Rob Woollen and Jason Frantz in 2014. The thing about that scenario is when people found the companies in Sutter Hill, they have zero. They don't have a line of code. They don't have an idea.

They push them out of the nest and tell them, "Go do something. Here's some money. Go come up with an idea and make it happen." That's a little different than a lot of companies that we're familiar with, where you walk in and show your prototype, you've got a full deck with you and a market plan and all this other stuff. None of that exists in Sutter Hill's world. They've already decided the area they want to invest in.

They're really just deciding whether they believe in you. So that was actually the founding.

To the extent that we can say Sigma is in any way successful today—and I hate to say that because I'm a super-paranoid person—but we're doing okay. And the expression is, every overnight success is 10 years in the making. This being just about 10 years since Sigma was founded, we're right on cue. But those are really hard. If you've built your own companies, it's never a linear path. You spend—matter of fact, it's a lot like evolution.

It happens in big bangs, and then nothing happens for a long time. And I think Rob and Jason—and I'll say this on their behalf because we've told this story with them and for them for a long time now—the idea that drives the company today is the first idea, which was: Why is the average person never given technology in Silicon Valley? Like, if you live in Silicon Valley—I live in California, I live in San Francisco—all the companies that you know and love that weren't consumer were built for technical people.

Whether it was VMware or anything that Cisco ever invented, everything was built for a technical person. Nothing was built for a normal person. And the normal people are—just by the way, I'll call them normal people; I consider myself a normal person—like the long tail, but the vast majority of the people at every single company that you go to. They work in marketing and inventory, and they work in HR and finance and everywhere. But if you ask them how they do their jobs, or ask them even a question I ask everybody, which is, do you have a working spreadsheet on your laptop today?

One hundred percent of people in the working world answer yes. So this technology was invented in 1985. If you went back and looked at Excel's release in 1985, when Microsoft released it, it looks an awful lot like it does today. Very little difference. So think about that. 1985 to 2025, the vast majority of your people are doing their jobs almost exactly the same way. It's crazy. They wanted to change that. And so the idea then is the idea today.

Having said that, it's a bumpy road. And I don't think there were the conditions in the market to make people think differently back in 2016. You could have a great idea if you start a company, by the way. But until something changes below you, as I like to say, no one wants to hear you. So, for example, in our world, we needed a Snowflake, a Databricks, a BigQuery, a Redshift. We needed something changing right below the access layer so someone could ask themselves a question like, should I change the thing I'm using on top of that now?

Because that's the way tech always works. Something changes here, then you change the next thing, then you change the next thing. I was listening to OpenRouter, right? That company does not exist without LLMs coming around first. Like, okay, how do I access the LLM? Okay, now he has a marketplace to go after. So I think with Snowflake growing, and if you lived in Northern California in, like, the 2018 timeframe, you could kind of see what was happening there.

They had an MVP product. They took it to Bob Muglia, who was the CEO at the time. Rob and Jason took a spreadsheet working on—and there's a little secret—on a Postgres database to Bob. It was not even working on Snowflake when he saw it. And his answer was, "I do not write SQL. I cannot use Snowflake as the CEO. This is a great idea. Go build this." And they built a prototype. And if you've ever worked at a company, this is a tragic thing I'm about to tell you, but this is the sort of stuff that happens at companies.

They built a prototype. They hired a president, which is kind of the playbook that Sutter Hill runs. The president runs sales and marketing, and the whole idea of salespeople at that time is to find product-market fit. Like, your salespeople are product managers until you really get to scale. And actually, I tell all our salespeople they're still product managers today. And then that person died six months later. He died of terminal cancer, actually. And it was really tragic. So you're building this company, you're six years into it, you have basically no revenue in one of the hottest SaaS markets, your leader has now passed away, and you're Rob Woollen, who was the CEO at the time.

If you ever meet Rob, he's like a big teddy bear. He's just an amazing human and the kindest person. But I think at that time, he was like, "This is just not something I can keep going with." So he said, instead of replacing this sales leader, we're going to replace me as the CEO. I'm going to go do what I love, which is build great product. And Rob and Jason still do that for the company today. So I joined almost seven years into the company.

They had less than $800,000 in ARR, 50 customers, and about 50 employees, which is, by the way, a testament to Mike Speiser at Sutter Hill. Because he has so much conviction in his companies that, despite overwhelming evidence that the company is a total failure, he keeps investing in it. And that's a little brief history about what got me there.

Just recently.

I never call us a BI tool.

And the first thing that we decided to do then, by the way, was shut the product down. I come from a product background. I think the only thing that matters is great product. You can build sales teams and finance teams, but, like, you have to have great product if you're going to win over the long term. We shut the— we sold it, but we deprecated the features. So we didn't build anything new in it, and we completely replaced it in June of '21.

So at that point, we had a couple million dollars in ARR.

In the middle of COVID. Actually, I interviewed with Rob the day San Francisco shut down. We were the last people in the office. It was a crazy time. We've all lived through that time. I don't even like talking about that time. I never want to go back to that. It's terrible. But it did prompt one thing for us, which is, in '21, even in face masks, I required everybody to come back to the office, and I still require that today, which is its own conversation.

But I'm not a believer in remote work, and you can throw whatever you want at me in the meantime.

Yeah, I've taken plenty of arrows on this topic. Five years later, we'll double the company again this year. I think we were able to raise money because there aren't many companies that accelerate growth over time. And we've been able to do that. And it's not because we're a BI company, because, as we said before, the world didn't need another BI tool. We do a lot more than that.

Yeah. So one of the things that changed, obviously, with Databricks and Snowflake and BigQuery and so forth is that the data volumes got huge. One of the things that I knew before—I was working in infrastructure—and one of the things I knew before joining Sigma was that the price per terabyte of storing data in AWS for five years in a row was minus 65%. So you don't have to sort of graduate with an economics degree to understand that if you have declining prices year over year, you're going to have increasing volumes.

That's exactly what was happening. Snowflake and the like came around and just figured out how to make—or, better said, I like to think of it as organize—that data so you could use it. So the first thing that Sigma had to do architecturally is abandon everything BI products had done in the past around managing for performance. All products a priori were built with caching layers. We do not do any caching. So everything you do in Sigma is live query on the warehouse.

Our customers have up to trillions of records. And our overhead on that transaction is less than one second. So you log in through Sigma, you build your assets in a Sigma controller layer, but we push down every bit of the query. So the data never leaves the warehouse. This has a number of very salient beneficial impacts. Number one is super-high performance, or, better said, the performance is 100% correlated with your warehouse performance. Number two is security and governance.

Data never leaves the warehouse. So whatever policies you put there through OAuth, for example, which you can even further extend in Sigma, are adhered to. So security teams love this. And three, if you're running a business, is that we have basically no cost of operation—COGS is what this is called—because the customer pays it through their warehouse agreement. So we have a nice financial profile model. So Sigma is effectively this: you log into Sigma, you build your assets.

We are a pushdown query model. We take advantage of a lot of the stuff that's on offer in the warehouse. You can use caching layers, you can use materializations. We've built some very fancy technology we call Alpha Query, which we leverage off your browser cache. And we've shown that 50% of your in-session queries after your first can be serviced out of that cache, which takes a lot of pressure off the warehouse. It saves you money, and it is a 500-millisecond response time.

So we do a lot of fancy things around performance and cost management as well.

Yeah, we love Databricks. Matter of fact, if you look at Ali Ghodsi's posts, he's posted multiple times about Sigma in the last few months alone. In fact, I would extend the comment just a little bit further to say this: if you're an enterprise customer, enterprises over time adopt multiple of the same types of suppliers, right? Like supplier arbitrage. And if you want to abstract end users from choices you make about warehouses or LLMs living in the warehouses, and you want to be able to access all of them without having to have your users go to interface A, application B, and interface C, Sigma abstracts that for them.

So through Sigma, you can work with both Snowflake and Databricks at the same time, or the Postgres warehouse you use to avoid paying those types of warehouse credits because maybe the data only refreshes once a day and you don't want to have that hitting your warehouse. But they're getting one interface. So there's simplicity for the end user, there's flexibility for the infrastructure and the data teams.

The only thing that we cannot do, but this is not a trade-off per se, is because we don't cache, we don't federate. So you can't select tables from two warehouses and return a result in Sigma because there's nowhere to join that data. You can use products like Starburst or Trino, and we support those. We treat those as warehouses as well. But that's the trade-off in the architectural choice that we have made.

I think that the models are going to live next to the data in these warehouses, and we're already seeing that. Anthropic doing that deal with Snowflake is a clear attempt to make sure that they're as close as they can be to the enterprise data, that they're getting great performance, that they're adhering to security models. I think that's a trend that we are going to see. I also think we're going to see people building smaller models in these warehouses as well, either tailored to whatever business they're in or trying to manage cost.

So I think that's inevitable. There are some trade-offs, of course, to not working on the premises side. But I always think about things as a train track in this particular case. Like, the train's left the station. It's gaining momentum. There's no point in building backwards.

How does that work?

So a spreadsheet's difficult to do on top of a warehouse because you're dealing with basically a flat table against a relational database. And these two concepts don't really work well together. And that's kind of the magic that we mostly made work. And when I say mostly, I always tell folks, what's the difference between a spreadsheet in Sigma and a spreadsheet like Excel? And the difference is, in Sigma, there is no cell reference, right? Everything is columnar. A table doesn't know about a cell.

So there's no way to make that proxy really work without massive performance implications. But the fun thing about all of this is, if you look at Sigma as a BI product and you want to build a pie chart, you can do that. And you can do that with AI, by the way. You can do all your favorite BI things. But the fun part is, like I referenced, J.P. Morgan, who uses our product. And when they connected the first time, the guy's response was like, "Can I swear in here?"

Holy shit. Holy shit, I just built a six-billion-row pivot table, right? Like, for people whose jobs every day are to deal with massive amounts of data and find something insightful, something unique, right? If you work in financial services, you're in a data arbitrage business. So the faster you can get to unique data, the better you are at your job. And we just gave this person the ability to do that over a massive dataset in seconds. In fact, it was so impactful that the head of technology, media, and telecom investment banking walked up to me and was like, "I demand to invest in your company right now."

They were that convinced. If you're old enough to remember the Remington shaver, they had a commercial, and the guy said, "The shavers were so good, I bought the company." It felt like that moment. But the fact that you can reach people where they are, and you can build a six-billion-row pivot table where you can build a scenario model, where you can do data reconciliation, or you can do a forecast model and it runs live, you can build 50 forecast models or variants of them and let them run live.

And by the way, set conditions on it and have it notify you when things change. Like, there are things you could never do in spreadsheets that you can do at massive scale in Sigma, but leveraging all the skills that you already had when you did your favorite VLOOKUP.

Yes.

Collaboration features are actually a lot harder to do than most people realize because of the synchronicity of the work going on across large numbers of people. So we have to track versioning and changes and communication all at the same time. So you can live edit, is the example I think that you were sort of alluding to. You and I, anybody in the audience and I, we can go build a table together or we can modify it and we can chat at the same time, and you can live edit it in the same way that you would do in Google Suite.

But we, of course, have to log all of this just like they do. We have to be able to roll it back. But the volumes here are now enormous. And we have a little bit of a higher bar because the enterprises that we do business with also have to be able to see who did what and when they did it. So, for example, I think probably one of the largest polluters in the world, by virtue of their company's role, is ExxonMobil.

They built emissions reporting as a data application on Sigma largely because all of the changes that were happening as they were modifying these records had to be logged. So they had to know what changed, who changed it, when they changed it, because this is showing up in their 10-Q, which is a board report every quarter. So making all of this work synchronously at huge scale, we learned a lot of lessons along the way, but eventually got there.

So maybe we'll start with: what is a semantic layer? We ran a little study, by the way. I'll give you a stat to indicate why these things become important. So we looked at all of our enterprise customers' table utilization over a 60-day period. 0.1% of the tables in their warehouses were ever touched. 0.1% of the tables. And we're pretty convinced that one of the problems that exists in most enterprises is just awareness of the data that's available to people, period.

So with that as background, a semantic layer helps provide meaning to people on data models, right? It gives you an understanding of what this data means. And obviously, dbt went out there as kind of one of the more successful ones a few years ago, with an open-source model, and they've been since trying to monetize that. And there are a few other vendors that are doing the same, and I suspect you might see warehouse providers do the same. Sigma does a semantic job for you but is not trying to replicate the semantic layer.

You can data model in Sigma. You can build all kinds of referencing capabilities so that you can search and identify that data. You can build metrics in Sigma. So, for example, what revenue is, and that there's a standard calculation for revenue on the standard revenue table. But we're not trying to be a full-fledged semantic layer. Most of those products try to automate ongoing changes that you've written rules around. And these are the things that dbt does well.

I think for those people that are thinking about adopting a semantic layer below your warehouse, the core question for you is: do you intend to have multiple applications accessing that data? If the answer is yes, semantic layers make a lot of sense. If the answer is no, they're just another layer of technology you have to manage.

I think having a completely separate layer is predicated on leveraging that layer across many applications. And I think it's that simple. Otherwise, I think we have seen a lot of fragmentation in data that's going to end up consolidating. So things like semantic layers, data catalogs, ETL tools, data observability tools—I think it's far too fragmented. I think if you were to fast-forward three years from now, you'll see all these categories, many of these categories, rolling into smaller numbers of categories.

I don't think that there's enough value in all of these categories to justify multiple competitors in the category. And unless you have multiple competitors in the category, you don't have a category.

Hyperscalers have never demonstrated they can do anything but provide server and storage hardware, in my opinion. And I'm directly accusing Microsoft here. I think their application products are not good in their cloud products. I think Fabric is a run-it-back strategy on Office 365, where they just provide a marketing name to some longstanding old products. And it really makes it—I hate stuff like that.

They basically take an old product like Power BI, another existing product called Power Apps, they throw Excel in there, some sort of connection to OpenAI, and they wrap it all into a marketing term called Fabric, and they charge you more money for it. And then they say, "Hey, look, this is a great modern data stack product." It's like, no, it's not. These products have been around for like 20 years. But I do think that the other ones, like the Databricks of the world, the Snowflakes of the world—Google made an attempt at this.

But I think Google missed the zeitgeist. For me, technology is always democratized. If you think back 100 years, the things that were considered complicated, you do them all the time today yourself. I always tell everybody, in 1920, if I wanted to send a communication from my company to your company, I'd have to go walk up to somebody that knew how to type, and they would type it for me, and then we would put it in the mail, and then someone would hand-deliver it.

And you would never think about these things today, right? All technology becomes democratized. And the interesting thing about when you democratize technology is it becomes—you use it more. So when we went from typewriters to telephones and telephones to text, we don't communicate the same volume we did in 1920. We communicate at exponentially higher volumes. And I think this is now happening in the modern data stack. People have always been constrained by the data they could access.

And when you release the constraints, they use more data. And more data is being created, which, by the way, fuels more data. Therefore, I think that to some extent, the strong are getting stronger in data, and they're going to roll up some of these categories because trying to manage a data catalog over here, a semantic layer over here, a warehouse here, an ETL tool over there, an access layer where you build visualizations over here, it's just too hard. So, I mean, I don't think that happens in AWS because AWS has never been good at end-user products.

They're very good at infrastructure products. And I think that Google's experiment with Looker has failed. And with Looker, they built LookML, which, by the way, I would say back in 2017, was probably the best product on the market. But they built a product for experts. And they ensured that it was going to be a product for experts because they wrote their own language. And then you had to learn that language. I mean, you want to talk about lock-in? I mean, that's the ultimate way of saying, "I only want to empower one group and not anybody else."

And I think the market consistently moves toward democratization. So if you're building those kinds of barriers, ultimately your product is going to fail. It's notable to me, I was watching the last presentation, that OpenAI's business is a consumer business, right? That's why they democratized in the most easy way: language. You just chat something in there, it sends it back, and people want access to find out information. And they have a $10 to $20 billion business on that. And that's—if I'm—and I'm not an investor, I'm an operator who has to think like an investor sometimes.

But if I were an investor, this is what I'd be thinking: Is this democratizing technology? And if it is, how big is the market of the eventual user? And if the answer is those things, the rest is fascinating.

I won't comment on whether someone will buy someone else, but I will say this: Snowflake and Databricks are frustrated by the cost and complexity of moving data into their warehouses. They will solve that problem one way or another. They're then frustrated by the inability to get customers using these products at scale over time, and they will solve that problem too. 0.1% of the data is visible to a user; they'll want to solve that problem. Because remember, they get paid on compute.

There's no license. So if you're not using it, they're not getting paid. So they're always trying to change the underlying conditions that will make it easier for the average person to consume. So I do think that they are highly invested in solving these problems one way or the other.

It's a terrible idea. And so it doesn't matter whether it works. It could be the best-executed terrible idea ever. No one wants to speak SQL to a system. Very few people think in terms of SQL. I don't think around, like, what bagel do I want today? I select the bagel, the poppy seed bagel, from the second shelf. No one does this. So the idea that I'm supposed to adopt a SQL semantic concept into my natural language is the ultimate hubris of engineering.

Like, oh, these idiots don't write SQL. So what we're gonna do is we're gonna get them to be able to speak SQL easier to us. This is not what AI was supposed to do for end users. So I think these products are a joke. I don't think that they're going to succeed, and we don't care about them. So what we do care about, though, is helping people and helping systems understand each other. So I think we think a lot about context to data.

Who are you? What data do you tend to use? Who do you collaborate with on an ongoing basis? What did you do yesterday? What are people like you in the system doing? Combined with systems-level understanding: what tables are out there, what tables are being actively used. And you put the two together, and you can come up with some really good answers. So Ask is the product that we are putting into public beta on April 1st and then for GA on May 1st.

And one of the things that we, when we built Ask, the premise of Ask was AI is always wrong. That was the starting point of the development effort. And AI is always wrong for one of two reasons. One of them is that the AI is wrong. AI performance, for systems people, if you ask what good performance is, they'll tell you 85% is really good. And they'll work their butts off to make it 86%. But if I told you that you had to go to your investment committee and I had 86% confidence that the revenue that the system told you for Sigma was right, you wouldn't go to that investment.

Like, 86% is not good enough. And the second is that people ask really bad questions, really bad. In fact, you will know this yourself, that if you ask, and I'll just use these examples, if you ask a question about how much revenue does Sigma have, did you mean that? Did you mean revenue? Did you mean ARR? Did you mean bookings? NACV? ACV? What did you mean? We did this in our own all-hands: How many customers does Sigma have?

It showed me the answer. And the person that runs our AI system looked over, and he's like, "Is that right?" I was like, "It's not even close to being right. It's wrong." And by the way, if you ask the same question twice to many AI systems, you'll get two different answers. This is anathema to enterprise. I mean, this is almost the worst thing, right? So we think the biggest problem with AI is trust and transparency. So when you look at Ask, because we started with a premise AI is always wrong, if you ask a question, and you can ask a question of a trillion records, the first thing you're going to get, aside from an answer, because if you don't give an answer to somebody, they get mad, is you're going to get a chain of thought.

And the chain of thought is the system literally, right in front of you, showing what data sources it went to, showing you what formulas or filters it applied and in what order, all the way in a serial process till it produced an answer. And you can go back and change any of that. You can speak that change in natural language, or you can just literally change it in the system. So now you understand, at least when it created an answer, how that answer was derived.

The second thing we're providing answers this other question about the quality of your question. We're providing context. So we're going to give you, I don't know, six or seven different elements that are associated with your revenue question. We're going to show you bookings. We're going to show you growth over time. We're going to show you an ACV. And we're going to help you iterate through your thought process because we don't think AI is intended to be a chatbot. We don't think it's supposed to be, I ask a question, we give you an answer, and the session is over.

So the way that we caught the problem that the AI was wrong about the number of customers Sigma has is because in those context charts, one of them was active and inactive customers. And so if you looked at the addition of the active and inactive customers, that was the answer it gave. But we as humans, we knew the question wasn't, how many customers has Sigma ever had? Because then we would have said that. We asked the question, how many customers does Sigma have?

And so you can see, well, wait a minute. Okay, exclude inactive. And then it nailed the number. But wow, you shouldn't have to have the mental stress and overhead of thinking through just a precise question. This is what developers do. Coding is discrete and deliberate. When you execute code and it compiles, it will do the same thing over and over again. But when you ask about revenue or active customers, you don't wanna have to be burdened with the mental overhead of, okay, but what exactly do I mean?

I have to phrase this just right, because if I don't phrase it just right, I'm gonna get the wrong answer.

Bye.