Understanding Data Engineering in 2025 | Ben Rogojan, Seattle Data Guy

Hey there, Matt. Thanks so much for having me on.

Yeah, it kind of feels like more of the same, in the sense that we repeat the same lessons. I think people kind of go through the same iterations. New people come to the space, or possibly maybe it's more the business side, get really excited by the prospect of what AI or data science or neural networks, whatever iteration we're in, can do. And then we kind of get down the line and we're like, oh, we need to get certain things in order first to kind of get to a point where we can actually use data well.

So yeah, I think it's going to be a big year. I think we're set up to try to actually figure out how these things work. We've had two years to figure out some of the basics and now see where things go.

Yeah, sure. No, great. Thank you for the light intro. So yeah, like you said, Seattle Data Guy. Now I live in Denver, so it currently snowed, I think, the other day. So it's white outside. That's new for me from Seattle. I've been kind of in the data engineering space for nearly a decade at this point. Started at a hospital doing a combination of data science, data analytics, data warehousing work. I think, like most people around 2012, 2015, I got bit by the data science bug, and I was like, I want to do data science.

It's the sexiest job of the 21st century. So I got into that, but eventually found data engineering and really enjoyed it. From there, I worked for a healthcare analytics startup that did everything from fraud detection, some stuff with opioid monitoring, and a few other things around population health. Again, around data engineering and building kind of a massive data warehouse, essentially, of pretty much, like, I think it was over half the U.S. in terms of who existed in that data warehouse. And then eventually did that at Facebook as well, data engineering.

All throughout that time, I've been consulting as well and putting out a ton of content. So I think I started putting out content somewhere around 2017. So I've been writing for a long time at this point, and then started putting out videos. So a lot of people might know me from different times. Every once in a while, I run into someone who's like, oh yeah, I saw your stuff on Medium, didn't know you had a YouTube, or I saw your stuff on Substack.

But yeah, so I've been really focusing at this point for the last few years on data infrastructure, helping companies figure out what and how to set up their data, how to find value from it. And really, that's kind of the space I've been playing in for the last little bit.

Yeah, I think the main lessons learned that I've picked up over the last few years is, like, don't expect the first thing to work. I originally started writing a blog before even Seattle Data Guy that was around food and culinary stuff, because before I worked in the data world, I had a stint in fine dining. And it did okay, but it never really took off. And you can see that as a loss, or it kind of helped me prepare for the next few things.

So, one, not everything you do has to immediately turn into something of value, something that produces dollars. It's just like learning programming or anything like that. Have side projects that are just there for the sake of learning, because you never know when it's gonna turn into something. From there, it's a lot about consistency. So just keeping up with it. And then just always looking for ways to improve yourself. Try to look around, see what other people are doing, see how to do what you're doing better.

And just look for little micro-improvements you can make. It doesn't have to be perfect day one. And that's kind of nice. I think a lot of people like the idea of having, like, 100,000 followers. It can be terrifying. Yesterday, when I put out my newsletter, I accidentally said that Databricks had bought Tabular. Or not Tabular. I said that Databricks had bought Iceberg. Iceberg was the next step. I meant to say Tabular. And I put that out there.

And then I was like, oh, I didn't see it until I was filming a video version. And I was like, oh, how did I not catch that? So those mistakes go out to now 100,000 people versus just 10, which is a much nicer situation.

Yeah, I mean, my view, or at least a big portion of what data engineering is to me, is taking all this data and making it usable for people and now machines. I think that's something I liked out of Joe Reis's recent talk with data modeling. We're really just trying to take data from multiple disparate sources that maybe don't naturally talk to each other and put them into a format that other users, whether they're analysts or data scientists, can easily work with, and have that in a repeatable fashion.

We don't just want to have it work once. We have to have this system, these data pipelines, these tables that exist that people can rely on. And so that's really kind of the core of it. It's like we're just trying to make data more usable for people who maybe don't want to have to go through the core systems, and you can have those standards and things of that nature set up.

When we say AI engineering or ML engineering, I assume it probably means more taking those tools, like the tools that someone else probably built, right? Like someone else built OpenAI; we're probably now implementing it. Same thing with ML engineering. A lot of that, you probably had someone else build some of the models and you're just building it. There's a lot of overlap because oftentimes we will build things for maybe ML or AI teams. We do some of the collection aspects of it in terms of collecting the data, putting it in one place.

Depending on how your company is set up, you might just access the data directly from something like a data lake. And if you're an ML engineer or an AI engineer, you might just be doing it very raw. Some companies do have maybe the ML engineers access data more directly from maybe the data warehouse, which was something I saw at Facebook. In some cases, people would be accessing data closer to the source if they were in ML or AI, but there's plenty of ML and AI people who are relying on the data engineers to kind of build the core datasets for them to then go from there.

I think the other place you often see overlap—I wrote an article about this, or had someone else write an article and I shared it—they talked about one of the ways one of the teams at Riot Games was deploying their ML models, and they were using Airflow. And I thought that was interesting because, to me, Airflow is a very data engineering tool. But you see that you're still having ML engineers in some cases deploy using things that are data engineering-esque, in the sense that you're doing a batch model or something where you're training it maybe overnight or at a certain interval, and you're just using some tool, again, like Airflow to run that.

These roles still get very much amalgamated into one. I still see people, and I still talk to people, who are like, yeah, we're hiring a data scientist to build data pipelines and things of that nature. And so, in many ways, if you're at a small enough company, if you're in specific orgs in a company, you might just be doing everything in that one role. But in a perfect world, data scientists are likely doing things that involve a little more research, are a little heavier on the statistics side.

So they're trying to find and create models or insights using different tools that maybe analysts would use. Analysts tend to, I think, lean more heavily purely on SQL and maybe digging deep into those specific aspects using SQL, whereas data scientists might be using tools that are a little more statistically heavy. They're using models like something like k-nearest neighbors or something to try to find insights, versus just maybe an analyst generally is telling you and being very descriptive, like, hey, here's what happened. We can tell you why it happened and tell you maybe a little less in terms of what could happen in the future, how you can maybe build models to predict around that.

So you're more just describing what's happened and maybe have dug into specific cases of why it's happened, can kind of give a report on that, whereas data scientists might be on the other side building models off the top of that information that can kind of predict maybe in the future how to avoid that or how to better segment your marketing spend in the future to improve that. So I think that's some of the main differences you'll see.

I'd say you're going to spend some time trying to figure out exactly what the business needs, or if not the business, likely the analysts and data scientists, kind of talking to them, trying to understand what problems they're trying to solve. Personally, I think it's better to also get closer to the business, understand, okay, well, why are the data scientists and data analysts asking me to build this table? Once you've gained an understanding of the business, maybe some of the requirements they're actually asking, you're going to spend time planning and data modeling.

So, taking whatever that raw data is, whatever it looks like, trying to create it into a format that, again, analysts and data scientists in the future can parse and actually use in their models or in their reports. From there, obviously, you're going to build data pipelines that actually push to those tables and then actually kind of put it all together. And on top of that, you're probably going to be spending some time fixing broken data pipelines here and there.

Because things break all the time, no matter how much you try to kind of get around it. People make changes to tables. We've got some methods we're trying to, I think, implement, depending where you are, whether it's data contracts or something of that nature, to try to avoid some of those breaks. But to this day, people are still having, like, a data type issue break a pipeline that then causes everyone to have to kind of spend a little bit of time fixing them.

Yeah, I don't think necessarily the goal changes. I think there are aspects of the fact that there are different tools. One, when you work at companies like Facebook and other big tech companies, their infrastructure is very mature. They've generally done, at least most of the ones that I've either seen or worked at—in the case of Facebook—they've done a good job of creating a centralized set of tooling that everyone's using and everyone's on, which makes your job very easy. When I worked at companies outside of Facebook, you have to spend a lot more time just setting up and configuring whatever tools you need to manage.

At Facebook, you essentially have your internal Airflow that a different team runs, and you can just push files to it, and it picks up the data pipelines automatically. Whereas in most other companies, you might be the person that has to even manage your Airflow instance, which is managing all your data pipelines. You go into a different company, they're going to have six different data storage or data platforms. One team's going to be using Databricks, one team's going to be using Snowflake, one team's still somehow using a SQL Server instance.

You've got all these mergers and acquisitions happening, which also cause further layers of, okay, we brought this other team in, how do we integrate their data into the rest of what we're doing? And so it can be a little more—I think it can feel a little more chaotic. And there's less support if you're a data engineer outside of Silicon Valley. And so you're having to solve problems that maybe a data engineer at a big tech company isn't having to solve.

Obviously, in big tech companies, the data is big, and so you have those problems to solve. But on the other side, you've got these more process and organizational issues when you're working in non-big-tech companies that, because again, they kind of change and move, and one VP wants to be in charge of some initiative. Every VP wants to have an AI initiative, I'm sure, right? Right now. So everyone's going to have their own. Yeah. So everyone's going to have their own tool that they've picked.

And now you've got ten different tools under one house, where I think, again, big tech is a little bit better. I'm sure they have their own issues in terms of initiatives, but they're a little bit better at picking, hey, we're all picking these tools, and we're just going to run with them.

Yeah, I see a lot of people move laterally into data engineering, often from either data analyst or software engineer. Data analyst, because I think there's just the sheer quantity of data analytics jobs. I think that's just the way it is. In terms of skill sets, it's usually things like SQL, Python, or some programming language. You should understand how a data pipeline works. You should understand how to put it together. Same thing with data modeling: you should understand how to model for OLAP or data warehousing models versus transactional systems.

Yeah, I'd say Python's definitely a good place to start because it is so prevalent. Obviously, there's probably some arguments to be made about how it abstracts a lot away. So you don't have to think about the difference between a list and an array as much as you would in other languages. But it's also just very prevalent. When it comes to data engineering teams, that's likely what you're going to be using. Obviously, in some places they still use Scala and other languages for one reason or another.

Python is probably the most prevalent. In terms of how proficient, I'd always recommend people don't feel like you have to cap yourself. I think sometimes people ask me how much programming you have to know. I'd know more than just for loops and the basics. I'd hope you can at least write code that operates in more complex systems. I think that's the biggest thing. Maybe you don't have to know data structures and algorithms to a T, but you should be able to, if I need you to write code against an API, be able to do that and iterate over it.

And it shouldn't take 30 minutes to process 100,000 rows of data. So I think there are aspects around that where it's like, yeah, you should be able to at least move data around via Python. And then obviously you're going to be spending some time in the terminal or command line as well. So know how to navigate.

I'd say that's where a lot of companies use SQL as their de facto transform tool. Obviously, there's some UI-based tools, but I think SQL has probably become the de facto tool in most cases. So I'd say the actual syntax of SQL doesn't take significantly long to learn. There's a few things that maybe I see every once in a while people struggle to learn, such as window functions, if you want to get specific. But that won't take you long.

What's going to take you longer, and this almost happens at every company, is getting a sense for how the data is shaped. Like, okay, when I interact with it via SQL, how is it shaped? When I join, where am I going to run into accidental places where I can't actually or shouldn't be joining things together? Or what do I need to filter out? Or how's the logic set up? So I often find that it's more about the data and less about SQL.

Because again, SQL you can learn quickly. What's going to take time is just getting a sense for how data operates and flows, and datasets in general.

Yeah, so there's a couple different ways you can essentially approach data modeling. Most people are probably accustomed to what they learned in their database course, which is transactional systems, which tend to be normalized to third normal form, depending on how far you want to take it. Obviously, now we can be a little more loose with that because there's so many ways you can handle data that's maybe a little less structured. But from there, that's generally not very good for analytics.

You run into various problems, and also you're generally trying to centralize maybe multiple different sources into one location, which is your data warehouse. And so we've kind of come around, and there's a few different ways you can model for what we call analytics. So people often reference that as OLAP, or online analytical processing. The way most people start is through this standard kind of data warehouse, which is facts and dimensions. And that's just creating data or putting data in such a format that, again, it's just easier for someone who's maybe less technical to come at and be like, okay, I can kind of see the fact tables in the middle.

If I need anything descriptive, that's in the dimension tables around that. And so that's what we're really trying to do at the end of the day, is get data from one format that maybe is a little harder, requires more joins, requires some understanding of business logic, because sometimes there's layers of logic in the application that maybe aren't shown in the data itself, and we're trying to capture some of that and put that also in the data model. That way, when an analyst comes to it, they can just look at it and be like, okay, I have a sense of what's going on.

I think that's the real goal, is we're trying to make it usable for analytics. Just as a quick point, the other thing is it usually, depending on the system you're running it on, is better for analytics in terms of performance for various reasons as well.

Yeah, once you've kind of built that base of SQL, Python, whatever your programming language is, and understand how to work with the terminal and command line, I'd say then you can start getting comfortable. I'd pick a few tools that are popular and companies are hiring for. I think that's obviously what you want to do. So things like Spark and Kafka, I think, are a good place to start. Also getting familiar with interacting with the cloud, however you want to call it, AWS and GCP, I think are always a good place to start.

Obviously Azure exists. I have my own qualms with Azure every time I use it. So I just like AWS and GCP slightly better. But yeah, pick up some of those tools, get comfortable, and then start building some projects on that. I think you should have enough that you can start testing out projects. You're going to run into more tools as you're doing that. You're going to be like, oh, I want to maybe test out dbt or SQLMesh or something like that for transforms.

You can play around with that. But yeah, I think initially just picking up Kafka, Spark. Those are still so heavily in use that even if they go out in the next three to five years, it's going to take time to migrate and things of that nature. And then pick some clouds just to get comfortable with and interact with. Great.

Yeah, I think that can definitely be more of a stage three, because Airflow is probably one of the most, if not the most popular, open-source solutions in terms of usage. We also know it best, so we tend to—I say we know it best; I'm sure there are other ones people know—but it's very well known, so we know the issues we dislike about it. So it's kind of this interesting space, right? Or it's in this interesting time right now where it's like, we have all these qualms with it, but also it's heavily used.

You're going to run into so many different data orchestration tools and data pipeline tools that you can pick some, and you'll run into a different one when you finally get hired. You're going to be using SSIS in some cases, and you're going to be like, oh my gosh, this is so old. But there are still companies using things like that, or Azure Data Factory, which isn't old, but these exist. And so I don't think you necessarily have to pick one.

But you can pick kind of one of the popular solutions for pipelines.

Yeah, I'd start getting comfortable understanding and talking to the business and understanding what their problems are and what you're actually doing. Because I think early on in your career, I think it's totally cool to just focus on tech and explore it, figure out what you like, figure out what you don't like, figure out how to improve performance of queries—do that. But you eventually need to start poking around and being like, okay, but why am I building this data model? What is it actually helping?

Who is it? Where's it going? What is this data actually being used for? I think asking those questions, the earlier the better. But if you're just focused on tech for the first year, that's fine. But once you can just start asking those questions and start trying to understand beyond just, like, okay, I'm building a data pipeline, but what is this for? Because I think that makes you a better strategic partner as you kind of grow.

And then that lets you open up new opportunities in terms of, like, you're not just this person that gets told what to do. You can start interacting with the business and being like, I think we should do XYZ, or these days, I think we should build this type of a data model, or we should start collecting this data over here, because that would help us on this side of the business and answer questions on churn or whatever it might be that your business is trying to answer.

Sure, I think that's never a bad idea. I think if you've ever talked to Ethan Aaron, he talked about how he would go around and do, like, a happy hour internally before he would do his low-key happy hours.

No, I mean, it's not the solution unless you're, I guess, an O-chem person. But you can find a non-alcoholic version. That was just a story. Honestly, whenever you take on a project, I think that's part of it, right? Just get involved and care. I think if you can do that beforehand, just what we did at Facebook is we would, every six months, go around and talk to our partners. We'd talk about what we did maybe the last six months and try to understand what problems they're running into and where they need help, whether it involves data or not, because we would try to understand, even if they don't know that data or something we can do can help them, can they tell us something about what they're doing in their workflows where we could maybe interject?

Ourselves and be like, hey, could we do something here to help you guys? We think if we did XYZ, that'd be helpful. So having those outreach moments is always good as well.

Yeah, I mean, I'd say one thing is probably, hopefully, you can spend some time understanding some of the language, especially if you're working with more early-career data engineers, just because they're probably not going to bridge that gap initially. So you need to kind of help them figure out where maybe they're talking or using language that, if they're interacting with a stakeholder, could be just a little too deep—how to kind of handle those situations and better bridge the gap.

Because I think that's one of the spaces that, like, I noticed even for my own self when I was especially early on. It's like you start getting too deep in the weeds and kind of losing people. And so I think that's a big place that always can be helpful. So if you can call out someone when they're like, hey, you're going a little deep. Is there a way you could say this that they could understand instead of thinking about it from just your perspective?

Like, how would you talk to someone who's never heard of even what a data warehouse is? There are cases where maybe they don't know what a data warehouse is, or when you say data pipeline, maybe they don't understand. Or when you reference Spark, they're like, why do I care what Spark is? Right? And really just helping them understand how to bridge that gap.

Yeah, I think there's definitely that aspect. I think in general, I found that if you're the data team, you're going to probably have to do a little more of the lift, just because it's maybe a little easier for you to make that leap, or maybe you just have more time. I don't know why, but it just feels like generally, in my experience, that the data side tends to be the one that has to bridge that gap. You also have, I think, an interesting view of the data where it's a—not data, sorry, the business.

You have this derivative view that is the data itself. So you can see the data, and it represents the business to a degree. So you might see spikes before the business even sees it. So if you can understand it, that can help you proactively just come to the business and be like, hey, we saw this spike. And we didn't just tell you we saw the spike; we dug into it. We think it's from whatever, store A or product B or feature A.

We think that's causing this problem. Instead of waiting for them to tell you, hey, we're seeing a decline in churn, you can already have proactively looked into it.

I definitely have tried to integrate, and have successfully integrated to some degree, AI into my various workflows. It's not to say that I've got OpenAI in everything, but it is to say that, yeah, it makes things easier. There are tasks I used to have to do. I was trying to do something as simple as create a CREATE statement where I wanted to rename the columns and already go through and unabbreviate all the column names. And I'm like, I could go through and try to figure out all the abbreviations and what they mean, but I'm like, I'm pretty sure if I put this into any form of LLM, it'll figure it out for me, and it does it considerably faster than I would.

So there are aspects where it will accelerate. I think there are tasks that are especially very basic that it's been helpful for, and I think it will be helpful for. I think in terms of risks, I don't think the goal is always to be faster. Code speed isn't always the goal. I think being clearer and being more understanding of what we're trying to design is usually the goal. It's like, what are we actually trying to build? Why are we building it?

So if we can get better at that, then I think a lot of these tools maybe pose some risk. But I think because there's all this technical knowledge I think you need to have to even be able to interact with some of these systems, in the sense that you need to know what to ask for, I think that still means a technical person needs to be involved with a lot of this work. So a data engineer is still going to need to be involved.

Otherwise, you risk people building the wrong model or maybe exposing data that they shouldn't be exposing. All this, maybe it's baked into the LLM, maybe it's not. And I think there's just a lot of risk if you don't have someone look at it. So maybe you need less data engineers in the future, but I still see that someone needs to understand what's going on. A good example of that is I did a video with Josué. I don't know if you've seen his content, but he does a lot of Databricks content.

And we were kind of going through and testing out Databricks Genie. And of course, maybe it could have been better configured. There's a lot of things about how it's set up that could play a role. But it ran into an issue writing a query that was not that long, I'd say, because there's like 5,000-line queries out there that exist. So I think this was maybe in the few hundreds, maybe 200 lines, and it had an issue in it somewhere. And had Josué not been technical, he wouldn't have been able to understand why that issue existed and how to correct it.

And he was able to correct it. And obviously then, from there, Databricks was able to correct it. So it takes time to do that. So I think we're a few years away from just replacing data engineers, which has been the goal for what feels like a decade or more now, since SSIS came out. They're like, okay, how do we get rid of these people that keep telling us to slow down with data?

I can, but it's weird because that's also something that I've thought about for a little bit, where it's like we've spent, it feels like, billions and billions of dollars every year just trying to extract data and put it into things like Snowflake or some other data warehouse. And it still feels like we're not great at it. Sure, good, but you'd assume we'd be better in some cases. It'd either be faster or cheaper. And in some cases, it's still not. I still run into people who, for certain ELT tools, they're like, it is way too expensive.

It's costing me way more than Snowflake. So I do still run into that where I'm like, wow, that's wild. You'd think that by this point it wouldn't be, but I still have people who are like, yeah, I'm only pulling from one data source, but because it's so many rows or something, they're charging us $20,000 a month for it. But in theory, you could hope that in the future you could write an LLM that, what do you call it, could write the API connectors better.

Because that's usually the big thing: the API connection tends to be the hardest.

That's something I've definitely been thinking about more recently, because when I first started in the data world, the common approach for a lot of companies I worked with would be, okay, we'll set up some cron jobs to run some SQL scripts, essentially, or some Python scripts that call SQL scripts, or PowerShell scripts that call SQL scripts, and we'll just time it all correctly so that they run in a specific order or set up dependencies somehow. And then what a lot of people would end up doing is just essentially building Airflow internally.

And so we knew the problem. We'd run into this problem where it's like these things don't talk to each other, and that causes issues. So then when Airflow, I think, came out, I think that's why it was popular for a lot of people. It was like, okay, cool, now we can write scripts and not have them bump into each other, or have them make it very easy to backfill and all these things that we find painful as data engineers.

And then somehow with the MDS, the modern data stack kind of push in 2020, '21, '22, we kind of ended up here again, where it's like, okay, we have one tool that does extract. Okay, we have one tool that, from extract, does transform. Okay, from there, you kind of can keep parsing it down. We have another tool that does data quality. Part of this works because I think it grew the pie, where now we have more companies that can actually access these tools, because before it's like, okay, you want Informatica, let's talk six, seven figures in contract costs.

Now it's like, okay, instead, let's talk a grand or two a month. Then that seems more reasonable. So I think you've expanded that. But at the same time, yeah, we kind of still built this very disparate set of tools. So I think some of it is our own—we're trying to constantly iterate and make things better. But by doing that, we take a few steps back occasionally. And so now I think we're relooking at things, and people are like, okay, how do we get out of what we've built so that we build some simpler systems?

I mean, I have clients who literally, if I bring up more than two tools, will look at me and be like, can we do this with one? Is there anything we can do to simplify this? So I do think people are feeling it and they want less. But I think it's just, again, kind of this natural bundling, unbundling that happens as people try to make a better version of just one part of a tool. And then you'll kind of hopefully bring it back at some point.

Oh, also, just to add another point, I think the reason you see so many tools is, I mean, VC funding was pretty high. So you can blame yourselves.

They're not alone, obviously. It's like VCs, I think. We data engineers and technical people, we love new tools, we love shiny objects. And so we'll test anything out.

I think if they execute well, it's possible, right? I think that's been the big thing. Like, when I use Snowflake, for example, they have some integrations with things like HubSpot and other tooling. And it's nice. I've used it for a couple of clients. It makes it considerably simpler. But then there's always just little things like, okay, Snowflake has tasks, but how I set it up, they're kind of getting a UI now. So they're kind of getting to the point where it's like, okay, now you're getting into this dbt-, Airflow-esque space where maybe, yeah, you could be everything.

And Databricks has already kind of been there, where they have their workflows that you can automate, whether it's notebooks or something else. So they've had that. So I think it's going to come down to execution, where it's like, if they can do it well, sure, people will go for it. But if they don't do it well enough to where it's like, look, I prefer to use Airflow, of course we're going to stick with Airflow, or whatever tool you're using for your orchestration.

And if you don't have a good extract tool, you're going to use whatever they can find out of the box if they don't have enough engineers.

Yeah, I mean, I think we had a few large acquisitions, right? Rockset got purchased out by OpenAI, which I thought was interesting. For, I don't know, was it like nine figures, eight figures? I don't remember where it was.

Yeah. So, large acquisition there. Same thing with Tabular. So there's some interesting purchases, I think, right? We've kind of now set, in a weird way, Iceberg as the de facto—if we're going to use an open table format, that's likely the one people will, I assume, pick, since everyone's now supporting it, right? AWS is trying to do their own version.

Sure. I think the big thing is that it sets a standard for how you're going to end up storing data and interacting with it, which, instead of Databricks stores their data in Delta, Snowflake stores it kind of in S3, but also in their own formats, now you're storing it in a singular format, which then allows all of these other solutions to essentially interact with it. Like, okay, well, now Snowflake can just have their method of handling Iceberg, and you don't have to think about it.

And same thing with Databricks. You can just have one data store and then pick which data compute engine you want to use, which is actually something that Facebook, when I was leaving, was already doing. They had kind of an HDFS sort of underlayer of where they stored their data. And then from there, you could just say, like, I'm going to use Spark, I'm going to use Trino—well, Presto internally, but Trino for people externally. So I'll use Presto. We still occasionally had Hive jobs.

So if someone wanted to have Hive, it lets you pick those. So you're not locked into a specific vendor so much. You are able to kind of pick which compute you want, which makes a lot of sense when I see things or when I talk to a lot of people where it's like, hey, we use Databricks for maybe the more expensive compute, and then we push it to something like Snowflake for our data analytics layer for BI work. So now, instead of having to do that back and forth, you just have one place you're storing the data, and you just have a catalog that needs to sit on top of it and tell everything else what's going on.

I think that's fair. And that's been kind of the recent fight between, I'd say, Databricks and Snowflake: trying to convince you that their compute is the compute to use for use cases or different workflows. It's really been a fight for workflows versus a fight for contracts, because in the past it was like, okay, you sign a $10 million contract for Teradata, you're kind of stuck with it, right? There's no choice now for the next three to five years. Now it's like, well, you could use Snowflake or you could use Databricks.

It's just whichever one convinces you it's better for whatever workflow you're doing.

Yeah, I mean, there's still people, again, I think I made a joke earlier, that have a SQL Server instance running underneath the desktop, essentially, and are just now thinking about migrating. I think I had three or four migration projects from SQL Server specifically to Snowflake. I think it's a pretty natural transition when they're going to the cloud. So yeah, there's all sorts of people and all sorts of tools. I think I also posted once, the more you think about Snowflake versus Databricks, the happier they are.

It limits your choices, but there's so many, right? You can use BigQuery. Not a big Redshift fan. I've not come back after getting burned in, like, 2016, 2017.

Oh, I think most people just, you go to the cloud, you expect certain benefits, and then there are little things. I think they fixed some of these, but, like, they didn't have a MERGE statement. You're still kind of stuck on a certain amount of storage. And so you just run into little things where you had to do VACUUM to clean up data that would get fragmented, or if you deleted it, it would be only partially deleted. So there's lots of little things where I think, because of who they, if I recall, they purchased a solution that was based on Postgres and then just moved it to the cloud.

Because of that, they were just so attached to the initial technology that they couldn't really change that much.

Yeah, no, I mean, and I appreciate that you call it the bias. Sometimes I think about that myself when I'm like, is everyone's data a mess? Or is it just the fact that you call a consultant when that's why you call the consultant? I do think there's that aspect for some teams, that spending $200,000, or whatever it's going to be, to pay for even a single data engineer is a big ask, right? Like, if your company's in the several million, $100 million, like, it's not nothing.

I like to think that the reason you have a data team is often because you've already run a decent business, right? Like, if you have, let's say, $500,000 to invest in data, which is what it might cost you depending on the size of your company, that means you've already run a business that has that much surplus, right, on top of your actual net profit. So you're running a good business as is without the data team. And so you're hoping that if you bring in a data team, you're seeing that 10x return on that.

And if companies aren't feeling like that, they're going to try to look for other ways to get it. Like, they still want to answer questions. All companies want to answer questions. I mean, if it's a $10 million or $5 million business, they're going to figure out a way to do it. So I think that's one reason you're seeing some fractional teams become popular, or at least more popular in my view. So it is kind of a little bit up to us to actually make it worth it.

We're also seeing that the ZIRP era is over, and people are just hiring less in general.

I'll try to avoid the "it depends" answer.

I do think if you're early, starting out, there's no problem in probably bringing on some sort of consultant to do maybe more of the data engineering work, and then bringing on a full-time, maybe, analyst to kind of work on top of that is what I'd imagine would be good. Have someone that can set up the infrastructure and then hopefully get it running, and then maybe have them come back occasionally. And then I do think it's valuable to have an analyst who understands the business, who gets integrated, who's actually involved, and not just kind of this someone that's a mercenary, so to speak, that's going to be gone and maybe wants to help you, but at the same time, it's just not as integrated in the business long term.

So I think that's, if you want to talk about a good baseline, have someone set it up. Does it have to be Snowflake? That just depends, to me, on the size of your data. I think what usually I see happen is, initially, people just report off whatever their transactional system is. So if you're using Postgres or something and you've got some application layer, you can report off that for a good amount of time to get your baseline metrics. But at a certain point, once you either need to get more funding or maybe once you get more funding, people want to see more metrics, right?

Like, VCs want to see more metrics. Your team probably needs to see more metrics for you to do better. Or maybe you just have a larger amount of data coming in because you're doing more transactions, you're having more users on your system. Then it starts making sense to look into, okay, do we need a Snowflake? Do we need Databricks? Do we need BigQuery to kind of handle some of this and handle some of your processing? And you'll probably start running into various issues.

And the issues I usually see is people run into, we only have a software engineer and they're kind of having to do this querying stuff. We don't want them to do that. We want them to be building features. So we're going to bring in someone to do that instead, maybe push it somewhere else. The reports are going too slow, or however we're doing it, it's too slow. So we need something that's more fine-tuned towards analytics, or we're trying to integrate more datasets, right?

Like, we want to bring in five or six different datasets. And right now we have an analyst kind of piecemealing it all together. So those are usually the causes that I see people say, like, okay, we should probably consider building more infrastructure, which might mean we need to bring a consultant or a full-time person based on kind of how much budget we have to build that, and then the analyst can work off of a more centralized kind of reporting layer.

I do think I tend to have things I dislike more than I like. There's a lot of tools. Snowflake makes it really easy, and I'd say BigQuery does too, but Snowflake makes it really easy. If you do not have a ton of people to manage things, I think it's hard to go against it. Similar with BigQuery. I think Databricks makes sense to me still more from an ML standpoint. I know they like the data warehousing side. I know that's where they're going to pitch themselves.

But I just tend to see, again, like I referenced earlier, the migration from SQL Server to Snowflake has worked really well, from what I've seen, for a lot of clients. Where the way I translate "it goes really well" is, like, when we migrated, they were only doing a certain amount of reporting. And by the time we ended migrating, they're already bringing in more data, they're already asking more questions because it's just easier for them to interact with. So I use that as kind of my baseline, where I'm like, okay, I'm seeing people just do more.

And that, to me, means that it works out well for them. So I always have a soft spot for Airflow just because that's kind of the first tool I ran into that made my life easier. Because, again, I came from this world of cron-scheduled jobs, and it just made things easier. So I think that always has a soft spot.

I think some of this I might have referenced, but I think, again, Iceberg's kind of cemented its stronghold, so to speak, right? In terms of where it will be used, where people are going to look for an open table format, that probably will be the one that they'll pick. Like I referenced earlier, I think at large enterprises, it gets really hard to pick a single tool or to pick a single initiative that's going to be run by one VP that everyone's going to agree to.

So I think in those places, you're going to see people pick other tools or maybe somehow have five different instances of Iceberg, or maybe one's using Databricks, one's using Snowflake, one's using AWS, and maybe they'll all talk to each other, maybe they won't. And I think the other side is a lot of VPs, directors, and even just data engineers don't all want to spend their weekends tinkering with all these tools. Some people love it. Some people love trying out all the new tools.

The example I gave in my recent article was I talked to a VP that I remember saying—he told me, he was like, "I like skiing on the weekends. I don't want to deal with something that breaks because we built some custom tool. So I want something that works. I don't really care if it costs an extra $5,000. Great, I'm going to pay for it." So I think there's that aspect of it.

Prediction number two, perfect. The next one I said: SQL isn't going anywhere. And I kind of said that for more than one reason. One, I feel like someone's going to tell us it's time to revamp Data Lake V1 again, where we'll just dump everything in there and we'll let an LLM figure it out, which I don't think will work. So I think that's why we'll still see SQL kind of be around. Maybe someone will convince us, and there'll be enough new people in the space and enough business people in the space that will be convinced that it's like, hey, again, we can get rid of data engineers in a possible way, or some layer of data modeling, and we can get faster access to data.

Yeah, let's do that, right? Saves us several million dollars. But I think we'll kind of end up in a similar space. And that's why even I made a bet in that space, more from my own angel investing side, where I'm like, yeah, I think some tools make sense, but others don't.

I mean, I obviously think it's interesting. I've seen the natural language-to-query kind of startups exist for a long time, and it's just been very hard. I don't think we're going to see that take over anytime soon unless you have your data in such a well-formatted way that, like, yeah, of course, it's easy to do. Like, you're very clear on maybe the terminology or something of that nature. So there's no confusion on what you're asking for, maybe. But even then, I don't know if I'm going to start seeing several-hundred-line queries be perfect from an LLM.

Maybe if it's, like, a very specific question, sure. But once you start digging deeper, I think that's where the problem comes. And that's usually the kind of barrier between self-service even now, is like, you'll answer your first question, and then there'll be like 10 more questions. And that's when suddenly they're like, okay, can I just put this into Excel because I'll answer it myself?

Yeah, I think we'll start seeing some AI go from press releases and press release-driven initiatives to real life, right? There are probably companies, and I know of some that I've talked to, where they've been building things. For the last few years, they've been quiet, they've been building things, they've been trying to build a layer of data infrastructure that will support doing things like deploying whatever they're doing in AI very easily versus having to cobble together something and make it very hard to even deploy something as simple as an ML model.

I think you're going to start seeing that become realistic. We talked about this before. People have done their POCs. People are trying to figure out where they work. If they've built a good foundation and they have some POCs that work, you'll start seeing that be realistic. I think we'll see, hopefully, things more than just customer service be the use case. Because that always feels like the first place people jump, is customer service. It's why we all hate calling anything at this point.

It's like, okay, I got to call the bank. I got to go through this, what they call a chatbot at some point, or some people call them phone trees. But okay, now I just get to talk to OpenAI instead, but it's kind of the same thing.

I think we'll continue chasing the same holy grails. I feel we want to see new things, but I think we're still trying to figure out things like self-service analytics or some of the other holy grails. Even being data-driven kind of feels like this holy grail that companies try to reach out for and, in some cases, get there, but in many cases just spend a lot of money and they're never really satisfied with the end. Some of that's just due to lack of data quality, lack of data governance, things of that nature.

Maybe they never defined what they really wanted in the end. They heard self-service and the salesperson painted an idea, but no one defined, okay, this is what we mean by self-service internally, and that's how we know a project is successful or not. So I think we'll just continue chasing some of those things because I still see companies struggling even there.

I think there's probably a few reasons. I think, one, we lose a lot of just brain—like, a lot of people leave, right? You have a lot of people who leave the industry before a certain point. And so you have a lot of new people come in. I say this as someone who, when I started, got very wooed by the idea of self-service analytics, right? I came in, Tableau was like self-service analytics.

And I was like, that's great. So I think that pushes a lot of the industry. Same thing with, again, on the business side, they hear a term, they're excited about it, they think it's going to work, and so we kind of look past all the stuff that actually matters. So things like, okay, but do we have the processes in place? Do we have the right people in place to actually make this happen? I think we're getting better.

I think we're starting to consolidate some of this knowledge. And I think that's the other part, is for a long time, all this knowledge was just everywhere, right? Yes, we had it all, but it was never consolidated into a playbook. It just kind of exists, right? Joe Reis only put out his book recently on data engineering, and I think the reason it did so well was because it was kind of the first book that consolidated information.

At a high level, it covered a lot of everything that you need to know. And I think that's why it was popular. So I think that's what we kind of need now, is just a playbook that becomes the thing that everyone's like, okay, this is how you run different teams. And I think there's always the business as well, where the business side pushes data teams one way or the other. And if you don't have a strong data leader that can either push back or make sure that you're doing things in a way that makes sense, you kind of can get trampled by the business side and what they need, and their kind of fickle nature where it's like, okay, now we're focusing on this, now we're focusing on that.

And that kind of can make things hard.

Yeah, I think we'll probably start seeing some vertical-specific data solutions come out, whether that's tooling or maybe just—one example I gave in my article was Tuva Health, which is trying to make it easier to just standardize healthcare, specifically claims data, which is something that I worked on a lot when I came into the industry. So that's why when I saw it, I was like, oh, this makes a lot of sense. Claims data in particular is decently standardized. I say that with quotes in the air because every company has their own system.

Some companies still have some older formats of data that people probably haven't seen, like positional files, where it just tells you column zero to 55 is this field, column 56 to 70 is a different field. But they've kind of tried to standardize some of that and build a vertical-specific kind of tool. I think that's something we'll hopefully see more. We've got all these tools that I'd like to think do maybe their job well, right? Like, okay, extract data really well, but they don't immediately drive value, right?

You still have to spend the next few months actually building something and translating that into some sort of business value. Just because you've got the data in your data warehouse doesn't mean it actually works. So I think that's kind of the next aspect that I'd like to see, and I hope to see, and we're starting to see. The big thing that usually holds that back is the fact that it's not customizable enough. So that's one reason I think some layer of open-source-esque access to the code will make sense, because people will always have these little, like, well, my data is shaped slightly differently.

I need a CASE statement for whatever, for how I'm defining this one product category. So it has to be this mix of flexibility plus standardization.

Yeah, like you said, YouTube, Substack. You look up Seattle Data Guy, I will show up there. Also on LinkedIn. But yeah, that's definitely where you can look me up if you need help or if you're just asking data engineering questions. You'll probably find some sort of answer in the content I've put out.

Oh, that's going to be a harder question to answer.

Also, it's like, who are we talking to? If you're talking to someone who's maybe in a data leadership position, I have a few articles around that. One's like, "Don't Lead a Data Team Before Reading This." That's just meant to kind of call out some things that I think are important. And there's a few other ones, like "Thinking Like an Owner," that it's like trying to find your data projects. So that's also on Substack. But if you're starting more and you're just trying to understand data engineering, there's a ton of content there.

One that you can look up is just the Data Engineering 100-Day Crash Course, which is just meant to be a bunch of content that should, I think, either be free or mostly free that you can just kind of go through and be like, okay, here's everything I need to know, at least from a high level, that gets you started in the space. It's not going to make you a data engineer, but by the end, you should at least understand what you do know, what you don't know, and kind of where you maybe need to dig a little deeper.

Really appreciate it.

Yeah, thank you. Thank you, Matt.