Deepset AI: NLP For The Enterprise with CEO Milos Rusic

Of course, happy to do that. So I met my co-founder, our CTO Malte, back then at university. That's 2013, 2014, 10 years ago. And after university, both of us actually pursued careers in machine learning. We have a background in applied mathematics, stochastic statistics. It was somehow very clear and an adjacent space to go into machine learning. So we worked for different companies in that space. And Malte was the one who actually worked for a company that was building online recommendation engines.

So natural language processing of an earlier generation, but really production-grade systems handling multiple million requests per day. And he was the one who excited me a lot about this discipline of using machine learning to understand language and process it in form of text or also speech. So in 2018, when we decided to build Deepset AI, the vision or the view on the world we were having was somehow very obvious to us. So all of these capabilities we see today, that we have very powerful models, that we can simply use human language to interact with our data, to send commands to a machine, this was pretty much obvious.

We didn't question use cases too much, if I'm honest. We had a little bit of a technological sense, if you will. If you're deep into a space and we spent then a lot of time actually in this space, you feel everything is moving so fast. You feel there's so much progress. It was a little bit like probably 2023 already for us. And we somehow felt that there will be big advancements in deep learning and natural language processing.

And they will probably enable a vast majority of use cases, and they will make it way easier and enable companies to be way faster in the way they adopt these technologies. What wasn't clear to us is how this new way of natural language processing advancement is in the end going to find its way into this vast majority of use cases. And in order to figure that out, we felt that we should probably start by building NLP systems for various use cases and enterprises.

And this is how we got started. And this is also how we financed ourselves until 2021, when we raised our first round of financing.

Exactly. We worked for companies like Airbus, many federal authorities in Europe, software providers in Germany, building all kinds of systems, all time and material. As you say, it was consulting. We went in, we scoped the problem, we looked into what do we find in open source communities, which tools are around, what can we use, how can we assemble and build a solution for them. And then really we started building that solution. So it was a great learning experience and at the same time also our financing strategy.

And while we were doing it, end of 2018 already, actually, our hypothesis became somewhat reality, right? You are well aware about what happened. Google released BERT, the first of its kind Transformer. We were among the early contributors also into Hugging Face Transformers. And this was when we felt, hey, this is somehow a new level of technology maturity, and this is what we were waiting for. So we exposed ourselves a lot to this technology. We only built applications for customers with Transformer models.

And then end of 2019, we somehow reflected and tried to find a common denominator in the way we were building these Transformer-based NLP systems. And this is how we came up with Haystack, because Haystack was—we understood that, look, there is somehow always a set of components you need, which is not just a model, but it's also, for example, a database where you store your text, and then you need to preprocess the text, and you need to serve the model with the text, right?

So we understood there are these different components.

Exactly.

100%. That's exactly what it is. You orchestrate all components you need, from models to databases. Usually a system consists of multiple models also in a row. If we look right now at retrieval-augmented generation, that's actually a nice example where you see you have your data, but then you usually have this retriever and then you have a model that generates on top of this retriever. So there are two models in a row, if you will. And to reflect such architectures and build such systems, this is where we built Haystack in the end.

Exactly. And then, of course, we focused on only continuing our consulting throughout 2020 by utilizing Haystack, so by using our own product. And this was a time when we helped actually many companies to really move into production with Haystack. And this was a new layer of learning that we made that we, again, same approach, somehow tried to bake into a product. And this is when the idea for Deepset Cloud came up because we realized that while Haystack is the tech stack to build these full-stack NLP or LLM applications, you still have a process that you have to follow, right?

There are still certain steps that you have to perform. You have to iterate over these components. You have to see which component is now the best fitting for my use case. What should I change? Where is it performing well? Where is it performing bad? Once you identified your ideal application, you have to move into production, you have to constantly monitor it, and probably you also constantly want to update it based on the learnings from production. And this is really like a full workflow that is required around the two lifecycles of experimenting and developing an LLM application and running it in production.

And it was also quite obvious to us that the only meaningful way to adopt NLP and LLMs is in the cloud. And this is why it was clear that this platform we're envisioning needs to be a cloud platform. And this is what we wanted to build. And this is then how we moved away from becoming a consulting shop to becoming a true cloud product company and then ultimately also a venture-backed company. Yeah.

I would say it wasn't as hard for us. On the one hand, the team wasn't super big at that point in time. I think we were around 10 or 12 people, but this was because we already used a lot of the money not to invest further into consulting, but actually to build out product and early versions also of Deepset Cloud. Actually, early versions were already built while we were bootstrapped. And then it's a bit—the initial aspiration: why do you start it?

And for us, it was always clear that we're looking for a way to actually capture some bigger opportunity that wasn't really around at that point in time. People weren't really aware of what we're talking about. No one knew the abbreviation NLP. LLM, this abbreviation wasn't even born. But we always had this aspiration. This is why I have to say it was comparably easy to, in the end, pull the trigger on this shift.

All right.

Mm-hmm.

Yes, let me explain Haystack best by focusing on a use case and then explaining what it does. So let's say you have your transcripts of your podcast and you want to actually enable people that visit your website to ask questions about the podcast. They might want to ask, I don't know, when did deepset get started? What was the year when the company was founded? So what you have is, first of all, of course, a model.

But this model, let's say we take one of the GPT models, probably doesn't contain this information yet. So it hasn't been trained on your data. It doesn't have access to this knowledge. Now you have all of your transcripts and you think, okay, let's store them in a database, which can be, for example, a vector database, and serve my podcasts to this model. Now you will run into a first problem, which is the model will probably not be able to consume the complete podcast as such and answer this question, right?

And then the second thing is, you continue your podcast, you have millions of episodes, thousands of episodes. That's a very, very large index. So your computing time is going up if you are serving all of your podcasts to this model in order to just find this single fact. So what you do is, on the one hand, you have to preprocess your podcasts, right? You have to chunk them, to cut them into small slices, store them in this database. And the second thing is you need kind of a filter that is on top of your database.

And this filter picks then probably only those pieces that are somewhat related to deepset or NLP and only serves, out of these millions of slices, only the, let's say, 10, 100, 1,000 most relevant ones to the model. And this model then takes those slices and generates an answer. And now you see a comparably simple application consists of a minimum of four processing steps, right? The chunking, the separate store, the retrieval step, and ultimately the generator or reader model. Now, Haystack gives people all of these components, and there are more than just these four, right?

For example, if you want to make sure that the model doesn't hallucinate, you probably want a hallucination detector on top, right? Something that classifies an answer and confirms: is this answer really part of my database? Is it a true fact, right? Or is it the right fact, the right year? At that moment, you have another component you add on top. Now, Haystack is the collection of these components and the logic to combine them. Under the hood of each component, you as a user can pick state-of-the-art technologies, right?

If you want to use OpenSearch, or if you want to use Pinecone as a database, you can simply load it in. If you want to use an open-source model from Hugging Face, or if you want to use GPT-3 or Anthropic models, whatever it is, you can simply use them in this full-stack architecture. And this is what Haystack is.

Some of the key ones, definitely, of course, the most popular vector databases, from Pinecone to Qdrant, Weaviate, Milvus. We also support, let's say, more traditional databases with vector capabilities like OpenSearch, Elasticsearch. It's actually changing also quite fast. So I often tend to struggle a little bit with what is now supported and what is in the making. But these are some of the most popular databases we see.

Of course, the OpenAI models, which are in particular very well-performing for all generative use cases. But we see the Hugging Face models from the Hugging Face Hub. The full hub is supported and they are still very popular. Of course, Haystack is an open-source framework. We're big fans of also open-source models. The more transparency we can create also around how models have been trained, the more research we have around the model, of course, the more trust it creates and the easier it is for companies to adopt.

So yeah, also Hugging Face, quite popular. But look, Cohere models, Anthropic models, all of those are also supported.

To be honest, as the capabilities of the databases also mature, it becomes more and more individualized. This is why we really invest also a lot in partnering with, for example, the database providers, such that it's somehow like a co-ownership of the integration, where also the database providers support us a lot to make it a good experience, to always keep the integration up to date. But yeah, the products, all of these products, are maturing and also differentiating, right? So it is becoming more and more of an effort, to be honest, to really manage this.

But this is also pretty much where the value proposition of something like Haystack comes in, right? That somebody takes care of it for the user, and the user can simply pick what they want or what works best for them for a particular use case. Great.

When you think about this, let's stick to the example we had in Haystack, right? We have this question-based search for your podcast. Now, the first thing you have to do is develop this application. That's the first big lifecycle phase you're in. And in order to develop it, what you have to do, and that's common in all of machine learning, right, is experimentation, right? So this means you start to somehow evaluate the performance of these applications and you try it out with different parameters.

For example, you will try out different models, right? You want to see, hey, for my podcast, what's the best option? GPT-5, or is it, I don't know, one of the T5 models from the Hugging Face Model Hub? And in this moment, you need some process. You need to follow some process to evaluate this and to make that choice. Now, in traditional machine learning—this is traditional machine learning, whatever that is—but think about a time series problem, time series prediction.

It is straightforward because you simply take a past time series that you observed about the phenomenon, let's say a stock price, you run predictions against it, and this is how you can iterate over your model, right? But how do you do it for a question-answering system or summarization system, right? The application gives you something, and is it good? Is it bad? Is it what users expect? Do they like it? So there is some more complexity around how you evaluate the performance of your applications, right?

And it's a lot about having a, let's say, quantitative theoretical approach with gold labels in the more classic sense, but also actually being able to quickly run small end-user tests, run A/B tests, all of these things. So you need a whole bunch of infrastructure and tooling to actually perform these experiments such that you can make your ideal choice. And this is one big chunk of tooling that you can find on Deepset Cloud, right? And once you identify this ideal pipeline, you want to move it to production, right?

So on Deepset Cloud, that's a seamless experience. You press a button, you have your application in a full-stack API, and you simply integrate it into your website. But this is not where it ends, right? How do you make sure that in production you don't see too many hallucinations, you don't see a degradation of the performance, for example, when your data changes? Or let's say people talk about completely different technologies in five years. How do you make sure that everything is still in line with user expectations?

All of these monitoring capabilities are also part of Deepset Cloud. And you see, it's ultimately an endless cycle, right? So once you are in production, you probably will go back. You want to iterate again, new models come out, you want to test out new capabilities, want to see if it makes sense to update. And in the end, this full lifecycle is something that we support on Deepset Cloud. Why we call it a platform for AI teams is because we realize that it's not only a data scientist's job and it's not only a software engineer's job, right?

These pipelines, these applications always end up being part of bigger products: chatbots, websites, internal knowledge bases, own software products. And this means that you need to have a product owner and a product manager in place that simply orchestrates this and orchestrates, in the end, a full team that takes care of everything that is important to move to production. And it's not only the model performance, it's plenty of things that need to be in place.

Look, I mean, cloud infrastructure in general is, of course, a very mature space, right? And in the end, it's a lot about maybe even undifferentiated heavy lifting that is behind what we're doing, right? This means maybe companies could help themselves with all of these components that are around. But the question is, is it worth the effort? And is it really the purpose of a bank, an online media company, whatever it is, to build the teams, build the expertise, combine them, take the ownership of this, right?

As we have this expertise in making use of best-of-breed components. And again, we talked about the vector database and the models, but it's a good point that you're mentioning here. What about scaling? What about, I don't know, cloud infrastructure that somehow also needs to be set up? How do you make sure that your pipeline on Deepset Cloud scales if it scales from 1,000 to 1 million requests or from 1,000 to 1 million files that you want to index, right? All of this is definitely nothing that we invent from scratch, and there are great tools, great frameworks, great companies that also develop this out there.

So, of course, we rely on their work for sure.

Yes, of course. So hallucination in general means you have your model and you want to, let's say, build a question-answering system. You have a question, you want to know a fact, right? And now you want to make sure that this model gives you the proper fact. And what we see is that models maybe don't give us the right fact. Maybe they even don't know this fact. We often don't know it, right? Because it's often not really clear what the model has been trained on.

And this means, in the end, hallucination is more or less like, hey, does this say something that is simply not factual, right? This is something that is not true, right? Or not real. That doesn't make sense. The first solution to this is to augment the model really with your data, right? So when you want, again, like if you ask ChatGPT today a question about your podcast, it might hallucinate. We don't know if it has been trained on transcripts of your podcast.

And even if we don't know if it gets the right answers. So the first thing is really limit what data it operates on. And this is the idea also of retrieval-augmented generation, right? So you have your corpus, you hand over these documents, and you can make sure that, in the end, at least everything that comes out of your application and everything that the model in the end spits out comes out of this document base. This is what you know.

But still, when you ask the question, in which year was Deepset founded, in the transcripts of your podcast, it still can happen that it mixes things up and it says 2020, which is not true. And this is really the fact of, or the aspect of hallucination, is a big blocker to really move into production. If you think about industries like financial services, where we see many risk management use cases, even if you think about something like customer support, online media, right?

We have many users and customers out of newspapers and online magazines. You want to make sure that these things are behaving the way you want them to behave, right? You want true facts. You don't want any racist language or anything in there, right? So you really want to control for that. And this is why we introduced a hallucination detector. This means there is a way, of course, to somehow classify a response and see, hey, is this response really in line with your underlying database?

Does this really make sense? Will you find this fact that way? And what's the likelihood of doing this, right? And if you are aware of this, you have multiple options. Number one, if you see that there's a hallucination, you simply don't show an answer, right? And you say, okay, wait, that doesn't make sense. The second thing is you can learn and understand what it actually is, or what are questions or facts where we see a lot of hallucination, right?

That, again, can help you to somehow tweak it, to tune it, to add models to your application, improve retrieval probably, right, and make that more accurate such that you can really correct hallucinations. But we really see that that's a big blocker for productionizing also RAG systems today. And that's where we introduced the hallucination node. And, yeah, we want to give our users as much security as possible when they want to move to production, right?

Everything in the LLM space seems to be consumption-based. And this is why we also sell annual subscriptions of our product right now that are, in the end, packages of different, if you will, consumption behaviors, right? So, for example, a customer wants one use case in production and wants to work on three further use cases. That's something we can somehow clearly quantify, and we can somehow put it into one subscription. But technically, it's all about consumption, right?

And this is, in the end, I think, probably the best metric to somehow reflect the value that comes out of these LLM systems, right? Because, in the end, the more requests a model gets or an application gets, probably the more valuable it is. The more data you load into it, probably the more value you get out of it, right? Because it's running on more and more data, and it's somehow solving more and more of a pain. So this is why really also offering it in a more flexible consumption way is the next step for us.

And overall, we think it reflects the value that comes out of these systems.

Great.

It's really, right away, we're not following this open-core approach, right, where we say there are—so everything you build on Deepset Cloud, every LLM application that comes out, you technically can also build just by using the open-source framework, right? Because the architecture, the technical architecture, everything you need from API point—the one API point, which is query, and the second API point, which is upload the files data—all of this in between, that's open source. This will always be open source.

Everything that is tooling for a workflow, right? So, how do we support evaluating? How does the backend look to track A/B tests? All of these, if you will, organizational challenges, right? All of this management layer, these are things that we simply see as platform features, and where we think that's simply, of course, usually also something a certain group of companies requires, right? I think not everyone in the community probably will also use it because, look, if you care so much about the factualness of your applications, you're probably not a startup, right?

Startups, you want to ship fast, you do whatever works best, you assemble something, it's shipped, that's it, right? If you are a big bank, you want to be quite sure on this, right? So this is like a—you see, there is a bit of a different need and different requirement. And this is where it makes sense to also say, hey, these things don't have to be open source because, in the end, not everyone who wants to succeed with Haystack will require this. But, in the end, everything that you need to really have an LLM application, like technically the full tech stack, that will always be open source.

And this is right now our mental model. But, of course, it's not always as easy as it might sound right now.

Look, even while being in consulting, even then afterwards when we launched Haystack, we see strong adoption really from enterprises, from large companies. And this is pretty much how Deepset Cloud as a product also has evolved and has been built. This means if we think about the segment, it's really like an enterprise platform and enterprise product where you have multiple teams, plenty of use cases, really high requirements for your application. And from a sales standpoint, that means we're often not really selling to our community members, right?

So our community members are adopting Haystack, and maybe they work for one big bank or big media company, whatever it is. But what they, in the end, do is they are the technological trust, right? They say this is a standard and this is a framework, this is technology we can really build our applications on because it's robust, it's reliable, all of these things, right? All of these tick boxes. The buyer is, of course, usually someone higher up the ladder.

And here we see that to sell into the enterprise, what you need is you really need a use case, right? And you really need to have clarity about the use case. And the best person to sell to is the person that is responsible for serving a use case. And this is a persona that we refer to as product owners or product managers, right? It doesn't need to be always that title because especially for internal products, for example, business intelligence stacks, or if you think about knowledge management within enterprises, these things are products, right?

And there is someone who owns, let's say, the business intelligence or financial intelligence stack for a Fortune 500 enterprise. But this person is pretty much our buyer. This is the one who will talk to developers and say, hey, there's this platform, we can ship super fast. They say, I have all the tooling I need to individualize my application. It's built on this Haystack thing. What do you think about it? And then you want someone to say, oh, well, that's good stuff.

Maybe we even have a small application running, or at least we looked into it, we read articles on it, we did some benchmarks ourselves. Yeah, we should build on this. We can build on this standard. But in the end, the actual sale and the actual buy happens through this persona who really is in an urgency to ship also, right, on this use case.

So in general, given that, we see LLMs allow, in the end, for a multitude of use cases, right? And we see that this whole RAG architecture is probably the first one that will really see broad, massive adoption, right? Simply because it's all about accessing and somehow working with information that is already around. And this is why we see these are probably really the first architectures that will see mainstream adoption. And given that these RAG architectures are pretty much focusing on information industries with a lot of value, or really high value in information, they are the ones that are really likely to adopt.

And the first one, the first big industry or, if you will, vertical, are companies that are selling information. And MANZ is such a company. So what they do is they sell all legal information in the Austrian legal system to law firms, even to public clients, right, to the government. So that's what they sell: information and knowledge. And what they do with deepset Cloud is they simply build features that improve the user experience for their customers, that make it nicer for them to stay on MANZ, to maybe even offload more and more workflows to the product, right?

That drive their retention up, that drive their attractiveness up. We're working—

Yeah, exactly. So they're working on multiple features. One is exactly like a recommendation system. So you are a lawyer, you read about a specific circumstance. Let's say, what are VAT regimes for patents, right? In that moment, you read about it, it's also interesting for you to understand VAT regimes around IP in general, probably, right? Or around brands or trademarks, right? And such an intelligent recommendation system is one feature that they're running in production. They also are running right now a question-answering system, a generative question-answering system, really.

It's a true RAG architecture where lawyers simply ask questions like, which paragraphs should I look into when I want to solve a case for IP, like make a VAT calculation for patents or whatever, right? So multiple features. For Airbus, the case is something they are working on. It's probably something that will take a few more years because it's all embedded in this overall vision of single-pilot operations. And this means that one pilot in the cockpit needs to be able to have very easy access to all the information that is essential to actually perform the flight.

For example, how are certain conditions on the ground? Also technical things in handbooks. How do— I don't know how to—

What to do if this red light is on? All of these questions, right? And this is again where our LLMs, and in this case also Haystack, is a standard that enables Airbus to build such an application, right? And to really simply be more effective and overcome the shortage of pilots that we'll probably run into in the future.

I think there's a market in both, right? In the end, I think it's simply two segments with different needs. And what we only realize is, when you decide to build a product in that space, you have to decide a bit earlier and you have to somehow branch out because, again, certain requirements on, I don't know, how much factualness, how much control do I want to have over the application, all of these things. Of course, a Global 2000 is probably more likely to be willing to invest in this level of security also, right?

And that's actually why we focus on this segment. I think there is definitely also a market in selling to startups. It's probably less about allowing individualization. It's less about being a platform that allows them to also customize many things. I think it's more about offering something more standardized and easier to use, easier to run. Probably many companies that also offer point solutions will be very good often in that part of the segment. But in the enterprise space, we really see this value of a platform.

That allows people to utilize standardized components and build really customized solutions, if you will. And, yeah, in the end, I think it's a choice. I think there's gold in both segments.

What's the plan? The plan is definitely investing in our go-to-market. For us, definitely the last years, we had a very strong focus on developing and validating the product as well. Now we really want to double down on the product maturity that we're having. And that means go-to-market, and of course, given we're a European company, we also want to expand to the U.S. And that's pretty much where our focus is right now: to really build out these go-to-market functions and teams in the U.S.

There's a lot, as you know, right? It's really a lot of stuff out there. Look, in particular, I think we love this somehow new space that is forming also in the area of LLM observability, right? Pretty much what I was also talking about with deepset Cloud, this idea of how to monitor applications in production, make sure that everything works as you want. And I think, I mean, looking at the observability market itself, I'm always fascinated because there are so many players, so many big companies, new companies that are founded, right?

And observability is an endless market, it seems. And I'm very excited about LLM observability because I'm not sure if it's just like a small subsegment or if it's not actually its own massive market itself. And it's definitely something where we see more of the younger, newer players coming in. One company I like is, for example, Context.ai from London. But yeah, there are plenty of players around. But I think that's something that excites me, where this is going, how the products will look, how they will differentiate, because I think there's plenty of space for differentiation.

That's what excites me.

I think we have great companies in Europe, and it's great to see also that many, many, many AI players are getting funding. I mean, the Mistral round, I think it's great for Europe in the end, right? Hugging Face is also still some kind of a—it's actually still a European company, right? So I think we have great potential, great companies, great ecosystems. That's definitely going well. I think that Europe is simply a bit more conservative in the way they adopt new technologies, right?

And this is usually then a bit harder for these companies to grow in their home markets, right? Or, let's say differently, you can grow, but to exceed and reach very ambitious growth levels. So this is, I think, something that's not going super great. And I mean, there are plenty of reasons for that. I think it's definitely also, to a certain degree, culture, right? And maybe some skepticism is also healthy to a certain degree. But speaking as a founder of an AI company, of course, we would love to see also a bit more appetite for adoption in Europe.

And this is, I think, a bit hard.

On LinkedIn, Milos Rusic. I'm happy to connect. I'm also on Twitter, and Haystack is on GitHub. So, also happy if you visit me there. But yeah, happy about making connections after this podcast.

Thanks, Matt. Thank you. Thanks for joining us for The MAD Podcast. We're back here every Wednesday with new conversations with leaders in the machine learning, AI, and data space. And if you like this show, you can also find a video recording of not only this episode, but many, many more over on the Data Driven NYC YouTube channel. Thanks again, and catch you next week. Thanks for listening.