2024 will be the year of Generative AI in production in the enterprise | Florian Douetteau from Dataiku

So our company is trying to democratize AI for the enterprise, building a platform to make it simple to prep your data, to build your models, and then move them into production. So it's a platform that we provide, focusing quite a bit on the large enterprise. To share with you some numbers, we are still private, so we share numbers only when we need or want to. And last time we shared numbers was last September, and back then we crossed 600 customers. We were at 230 million of ARR back then, and we have more than 1,000 employees.

So we focus quite a bit on medium to large enterprises, helping them essentially make AI models in production, prep the data, like getting all of these boring things required in order to get value out of data.

I would say that it probably needs to be a differentiated answer compared to AI in general and generative AI and non-generative AI. So first, I think we should all have a common name for non-generative AI. Call it conventional AI, non-AI, traditional AI. I don't know.

Tabular AI. But it's not only tabular. If you get to a specific use case, like getting the images of wafers for a microchip company and predicting quality control, it is traditional AI. It's non-generative AI, but it's deep learning. So is it generative AI or not? We don't know. Well, that's the issue. But I would say that from my perspective, at least, the maturity of AI, as in having predictive models in production in the enterprise, has grown significantly in the last 10 years.

And I see multiple customers having hundreds of models in production and being happy with it. I would say it was not the case 10 years ago, and I think there was some maturity in terms of moving analytics to the cloud, in terms of understanding how to manage the cost of machine learning, or to distribute and give access to the data more broadly. And so not every enterprise is there, but there has been some progress. It's a continuum in terms of progress that I've seen in the last 10 years.

On generative AI, I would say that for the enterprise, and here I'm talking very specifically of enterprises leveraging generative AI for their own internal kind of use cases, building their own generative AI solutions, I would say that it's a mixed situation where most enterprises are testing stuff, are doing some proof of concepts and so forth. But we've got about 100 customers that use our platform for some LLM-associated use cases, and we've seen some use cases in production, but it's not the majority yet.

And I suspect that indeed 2024 will be the year where we see way more use cases in production and see some durable, real use cases associated with generative AI for enterprise use cases.

It's possibly obvious or not to the audience here, but indeed today in the enterprise, most large enterprises would have quite a bit of machine learning in production, leveraging traditional AI in order to manage use cases ranging from predicting the churn of your customers, segmenting them, targeting them, understanding affinity of your product with customers to recommend some products or do dynamic pricing, optimize the quality control of your assembly lines, manage supply and demand, and all of those use cases. And it's becoming fairly common. Why is it fairly common?

Because it's doable now. And that's the best way to actually optimize your business, especially when you reach a certain size as an organization. So this is mature in the sense that, in each industry, you would have the major player of such industry having done and implemented traditional AI in production. And that's how the economy is working these days. So AI is already making this work, meaning probably the electricity in this building is provided with—not AI per se—but there are indeed some AI models, machine learning models, involved in predicting stuff associated with the production itself.

Almost sure that's how it works. And on the other side of the spectrum, generative AI, which is how AI gets to be known by the general public, which is fun for us if you're a practitioner. Like, you've got AI as you know it, and now someone else is coming and giving another definition of AI, and your grandmother is talking to you about AI. I think for us that generative AI might be this additional layer on top of traditional AI, reaping and adding more benefit to it, more automation, and lots of promises there.

But yeah, it's the early innings of it.

Most important use cases are not fully scoped yet, but there is a mix of use cases that are brand new, use cases that are not associated at all with tabular data, which are about finding information, very much about documents and so forth. You've got also many use cases that we see that are just, let's say, natural extensions to existing traditional use cases. As in, if you already have in production some product recommender system, one issue with the last mile is that if you start having some sophistication there with multiple products to push to someone with limited channel, maybe your last mile of personalization is not so obvious.

If it's advertising, it's like, how do you craft the content as a display? If it's email, what is the content? If it's text, how do you actually craft a text message pushing for two products that would be personalized enough? And so this kind of last mile can be done now with generative AI. And we could not do that before because human time to actually personalize the content was not realistic. Meaning, if you have two unrelated products and you want to craft a very small text message which is very personalized per the profile and demographic of the individual, like, you don't have time to do all of those combinations and make them very personalized.

And now you actually can. But in order to do so, you still need traditional AI because if you don't have a product recommendation system, it wouldn't work. So in that case, generative AI is providing this last mile, this additional benefit in terms of traditional AI: the last mile of, let's say, personalization, for instance; the last mile of accessibility in other contexts and in other situations. It's like brand-new use cases, indeed.

I think that, if I'm intellectually honest, they have so many people telling them so much information that I'm trying to focus on one aspect that I truly believe in. Because scoping use cases is the job of already too many people on the planet, from my perspective. And so I'm focusing on one thing, which is back to basics. Well, you must be aware that you need some ROI to your use cases. Good. You're not stupid, so I'm not telling you that.

But you must be aware that, probably because of the evolution of technologies and LLMs out there, it's very likely that you will have to switch from one provider to the other over the course of your application, if your application is successful. Well, if your application is not successful, we don't care about it, of course. But if you plan to run it for one or two years, it's very probable that you'll switch from OpenAI today to Anthropic tomorrow and maybe Mistral—they're French—on Friday.

And maybe you will have to, for this or that country, if you really deploy it globally, you might have to self-host an open-source model because of whatever regulation. What do you think about this complexity moving forward? And what do you think about the overall state of your company when you don't have one application in production, but like five, 10, 20, 50, 100? Because it's actually fairly realistic for an enterprise to have hundreds of generative AI applications in production at some point.

More talking about this aspect of things.

Well, first I'll tell them that it's true: they might waste money. That's just a fact. And so, from my perspective, the way to approach it is pretty basic. Today, you might have multiple providers, multiple applications. Some of them are in dev, in prod. And the issue with generative AI is also that you may have unforeseen costs when moving to production because you don't anticipate the size of the context you're pushing at. You don't anticipate some user behaviors and so forth.

So you need some actual monitoring of cost and attribution towards users and apps. So my first step is telling them you actually need to have a gateway between your application and your LLMs in order to track all of the costs and understand, before moving things to production, how much you can anticipate in terms of cost. A real-life scenario is that even us internally, at some point, we built some internal use cases associated to sales calls or what else. And we realized that the running cost would be like $100K.

I was like, "$100K? No way. No way we are doing that. We have to optimize this and that." But indeed, without having some control, we could have clicked on a button and spent maybe not $100K, but like $20K or $30K before realizing it was completely stupid. And I think that it does happen quite a bit on the market right now. If you fast-forward to a world where you move those LLM applications into production, managing the cost is indeed a bit boring, but it's just one requirement, especially when we move from a state where we focus on the easy, obvious applications that are reaping lots of benefits to multiple applications for which the ROI might be more uncertain day one.

First layer of concern associated to security, to putting guardrails to your application, understanding what are the risks associated to that leakage of information. A lot of those things. I think there are solutions to that, but it requires some discipline. Potentially one of the roadblocks is indeed around skills, ease of use, and ultimately making things work. Because if we're again a bit honest, it's not that easy to make generative AI applications work in practice. We are playing a little bit dark science there where you get your LLM, you fine-tune, you RAG your stuff around, and you decide, like, okay, good, this size of context, chunking yes or no, this way or not.

And it works, and you're like, oh yeah, it's like cooking. It ended up working, and you stop because once it's good, it's like a sauce. You stop cooking. You're just like, yeah, I'm good. So it looks like that really in real life these days. So from my experience of this type of field, it's very realistic that six months or one year forward, we will look at us now and we'll be like, okay, they were playing a bit with new technologies, and maybe we will have more sophisticated ways to do things that will work better.

But today, there is a part of experimentation there. It's part of the roadblock. I think it's important for many enterprises to do so because that's how you innovate. That's how you will learn faster than others. That's how you will actually maybe get one year or two years ahead in terms of adoption of AI. But indeed, it's a bit of a risky business because not everything is working. And so us at Dataiku, we are working on making things simpler for the enterprise, but we just have to be realistic on the fact that not everything is working today in terms of making LLMs the next generation of things doing everything in an enterprise.

It's not magic yet.

Yeah, Ratatouille.

We've added LLM and GenAI capabilities recently, so of course they are the most exciting ones. It's like connecting to the various LLMs out there, doing prompt engineering, and an easy no-code UI to build your app. That's all fine, and I think I talked about it already quite a bit. So a lot of the interesting aspect of our platform is we decided to build one platform where you could do data prep and all of the data engineering very well, but in a way that can be very accessible by the business, so that business can be independent in terms of doing data prep, and to build a platform where you can also do AutoML at scale, AutoML and MLOps.

The vision is to have, in one platform, the full end-to-end lifecycle of data as it should be. As in, you connect, you prep, you build your models, you test them, you have all of the backtests there, you move it to production, you can visualize things at every step, and you can lifecycle all of that. Just because I do believe, as a practitioner, that it's easier to do it this way than having multiple different tools or different teams or stages responsible for that. And the other thing of the platform is, I do believe that there is a need to make this more accessible towards non-technologists, as in non-nerds, meaning people that don't love doing Python.

And I say that loving doing Python, but also getting old. So I do know that I'm a very bad coder today. And so I sympathize with everyone that actually needs and has some domain expertise, but won't get into code anytime soon, even with Copilot or whatsoever, and that still need to leverage their data in order to get things done. I think there is this gap that we are fulfilling in the market in terms of providing this platform for the enterprise.

One of the lessons was the lesson of cloud ETL, because it's not an obvious bet. Because in theory, you should focus on one aspect of data and just do the best product at it. But my own take, which was to some extent very personal, is that data is a plumbing problem where most of the time you waste is connecting things from one step to the other. And so, consequently, there is more value by helping people do each step, but consistently, so that you have one platform where you can get everything done.

And this bet, for instance, was, from a product design perspective, a bit controversial indeed. But nonetheless, we made it, and it led us to make some compromises because, indeed, you can't be the best platform at everything while doing everything at the same time. I would say it's a controversial product bet to do so.

Some of this audience would be familiar with data mesh. Okay, so why not LLM Mesh? Why not? Because really, if you think about the enterprise, fast-forward a few months, if not a few years, I think that most enterprises will leverage multiple LLMs coming from multiple vendors. You will have large ones, small ones. So technically, you should not call them LLMs, but whatever, easier for everyone. You would have some that would be fine-tuned. You would have some virtual LLMs where you have some RAG, and so they become an LLM augmented with some knowledge or information.

And so you have to manage all of those LLMs together. And ideally, you would want any kind of application from your company to be able to tap into any of those LLMs with some flexibility. So, having proper routing there, a virtualization layer for that, the same as a data mesh is a contract or virtualization layer for data. And so, talking about contract, it means that you need to add some security or conditions there in order to help them move to production. And then the idea is, can you add a universal layer to ensure privacy filtering, to ensure content filtering, to ensure cost control, given that privacy, content control, and cost are the things that are today the most variable from one LLM to the other.

And so if you want to virtualize them, you need to be able to have basic guarantees in terms of making sure they don't go crazy, either in terms of cost or in terms of content they push to the outside.

Yeah. So we built this in order to essentially integrate with all of the vendors on the market, integrating with Hugging Face, AWS Bedrock, OpenAI, and Google Gemini, of course, Anthropic, and AI21 Labs. Also with vector databases that can provide RAG for virtual LLMs, such as Pinecone, for instance, as a local vendor, but also, of course, LlamaIndex. And so all of this being an ecosystem of partners and vendors that our customers are using in order to build GenAI applications.

Exactly. And if you want to switch vector databases or LLMs between your open-source ones, your local ones, or from one vendor to the other, it's more like a dropdown when doing prompt engineering, easily test from one to the other. These kinds of things are important for the enterprise when moving things to production. We see quite a bit of usage patterns where people would start with GPT-4 for design and then decide to move to essentially something cheaper, like Mistral self-hosted or Mixtral self-hosted, when moving to production.

And so I think that you have to help these kinds of usage patterns.

When you build the models, especially the RAG models, there are lots of integrations because of the, let's say, data analytics part and the generative AI part. Because indeed, a big problem when doing an actual RAG is that you need actual data. And you need to clean your data and so forth in order to have the relevant data to use. And so at the end of the day, you've still got big data pipelines which can involve some traditional AI in them. And then you've got applications such as personalization or recommender system-type applications where you kind of mix traditional AI, a product recommender system providing the basic inputs that then feed into the generative AI layer to generate the output.

And so, in that sense, you chain them.

Well, from my perspective, on the traditional side, it's a fairly mature market where a lot of the movement is associated with the modernization of analytics, which is, from my perspective, a big movement of analytics requiring moving to larger datasets, moving to the cloud, more sophistication involving more ML than before, or more sophisticated ML techniques. And so, it's all about moving from the desktop to the cloud with more stuff. And so, we are part of this movement where typically our customers would also set up cloud data warehouses or data lakes such as Snowflake and Databricks.

And we would be a layer on top, helping the overall orchestration and ease of use by the business. That's a big movement we see in the market. And from my perspective, this movement is, let's say, fairly mature. On the GenAI side, it's, I would say, today more consultative-oriented because many companies are still figuring out the scoping, the cost, and don't have the resources or the bandwidth to do so.

Yeah, we were honored to be last year both AI and ML Partner of the Year for Databricks, Snowflake, and AWS, which is great, meaning we can work with everyone—Snowflake, Databricks, AWS, and so forth. Our focus is actually indeed enabling a broader user base in the enterprise to get faster there, like get faster to, let's stop, just do my own things on my laptop with my old tools, and move to the cloud, get more sophisticated faster, which is complementary to, well, and actually generating consumption and a faster drive for the cloud platforms below us.

I think it's really a matter of vocabulary. Do you think that VC can be a tool or should be a platform day one?

Okay, I didn't say that. And so, again, on our side, it was really about focusing on what is the problem at hand. The way I see it, enterprises need to see analytic pipelines, ML pipelines, and their apps as assets themselves, like IP assets they have to manage over the course of one, two, five, 10 years. And so previously, those assets were not considered as assets, more like programs done by some people you don't talk to on their laptop. But because they are becoming more important, they really need to be managed.

And so if you want to be the platform managing them, you need to have the ambition fairly early on to be kind of end-to-end there, hence being a platform. I think it's specific to a market where there was this kind of disruption of the lifecycle.