Glean AI: The ML-Powered Accounting Solution with CEO Howard Katzenberg

Yeah, so my background: I'm a former CFO. Before starting Glean, I was the CFO of Better, the mortgage company, here in New York. And then before that, I spent 10 years at OnDeck Capital, another fintech pioneer. In each of those roles, at least once a year, I'd task my finance team with trying to figure out where can we squeeze our vendors and where were the savings opportunities. So what they would do is first figure out who were the top 50 vendors, and then we'd print out

the last 12 months of invoices. By the way, just finding the invoices was difficult. In Bill.com, we could get them there, but if something was being charged to a credit card, it was probably in someone's inbox. We'd try to locate the contract, and that was sometimes challenging, but then we would do just very manual analysis of, all right, what are we purchasing? What are the trends? What's the unit price? Does the unit price match what's in the contract?

What's the volume trend? What's the KPI that's driving this service or product? And then we'd come up with a bunch of questions, and we'd also just pick up on, like, hey, we got charged a late fee here, or this thing just doesn't make sense. And we'd ask the budget owners, tell us about these charges. Do we need this vendor? Can we consolidate spend with these other vendors? Can we move to an annual deal? All the typical questions that you would ask in an FP&A-type budgeting session.

And we'd really grill them. And inevitably we'd find errors, we'd find consolidation opportunities, we'd play bad-guy negotiations, but we'd find about 10% of savings opportunities each time we conducted this exercise. Five million dollars in savings. So I said to myself, wow, that was a really productive exercise, but my team hates me. We just spent three weeks going through all these invoices and bills, but it's a very tedious manual process. And why is software not helping us? Because these are very data-rich documents.

They're machine-readable documents created by a piece of software, and we're just kind of reengineering what a billing platform knows. So that was my first realization. This is at Better. And we have a very sophisticated approval process. It wasn't just going to finance every time a bill came in; it was going to the department heads as well. And why were we not detecting the same savings opportunities that my team did during the typical AP approval process?

So the more I dug into that question, the more I realized, well, when they get the first email, it gets ignored. Then they log in, they do a kind of quick sanity check, and then they select all, approve. So they're not getting any scrutiny of the bill. But to the extent they actually did want to review a bill, there's no context or analytics to give them comfort that that bill should be approved or denied. And when I actually analyzed what our approval rate was at Better, 99 out of 100 bills were getting approved.

It defeated the whole purpose of the whole approval process. So I realized it presented a huge opportunity to provide analytics. If a legal bill comes in and it's up 20%, well, why? Did we add another project? Did the number of hours go up? Did a partner get involved and the blended rate went up as a result? So I saw it as a significant opportunity to be much more strategic and proactive at having these conversations and identifying that money before it leaked out.

Great.

Yeah, we took a first-principles approach to say, listen, if we're collecting all this data, what else can we be doing? So in addition to standard, what's called AP automation, which is bill comes in, invoice gets extracted, and then you can sync it to your accounting system and get it approved and get it paid, we were like, all right, well, how about we do vendor intake? So if a team wants to bring on a new vendor, let's have a flow for that and support approvals for that.

If you want to set up a budget for your vendors, let's set up monthly budgets. So when the bill comes in, we're comparing it to a budget, not just to last month, but we can get an alert that day that you were over budget. And with every bill, you can conduct a variance analysis to any prior bill to see what changed. So there's a lot of functionality that, again, took a first-principles approach to say, how do we just be much more strategic in terms of managing and optimizing our vendor relationships and spend?

Yeah, every business needs to pay their vendors.

Yeah, so I had the idea in late 2019. We had our pre-seed round in early 2020, right as COVID was kind of happening. We started building models to ingest documents. So from an AI perspective, what we need to do is, when we receive a document, determine what type of document it is. Is it an invoice? Is it a receipt? Is it a billing statement? We can gather intelligence from those types of documents.

If it's a contract, maybe it's an NDA, maybe it's a remittance slip. That's not something that is going to get processed. So we have to determine what type of document it is first. Once it's a document that we know we can analyze, then who is the canonical vendor? So we had to build models, and this is all done with NLP models, back in the day before LLMs existed. But who's the canonical vendor?

We had to differentiate between Google Ads versus Google Workspace versus Google Cloud and all the different taxonomy that exists with that at the vendor level. Then, extracting all the elements off of a page. It could be invoice date, due date, all the various fields. But the thing that made us very unique is we're extracting all the line items too. So what are you purchasing, for what dates? What's the unit price? What's the quantity you're ordering?

And for any specific item, how do we think about that canonically? And we've built a lot of AI around the mapping of raw text on an invoice to what is the true canonical product or service.

Yeah, initially we started with using, like, OCR vendors on the extraction piece. A lot of the mapping models were models that we developed in-house, and then over time, we saw that the predictive value of the OCR vendors was no longer adding the information value we could get versus an ensemble model of proprietary models, and they were not worth the cost of maintaining those relationships. So it became 100% proprietary, again, like an NLP ensemble model. But over the course of the last 12 to 18 months, we've done a 100% shift to LLM modeling for the complete stack.

And we're using Vertex and OpenAI, and we're starting to experiment with Claude.

Yeah, it's been great. I mean, the accuracy rates are better. Our cost to compute has gone down, and it is the data scientist. We don't need a team to do annotations anymore for the models. And it's just like, our gross margins have done two step functions over the course of the year as a result.

Yeah, so I'm still learning from my team on some of this stuff. One of the drawbacks of using LLMs is we're not receiving confidence scores on the extractions. So one thing that we've created is, we're pulling from multiple LLMs, and that's a substitute or a proxy for a confidence score. When our proxy for confidence score is kind of lower, we will send it to a human team for review and validation. And also anytime we see if it's a vendor we haven't seen before, so we haven't mapped

that vendor from a canonical perspective, it's going to go for a human review just for validation, at least.

Yeah, so I think two things. One, we have a really rich history of the diversity of invoice structure, like I mentioned before. What are the true canonical items? So for any product or service, what's the billing? Is it billed quarterly? Is it billed annually? And for each type, what's the pricing difference? We've built the data model that understands all the gradation that we see across the supplier landscape. We also like that data model supports, and I think we're unique in this regard, a view of what's being purchased, not just on an individual invoice, but over time.

So we, unlike our competitors, can feed an LLM a CSV file that doesn't just say, hey, tell me what's on this one specific invoice, but here's the history with this vendor. We're going to do the calculations because the LLMs are very poor at that. And then we can say—and this is in development right now—tell us all the interesting things. And then we can say, here are 10 examples of us, or 10 other files, but 10 great observations, and give us a text-based summary of the story of this bill.

And we've demoed this beta feature to a couple of customers, and it's wowing them. But no one else can do that because they don't have that history and the structure of the data model.

Yeah, well, we just did a case study with a company called LeagueApps, so it's fresh in mind. One of the suggestions—or we call them gleans. So we took the verb to glean insights and we made a noun out of it in our product. So you get your glean list. So they had a Salesforce negotiation coming up, and one of our gleans was basically pointing out that they were overpaying for—I forgot what the specific service was for Salesforce.

But then in this case study, he reported back that in the contract negotiation—and we provide benchmarking data, that's part of the service that we offer, because we can compare your pricing to what our other customers are paying. And we know we can compare the same number of seats, et cetera. So we gave him benchmarking data, and he reported back that they saved $20K versus their prior contract. So that's just fresh in mind. And then he also said he estimates that Glean, the value is like 2% to 3% of non-payroll spend in terms of how the team collaborates and how we help save across the board.

But the best part of my job is when I get an email from a CEO, not even a CFO, but a CEO who's become a power user of Glean, and they say, "You've helped us change our spend culture." That's the problem I hoped to solve when starting Glean. And people were acting like drunken sailors with budgets, and now the spenders are feeling much more accountable and acting much more like owners.

I think the value prop is twofold. What I was just describing was more the savings opportunity, like how better visibility, better collaboration leads to better decisions, and then that leads to more cash flow, and you can decide to reinvest that back into the business. The person who provided the case study, he also estimated that from an automation perspective, we're saving half an accountant, in terms of half an FTE from an automation perspective. So there is definitely a time savings benefit as well.

Yeah, there's a lot that we're working on right now. Every bill goes to get approved, and I think using AI, we should be able to say this bill has been the same, or within a specific parameter, within the last six months. And as a result, it shouldn't need approval, and we should suggest what the policy should be. So I think there's opportunity to, again, reduce the amount of work required and then become almost like the exception: like, hey, this is out of bounds, we're bringing it to your attention.

That's kind of one thing. Right now, we can tell you that you increased the number of seats on Zoom from 40 to 47 this month, but we can tell you how many of the 47 are being utilized. So there's forms of data integration, like API integrations, that we can do to say, all right, 40 out of the 47 are being utilized. Here are the seven people that aren't working at the company anymore. And then maybe have some type of workflow where you can cancel those seven.

So I think there's a lot of exciting stuff to use the intelligence that we have to save money and save time.

First off, I'm very transparent with my team about what I don't know, and I ask them to teach me. So I had a teach-in the other day where we were discussing the merits of potentially bringing the modeling in-house, and someone was very opinionated on it. And I was like, I need to get up to speed here. Please tell me. I want to hear both sides of the argument here because I'm fascinated by this. And he just appreciated this.

I didn't come with an opinion. And then I asked him, please, you're the expert here. I need to get more up to speed here. So I think you just need to be realistic with what your capabilities are. And I had a one-on-one the other day where one of the lead data engineers, who's making a lot of progress here, is kind of siloed. And I was like, at least once a month, let's promote your brand, what you're doing, to the rest of the organization.

Because I know everyone's interested in this stuff, so let's create a newsletter and send it to everyone. And he was like, oh my God, you think they're interested in all the metrics? And I'd be like, no, no, no, just show some examples of some of the cool work you're doing. But his eyes lit up. He's like, oh, this is cool. So yeah, give people the opportunity at your company to talk about this. It's really fascinating stuff, and it's going to be game-changing for industries.

So allow your employees to really engage with it.

Thank you. Hi, I am Tanya Dua. I'm the tech editor at LinkedIn News. I spoke with Matt last week, and we discussed how there's a software recession going on. Given that your product, Howard, you said people were spending like drunken sailors, seeks to inhibit that spend. What is this sort of unrealized advantage you have over your peers? Because there's a lot of AI-driven companies that are trying to pitch themselves. Are you eating their lunch as a side effect because you're helping people check their finances?

Our business is up over 3x so far this year. On G2, our rating is 5 out of 5. I can't speak to other software firms. All I know is this was a problem at every other company I worked at. And when I did discovery back in 2019, this was before even the crazy times of 2021, every CFO I spoke to said, yeah, we know there's leakage in our vendor spend. We just can't figure out where.

But if you could build software that could help pinpoint where the problem is and can promote collaboration and people asking questions, that's going to help us a lot. And that's basically what we've done. And to me, when I did the research in 2019, I was amazed that no other company built this feature. And part of it was that it was so hard, but now it's become easier with all the AI advancements over the last couple years. Hey there, Ed Manzi.

Thanks for speaking, Howard. One question I have is on the channels you look at for sales. Have you considered private equity firms or companies that do a lot of M&A around post-merger integration and helping them with vendor consolidation? I'm not sure if that's something you guys think about or when you think about it. Not yet. My biggest challenge is no one's heard of Glean AI. It's a great idea. And yeah, I think there's a lot of industries and different approaches that would work for us.

So it's a good one. Thank you. Great, great, great talk. I was wondering, you mentioned a conversation with one of your data scientists around in-housing your model. I feel a lot of folks are now thinking about using public models versus bringing models in-house, doing domain models. Anything you can share regarding that process at your company that wouldn't be giving anything away? Again, I'm not the expert on it, but I ultimately came to the view that his perspective was that, from a cost perspective over time, we could have a lower cost than using the external models and continue to fine-tune and get better results by bringing it in-house.

And I think the latter is probably true, although I think their models will continue to improve. But also, I think their pricing will continue to improve. So we can't just look at where their pricing is today and assume it's going to be static over the next 12 months or 18 months because, as they compete, their compute costs will come down. I think their pricing will come down too. So that was the assumption I just disagreed with and why we decided, for at least for now, let's continue as is and we'll continue using the external models.

Thanks for your talk today. About the benchmarking service that you talked about, did you get any pushback from vendors about the service? And also from customers, did you get any pushback about whether to participate in the service because there's privacy and competitive reasons as to why someone should participate? In other cases, they may want to opt out. So what was interesting, what you may have heard from competitors, customers, as well as vendors? Yeah. So we're too small for the vendors to care yet.

And then from a customer perspective, on a very rare occasion we've had some pushback, but in our terms of service, basically the way it works is the customer agrees that the data on the invoices can be aggregated and anonymized, basically for their own use to provide benchmarking data back through our platform. They can opt out, though, so we give them the ability to opt out. If they opt out, they're not going to be eligible to receive the benchmarking data. Kind of had two parts.

I think one of them was kind of recently answered, but maybe I'll start with more of a comment. I'm a data scientist, and a year ago I had a friend whose wife was working at a company where they call all their vendors and try to reduce costs. And I didn't know that existed, but apparently it's a company that does that for other companies.

I would like that. That could be an amazing actual use case where you could be ranking, basically, from if you have 100 vendors, then you can rank maybe the top 10 that you should be renegotiating with. But the other question I had was more connected to the switch from building in-house to LLMs. Besides the cost, what about staff size? Did you see any, was there more moving people to different projects or anything?

We had a team of three, and then we brought on an external contractor to help, but not everyone was focused on that full-time. Given the cost and savings implications, it was a significant priority for us. To your first point, though, I was amazed that most accounts payable platforms and even contract management systems, the contracts aren't stored in the same platforms as invoices. So anytime you want to go embark on a cost savings exercise, there are two different platforms where you have to download the contract from and the invoices.

Not with Glean.

Thank you, everyone. Thanks for joining us for The MAD Podcast. We're back here every Wednesday with new conversations with leaders in the machine learning, AI, and data landscape. If you like the show, you can find the video recording of this episode, along with many more, on the DataDrivenNYC channel on YouTube. You can find all the important links in the show notes.