AI: Accounting Intelligence | Pioneering AI in Law and Accounting

This week Isaac is joined by Noah Waisberg, founder of Kira Systems and Zuva, to discuss his journey from lawyer to AI entrepreneur. Noah shares insights on the evolution of contract analysis technology, from early machine learning approaches to today's large language models. The conversation explores the intersection of legal and accounting AI, delving into topics like structured information extraction from contracts, risk scoring, and the potential for AI to enhance pre-signature contract review processes. Noah also offers his perspective on the build vs. buy decision for AI tools in professional services firms.

• (00:00) - Introduction and Guest Welcome
• (00:45) - Noah Waisberg's Journey from Lawyer to AI Entrepreneur
• (01:57) - Challenges and Successes in Building AI for Contract Review
• (05:04) - Intersection of Legal and Accounting AI
• (07:04) - Deep Dive into Contract Review Processes
• (11:01) - Evolution of AI in Legal and Accounting Tech
• (16:06) - Early Days of AI Contract Tech Industry
• (18:57) - Teaching AI to Understand Contracts
• (24:21) - Leveraging Open Source LLMs
• (25:59) - Co mbining Old and New AI Technologies
• (30:59) - The Importance of User Interface in AI
• (34:16) - Build vs. Buy: AI Solutions for Companies
• (40:24) - Exciting AI Applications in Legal and Accounting
• (44:19) - Conclusion and Future Prospects

Learn more about Trullion
https://trullion.com

Learn more about Zuva
https://zuva.ai/

Connect with Noah Waisberg
https://ca.linkedin.com/in/noahwaisberg
Buy Noah's Books
https://www.amazon.com/stores/author/B08QNG8B13

Connect with Isaac Heller
https://www.linkedin.com/in/isaacheller

Get NASBA Approved CPE or IRS Approved CE
Launch the course on Earmark to get free CPE/CE for listening to this episode.

Review
Leave a review on Apple Podcasts or Podchaser

Subscribe
Subscribe to the AI: Accounting Intelligence podcast in your favorite podcast app!

This podcast is a production of the Earmark Media

The full transcript for this episode is available by clicking on the Transcript tab at the top of this page

Creators & Guests

Host

Isaac Heller

CEO @ Trullion | Modern Accounting Technology

Guest

Noah Waisberg

CEO of Zuva AI. Previously co-founder/CEO of Kira Systems. Author of "AI for Lawyers." Canadian, dad

What is AI: Accounting Intelligence ?

Join Isaac Heller as he meets with leaders in the accounting technology space to discuss how AI and automation are transforming the accounting industry, how technology has evolved, and how AI can help accountants work more efficiently. You’ll also learn how accountants can embrace innovations to improve their careers and lives and get forward-thinking perspectives on where the accounting profession is headed when it comes to new technologies and AI!

Attention: This is a machine-generated transcript. As such, there may be spelling, grammar, and accuracy errors throughout. Thank you for your understanding!

Isaac Heller: [00:00:06] Hey, everyone, this is Isaac. I'm CEO at Trullion. And today on AI, accounting, intelligence. We'll be talking with real people about real things related to AI and the impact on the accounting industry. Stay tuned. All right. Hey, everyone, this is Isaac Heller from Accounting Intelligence. I'm really excited today to be joined talking with a friend. We'll call it a new friend. Noah Waisberg. Noah, how are you doing today?

Noah Waisberg: [00:00:33] Pretty good man. Pretty good. Uh, yeah. Good to be here. Look, I've enjoyed getting to know you over the past while.

Isaac Heller: [00:00:40] Yeah. So, so good to see you. Look, you guys are in for a really cool episode today. Um, you know, I know there's going to be a lot of accounting and kind of CPA related material, but interestingly, Noah is a lawyer turned AI entrepreneur. Really one of the early pioneers, I would say, with a with a really successful initial exit. And now onto his. I don't know if it's second or third company. I mean, we'll find out. Okay. Very nice. Um, you know, I know some people don't like hearing from the lawyers, but today, this this could be a really interesting story. My goal Noah-

Noah Waisberg: [00:01:12] The lawyers like hearing from the accountants just as much as the accountants. Like hearing from the lawyers, I suspect so.

Isaac Heller: [00:01:18] Okay. Very nice. So we've got it. We've got a duet today. We've got the the AI accounting voice, and I'll do my best to represent it and the AI legal voice, and I have a feeling we're going to intersect a lot, which is kind of one of the cool parts about this AI journey. And I actually learned about Noah's company earlier when I was in the accounting industry, and accountants started looking at Noah's tool, which it started in the legal tech. But we'll talk a little bit about accounting and AI. We'll talk a little bit about legal AI, how they intersect and have a lot of fun. But before we get into all that, like Noah, you got to you got to tell us a little bit about the journey because it's it's pretty cool. Where are you now and then how did you get here?

Noah Waisberg: [00:01:56] Yeah. For sure. So I started out as a mergers and acquisitions lawyer at a big New York law firm. I thought a lot of the work that I did and that I supervised was just not done that efficiently and could be done better. So back in 2010, I quit and I sat and I thought, and I started thinking about contract review, and I realized that it sort of fit what I was looking for pretty well, that junior corporate lawyers spent huge amounts of time reviewing contracts that even at the world's best law firms, they make mistakes at that, but that they're often looking for the same things over and over again. And so I got together with Doctor Alexander Hudek, who has a PhD in computer science from the University of Waterloo, and we set to work and from talking with Alex and other comp sci PhDs, we thought it would take us four months to solve the technical AI issues at that time and get our software really accurate at finding information in contracts.

Noah Waisberg: [00:03:01] And we thought it would take us like six months to raise money. So we're like, let's just build. And we set to building and instead, like a year later, our software did not work. And like two years later, our software did not work. But after about two and a half years, we got the tech to be pretty good. And, uh, then we faced a sort of second problem, which was getting lawyers who are mostly billing by the hour to pay us to use it. And there we kind of kept running into a wall over and over again.

Noah Waisberg: [00:03:32] Uh, interestingly, one of our ways around this wall was we ended up signing a big four firm and their audit side, who was looking for tech like this. Around the same time as we started to get some lawyers on and their support took our software in a slightly different direction, but also like really meant the world to us in terms of, uh, getting the business to kind of catch on. Um. Anyways, we kept going. Eventually, the lawyers sort of came on. Uh, we bootstrapped the company to like 100 ish people, then took in some outside funding in 2018, continued to grow the business.

Noah Waisberg: [00:04:10] When we sold Kira in 2021, we had 18 of the world's top 25 M&A law firms using our AI to pull data out of contracts. And we also had parts of all the big four, as well as a bunch of companies using us directly, um, as part of selling that company. We got to do a wild deal, which allowed us to keep a copy of the underlying tech in Kira systems, uh, the underlying AI and our training data. And we got to keep about 30 of our 180 people. And we spun that into Zubr, which is what I run now. And Zubr is pretty similar to Kira. It's, uh, right now, I think over time it'll shift to be different. But it's tech that, uh, uses AI to pull data out of contracts. Just not for law firms right now.

Isaac Heller: [00:05:03] Amazing. Amazing. Sounds very, very familiar, but I'm sure there's all different types of nuances about how you deal with contracts in AI, which we'll get into, I think for those of you on, uh, listening in. Um, Noah started his career, I think it was Weil Gotshal, which is one of the top law firms in the world. Um, and so, you know, you think of the parallels between the accounting and legal industry. It's kind of like getting into a Big Four level law firm, top law firm. Uh, and then and then going on to, to found Kira.

Isaac Heller: [00:05:34] Um, the other thing I remember is I remember Kira as it was, was growing, took that funding from Insight Partners, which is one of the top, you know, software investors in the world. So it's funny, you know, Noah and I know each other now, but at the time, I was working at a company called visualize. Um, and when Kira was kind of emerging and visualize was lease administration company for real estate, which then ended up getting into the lease accounting industry. But all of the data was entered manually from a contract perspective.

Isaac Heller: [00:06:03] So I started to like be curious of what are the people out there that are doing these things? And that's when I saw Kira. And so while you were moving from kind of legal to accounting, I was coming from, you know, the accounting side or the real estate side into the accounting. And Kira was that intersection point. So I think it's really cool that there was a lot of overlap that you're able to take, like that one legal kind of bend, but then apply it to the accounting profession. Um, but I'm curious like about deep diving deeper. So we've got, you know, big four accountants or, you know, controllers on the line take us into maybe a junior corporate lawyers world as it relates to contract review.

Isaac Heller: [00:06:43] And I'm, I'm thinking like there's two buckets I'm thinking about. One is like that eDiscovery of, like, going deep and finding a needle in a haystack. Maybe one is like transactional contract review where you're going through those one by one. I don't know if those are the right categories, but give us a sense of like, what are the different buckets of contract review that these these lawyers are experiencing.

Noah Waisberg: [00:07:04] So lawyers, as you can imagine, do lots of types of contract review. But I think there's two big buckets. And the way that I break them down is pre signature and post signature right. So pre signature is you're trying to think about should I sign this contract or should my client sign this contract. So I'm going to their feel like reviewing the whole contract and just seeing if there are things in it that I need to have in you know for example, my company has a policy that says X and then things in there that could be harmful for us. Like, for example, we indemnify we're doing a deal with Philip Morris, and we're indemnifying them against all legal expenses related to X or something like that. Like that might be something where you'd be like or like instead of Philip Morris. Maybe it's, you know, it's asbestos company or whatever sort of sketchy, harmful thing is out there. Um, that's where as a lawyer, you're going to be trying to think about, like, what are the risks and how can I avoid them? So that's the signature area.

Noah Waisberg: [00:08:09] Um, and then the other big bucket, and this is the area where I'm most expert is post signature contract review. And in post signature contract review. You are a company. You're a law firm representing a company who knows, right? But you got a whole pile of contracts that already exist, right? Like the company has already signed them. Maybe you're doing an M&A deal and you're thinking about buying a company that has these contracts. Maybe you're doing an M&A deal and you're a company and like you're selling your contracts and you need to know what's in them before you sell them. Maybe you're a company where a regulatory change has happened or something like that, or you think something sort of sketchy has occurred, like for example, uh non-competes have been banned in employment contracts, and you have employment contracts and you feel like going through them. Maybe you're a company who's trying to populate some sort of database, like a lease management database, or a contract management database or the like, and you need to go through and you need to find normally a pretty set menu of things that are in those contracts. So for example, if I was doing an M&A deal, I would care about things like the change of control implications of doing this deal, like what happens when you do the deal. Um, I'd be thinking about, oh, but maybe it's a financing.

Noah Waisberg: [00:09:27] And instead of looking at change of control, I'll be thinking about are there any restrictions on doing a financing or an IPO or something like that? Um, I'll also be thinking potentially about like, does this company have what it says it has. Right. Like if I was doing an M&A deal and someone was like, we have a 10% of our revenue is with Walmart, I would probably feel like looking at that Walmart contract and just figuring out how Walmart could actually get out of that contract. Like, is it like a sort of hard contract, or is it one where Walmart can terminate for convenience on 60 days notice? And like potentially the former might be more valuable or less valuable? Who knows. Right. And then a third thing I'm going to be trying to figure out is do these contracts come along with anything bad if it's an M&A deal? Right.

Noah Waisberg: [00:10:15] So do they have sort of the proverbial hazardous waste dump in the basement? In the case of a contract, that's probably stuff like exclusivity, non-compete especially maybe. And most favorite customer pricing, maybe Non-solicit. And some of those like indemnification limitation of liability type issues that I mentioned before. So that's what I'd be looking for if I was doing an M&A deal, if it was more like, uh, loading a contract management system, the company's just going to have a list of things that they feel like in there, and you'll pull from that list. But basically what you're tend to be doing with post signature is pulling out a bunch of pre-identified data and doing something with it afterwards, like putting it somewhere, sending out note taking action on it. Something like that.

Isaac Heller: [00:11:00] Very interesting. So you guys, yeah, you guys have always lived in the post signature world. Um, they're mostly there's definitely a pre signature component I think what I'm what I want to figure out just from your perspective, um, is like the timelines of AI contract innovation in legal tech versus accounting tech. My my thought even just, you know, looking at our journeys together is that the accounting tech was like a wave much after the legal tech, which would make sense because a lot of times the contract is is the center, you know, of the legal world. Um, but I just I'm thinking of a few things when you're, you're talking. And when I started my career in finance and accounting, the contract was obviously important. First of all, it was always post signature. I think one thing a lot of people can relate to in the accounting industry is it's always post signature. Unfortunately, I think there's some good teams that-

Noah Waisberg: [00:11:56] Yeah. They don't even bother to consult you on it or something.

Isaac Heller: [00:11:58] Yeah. There. Look, there's some good there's some good, uh, you know, teams out there, salespeople that will ask their controller, ask their CFO, maybe to ask their auditor, hey, before I sign this big deal, like, what are the, you know, ASC 606 implications? But I would say for the most part, you know, me and my accounting buddies live in a post-contract world. Would you say that it's the same?

Noah Waisberg: [00:12:19] I will tell you, like at my at our business. And I think I'm not trying to like make some recruiting poster for Uber or something like that. Zuma's finance team. But-

Isaac Heller: [00:12:27] check out Zuma by the way-

Noah Waisberg: [00:12:30] Yeah. No, no, no, I was saying, um, that one of the funny things about this is, is that I come at contracts from a Ylawyer's perspective. I've also been running a business since 2011, and I've raised money for a business. I've sold a business. And so there's definitely some things in there that I'm thinking about from a finance financing perspective, and I think we think about it from a finance perspective. Uh, you know, the big one obviously is kind of rev rec issues and stuff like that. And uh-

Isaac Heller: [00:12:57] Right. Well-

Noah Waisberg: [00:12:57] - they didn't have a contract.

Isaac Heller: [00:12:59] Totally. Well, you mentioned rev rec. Um. Rev rec those those those two big words when you were talking about, you know, use an example of the, I think used a Walmart contract. You were talking about a 60 day cancellation clause. And, you know, I, I came at this, um, you know, one of the, the ways I arrived at Trulia was because I had back to back career experiences. One involved a lot of rev rec and then the other involved a lot of leases. So like, naturally, my world is oriented around this. Well, you got to review a lot of contracts and you got to, you know, figure out how to put them into a database or a journal entry format. Um, it turns out that the software company I was working at, we were a pre-IPO travel tech company, but we were a very complex portfolio. And so the Rev Rec was just a mess because it was a lot of services and different product lines. There are some industries that don't have as much to do with Rev Rec, but I basically it felt like hell for me at the time, especially because we had done a lot of M&A.

Isaac Heller: [00:13:57] But what's interesting is this idea that when rev-rec came in and even leases, the contract just became more important to accountants. So the example of the 60 day cancellation, you know, hopefully all the accountants out there know that like that wipes out rev rec. If you have a 60 day out that now wipes out rev rec. So there is a quantifying. There's a clause that can be quantified that's brand new. When you're talking about leases you know your renewal options can can be exercised. And those can, you know, extend the term of the useful life or however your accounting of those leases.

Isaac Heller: [00:14:32] So, so those clause whereas before they weren't as quantifiable or maybe had nothing to do with revenue afterwards they matter. And I guess what I'm trying to think about with you is like what? You know, the the the contract in accounting. Did it come in because of the regulatory standards? Did it kind of evolve because I was improving? Like why did all of a sudden the contract become a little bit more interesting in accounting, or was it just inevitable?

Noah Waisberg: [00:15:00] I'm so I can't answer the accounting side thing. The thing that I will say is the big four firm that we were working with in the early days, which was Deloitte, was like pretty on top of this stuff back in 2014, like they were getting into using AI to review contracts back then, which is I think before all the lease accounting stuff came in.

Isaac Heller: [00:15:24] Interesting. Okay.

Noah Waisberg: [00:15:26] You know, a lot of that stuff. Otherwise, I think their idea was that just as part of an audit, you got to review a sample of the the audited company's contracts and that they felt like they could do this faster and better using AI. So it became, I think, part of their audit procedure back in 2014. So they at least were on top of this for a while. But due to some contractual stuff with Deloitte and the like, we weren't able to get to know too many other accounting firms for-

Isaac Heller: [00:15:59] I understand. Yeah, it's a small market, and of course everyone's trying to sometimes build internally versus third party vendors, maybe back up a little bit. Um, legal AI and legal tech and especially contract tech. I mean, you started an AI company in 2010. That's crazy. Like. That's insane.

Noah Waisberg: [00:16:18] -No, it's pretty cool. It was actually, like, before I remember writing a blog post on the implications of Watson winning on Jeopardy. Right? Like, well, what was we were doing this before then-

Isaac Heller: [00:16:30] Okay, so just a side note. So I started truly in in 2019. And even then people were like, whoa, that was so early for AI, which we'll get to what ChatGPT has done to everyone. But like when you started-

Noah Waisberg: [00:16:43] I am you know, in addition to everything else, I think I am the world's first author of a children's book on on AI.

Isaac Heller: [00:16:51] Wait, that's really cool. Okay, what's the what's the book called? Like, what's the book called?

Noah Waisberg: [00:16:55] Yeah, for sure. It's Robby the robot is Robby the robot learns to read that. We got to redo it. I, uh, if you get. It's like a board book. So it would only be interesting if you have, like, super young kids.

Isaac Heller: [00:17:07] Very nice. And you're based in Toronto these days or where?

Noah Waisberg: [00:17:11] Yeah. Toronto.

Isaac Heller: [00:17:13] Okay, so. So, Rob, we got to check out that book-

Noah Waisberg: [00:17:16] - Toronto hit me up and I could, uh, they're actually kind of hard to find copies. Now we need to do some changes in a reprint, but if you're ever around here, hit me up. I'll give you autographed copies.

Isaac Heller: [00:17:27] There we go. So, 2010, Kira is getting started. Where was the AI contract tech industry? Because I know there are some big like, legal tech. It was new. It was brand new. It was still lazy-

Noah Waisberg: [00:17:40] -it turns out when we started, we thought there was no one doing anything like we did. And I remember 6 or 8 months in, like my worst, one of my worst days in business at the time was realizing that there was someone else who was doing this. So it turned out there were a couple other people who tried to do something like that who had done something like that. There was another company doing something like that. This company, Seal Software, that ended up getting bought by DocuSign, that was kind of doing it simultaneous to us, more focused on the contract management side of the world. But there was one that kind of came and went. There were two that came and went before us, um, that had some different approaches to what we were doing.

Isaac Heller: [00:18:25] Interesting.

Noah Waisberg: [00:18:26] I think one was maybe a bit more pressing and then one was like also contract management focused and, uh, quite rules based. Like, they built rules to find everything as opposed to our kind of machine learning approach, which also differentiated us from seal, which was also try to use rules to pull this stuff out, whereas we were just hard machine learning all the time. That's it.

Isaac Heller: [00:18:52] Interesting. And I want to come back to those approaches. Another another question. So you come at it from a legal lawyer lens applying AI to contracts. One of your early customers you mentioned is Big Four. You know, Deloitte, when you were in those early days in the rooms with these accountants and these audit partners or whoever it was, did you like notice different ways they approached the contracts post signature in their audits versus a lawyer? Like, was it two different lenses, or are you guys were kind of in the same ballpark?

Noah Waisberg: [00:19:25] Um, I think in general, like the way that we were thinking about contracts is like people have a pile of contracts and they're trying to pull data out of them. And, you know, we know some of the ways that people can pull data out of these contracts, but we don't know everything. And so in fact, like one of our, one of our biggest business sort of unlocks in 2014 was building, uh, so pre 2014, the software could find what we had taught the software to find, and we taught it like 15 things. And then I think we got it up to like 28 different data points that it could pull out of contracts. Okay. And that was like that. Was it okay. Like that's what it could find.

Noah Waisberg: [00:20:08] And then we're like, but we can teach it more would be part of our our sales pitch. Right? Like we go into places and we'd be like, we can teach this more. And places would be like, yeah, sure you can. Like, great. You taught it to someone like you did it over the course of a weekend for someone, but it just wasn't that great. And in summer 2014, my partner got done this thing that we'd long been trying to do, which was enabling people to teach the software to find whatever they felt like themselves. And so one part of our sales process at this Big Four and at other places was we would go in. Uh, go in there and we'd be like, let's teach the system something new together. And they got to try teaching the software something new. And it worked.

Noah Waisberg: [00:20:54] And off we went in terms of them adopting us, but the basic way that we had thought about contracts by 2014 was you have things you care about in your contracts. We don't know what the like. Maybe those are one of the 28 things that we've prebuilt out. Maybe they're not. And if they're not, we've built technology that enables you to teach it something new. And so that's it. And if you think about software like Post-contract post execution contracts that way, it kind of doesn't matter what you're reviewing for, right? Like maybe you're reviewing for Brexit relevant provisions. Then, maybe you're reviewing for LIBOR relevant provisions. Maybe you're reviewing for rev Rec relevant provisions like none of those three were things that we had built out, but it didn't matter because we built the tools that enabled people to build more. And in fact, so over time, we ended up expanding our coverage of things that we found pretty significantly. So I think today you can find 1300 different data points out of the box instead of 28. But our customers between Ziva and Kierra have taught the technology to find 20,000 plus additional data points. So-

Isaac Heller: [00:22:08] That's fascinating. And it's it's crazy to see kind of the exponential growth from 28 to 13 or 20,000 over the years when you when I'm thinking about, you know, the question about like an audit partner and a lawyer partner, like in a room, like looking at a contract differently. It sounded like there was a lot of overlap in terms of the types of things they're looking for. I'm sure there's a little bit more of a dollar sign ringing in an audit partner and a little bit more of a risk, you know, red light, yellow light in a lawyer said. But overall, there's a lot of overlap. And the reason that's interesting is because when you get into these AI models, like they seem to end up going to be very like, in order to be adopted, they need to actually really deeply serve the end user.

Isaac Heller: [00:22:51] And so if it's too far adjacent, even though you're looking at the same thing, it can be totally off. I mean, I mean, because lawyers know what they want, audit partners know what they want. It just so happens there was a lot of overlap, which is awesome. And as I think about like how to actually train these models and how it's evolved, I mean, you're probably in the sky in terms of all the, all the evolution of this, but I just want to kind of share for, for everyone here. I mean, Noah is coming in in this period, 2010, 2014 where correct me if I'm wrong, you guys, you mentioned rules and machine learning. You had to train a lot of the models yourself. You had and probably still have your own models. That's correct.

Noah Waisberg: [00:23:31] Yes, we had our own, so we always did machine learning and what we built was machine learning. That was really good at finding words, paragraphs, sentences, data points in contracts. And and that was it. We also built some other stuff along the way. Like for example, we built technology that was really good at copying tables into Excel. Thanks to this-

Isaac Heller: [00:23:55] Interesting, by the way, that account- [CROSSTALK]

Noah Waisberg: [00:23:58] - no, totally. It's like a total I I've heard there's a big company that does that now, and it's like this funny thing where we kind of built something along that line back then. But yeah, we built our own technology back then and we continue to use that. So now we kind of use some of that as well as some, uh, you know, commercially available LLM stuff too.

Isaac Heller: [00:24:18] Okay, good. And I want to talk about that because I think that's a really interesting concept. A lot of people especially like in the past couple of years, your average, you know, I enthusiasm asked or maybe your recent AI enthusiast, a lot of I think a lot of people don't appreciate that a lot of these AI companies coming out today are leveraging the open source, these broad llms from whether it's OpenAI or Llama or Anthropic, and they're kind of embedding it in their tool. Whereas in the early days, no, it's fair to say you were building like an almost like an open AI for legal. It was like your LLM. Is that correct? Like and now-

Noah Waisberg: [00:24:55] Yeah, but we didn't- yeah. It wasn't even an LLM. Right. Like it was supervised machine learning.

Isaac Heller: [00:25:00] Got it. Okay. Got it. Okay. So supervised machine learning. And by the way, you mentioned you're in the Waterloo community. That's one of the top premier. Um, I think AI research and development ecosystems in the world, am I not mistaken?

Noah Waisberg: [00:25:14] Yeah, we've we've hired a bunch of people from there over the years. Great school.

Isaac Heller: [00:25:18] There you go. So shout out to the Toronto AI ecosystem. So, so, uh, you know, ten years ago you would have had to do a lot of your own supervised machine learning. Today, whether it's right or wrong, a lot of people can start with a little bit of, I won't say a lot, but a little bit of a head start with open source LMS. Sometimes you have best of breeds for for different use cases. I mean, give us a sense of like how it's changed today. One one vantage point would be like if you started Kira today, would you have to be doing the same thing, or would you be starting with an open source LM model and enhancing that? Like, how would it be different if you start an AI company today versus ten years ago or 15?

Noah Waisberg: [00:25:59] Yeah, I think if you started today, you'd definitely be using some of the like we spend two and a half years trying to get or we die trying to get tech to be really accurate at finding these sort of the text, the text classification problem of finding word to sentence to paragraph length text. And today you would have to do some experimentation with LMS, but um, but you'd probably get to the same place. Interestingly, we've seen numbers on one sort of for for from customer of customers did a sort of thing where they went and they tried to take an LLM and get it to work the sort of same way, and they were able to get it to work, um, at kind of around the same ish level to which we were able to get our tech to work. But the interesting thing about that to me is like, you know, you got ten years newer underlying technology and it's still like you're basically getting the same results that we were getting back in 2013.

Isaac Heller: [00:27:03] That's really interesting. It's the old [CROSSTALK] That's awesome. It's the old. They don't we're in the AI phase of they don't make them like they used to. You know there's-

Noah Waisberg: [00:27:14] I think there's like there's pretty cool stuff you can do with the new tech. The interesting thing about the new tech is it's just like super inefficient at getting to its answers. So it's like really, really, really cool. But it's like very computationally costly to get there. Um, and financially costly. So the old tech is like, because you are possibly relying on less powerful computers or whatever, you built tech that was a lot more efficient. And so one of the interesting things that we're seeing right now is that we can combine some of the stuff that we built once upon a time with Llms and use it to kind of generate a result, but with a lot less effort and maybe a lot higher reliability.

Isaac Heller: [00:27:58] Okay. Interesting. Yeah. So I'm you know, you mentioned like a lot of those original models that you built, you know, with Kira maybe a decade ago are being benchmarked against the new models and they're just as good and in some cases better, right, which is kind of an interesting paradigm. Or what are we saying? Yeah-

Noah Waisberg: [00:28:18] Well, yeah, the interesting thing is the new models, we can do some interesting things today that we couldn't do before. So one thing that we can do now that we sort of have struggled with for a really long time using the older tech, is we were always really good at Finding the relevant section of the contract, like you'd like to find the pricing section of the contract, or the limitation of liability section of the contract, or the termination for convenience section of the contract or tech. The preexisting tech was going to be pretty good at finding that part of the contract, but where we couldn't do anything was interpreting that section of the contract and what we and we had like probably like quite a lot of people working on this in 2018, 2019. And it just the tech never really got working to the level that we felt like having it go to.

Noah Waisberg: [00:29:09] And what we figured out with LMS is that if we layered them on top of a preexisting models, that we could not only find a termination for convenience clause, but we could also interpret that termination for convenience clause into, say, a structured bucket. So now if you use our tech, it'll not only find you the termination for convenience clause, but it can also group that termination for convenience clause into one of, say, five buckets like contract is not terminable for convenience, contract is terminable for terminable for convenience with prior note like ex set amount of prior notice and whatever the other structured categories are, and that's through a combination of our preexisting tech, which is sort of reliable, not going to make any hallucinations, and pretty accurate and quite efficient. And then layered on top of that, we've used Llms to kind of parse the results out into one of those structured answers.

Isaac Heller: [00:30:04] That's so interesting.

Noah Waisberg: [00:30:04] I think there's kind of cool things you can do.

Isaac Heller: [00:30:06] Yeah, definitely. And you know, in my head, the way I think about it is like it took so long ten years ago to get from like, you know, I don't know, 0 to 80%, let's call it. And then and then now you can but but let's say Kira or Zuva, you know, got to 90% and 92% or whatever it is now. You could start at 80%. But the challenge is, the more people are adopting AI and getting into AI, they care about not just 80 to 92%, but like 91 to 92. So those five use cases you mentioned as users are getting more sophisticated. They're like, well, wait a minute. You know, Zuba does this, so can't you do this? Like you're starting to compete not just on those 28, you know, fields, but like the nuanced 13,000 and then all the different scenarios. So I'm thinking, you know, you mentioned a few things about like just the advance-

Noah Waisberg: [00:30:56] No, actually like what I'd say is like, yeah, but really I think like one of the other things, as a buyer, you should be thinking about in this area is more like the interface, right? Like there are interface things that are really because you are getting to like say 92%, right. Probably recall not necessarily F1. Right. So you're getting to say 92% recall. But actually like maybe for some use cases you'd rather be at 85% recall, meaning like you'll have more misses, but you'd rather have less false positives, right? Because for that use case, maybe you're like just happier with that.

Noah Waisberg: [00:31:31] And then another use case is maybe, like, you really can't have misses. So that 92% recall, which measures the level of misses is sort of more important to you. Um, as well as perhaps a user interface that's really well set up to help a reviewer get rid of false positives, or maybe help you do other things like group relevant documents together, or assign and review work. One of the things we remember from dealing with the auditors was like, they were quite obsessed. Uh, like I mentioned, with being able to pull tables out and put them into Excel, they were really obsessed also with being able to like, tick and tie numbers back in some sort of workflow.

Noah Waisberg: [00:32:11] So I think that part of the thing with the systems today is like the tech, you can get to 80% a lot easier, but actually a lot of the hard stuff is number one getting from 80 to something higher. And then number two, putting the results in some kind of form that's really useful for your end user. And number three, wrapping it in an interface that suits your end user.

Isaac Heller: [00:32:31] Okay, so I think I think that's super interesting. And like we're we're I'm thinking now is, you know, I'm a I'm an accountant on the line or listening in or I'm an auditor listening in. And maybe I work at a, at a big company and they're doing some AI initiatives. Right. And how are you going to use AI on your team this year in 2025? And we hear it all the time. And I know there's this there's this tendency to build internally. There's these revivals of new tech and accessible tech, and AI is one of those. You mentioned that the cost equation has changed, so it's now much easier and more accessible to be able to deploy. Llms I'm oversimplifying it, right. You still need a lot of help. But then but okay. And you're previewing like how you feel about this. What I'm trying to get at um, you can you could potentially build something internally with an LLM because you say, hey, I could get 80% and I could save money.

Isaac Heller: [00:33:26] Um, then you mentioned, like, the interface, which, by the way, is really hard. Like, yeah, you can get a cheap LLM into your. I don't know, big four firm, but do you have a do you have a product designer that's like. Doing a bunch of research on false positives and you know, are you going to increase. Your number of errors because you guys built something internally without user experience? I don't. Know. But you know where I'm going Noah. Like I'm trying to think today with all these. Enthusiasm from CIOs and power users who want to build this stuff internally. Like is it? Is there a build versus buy equation or it's still, you know, look, look for the next Zuba or Kira. Out there. What are you thinking. And you could talk about legal and accounting. I'm interested in accounting and audit and you know how we feel, truly. And but there's definitely some use cases internally that I've seen that have worked and then others where you got to look outside your walls.

Noah Waisberg: [00:34:16] Yeah. So I the way that I think about it is if you can. But I'm kind of biased in this because I've mostly been on the vendor side of these types of questions. But if you can. If there is software that you can buy off the shelf that works for you, I think, and is really well suited to your use case. I think companies are often really well served by using that software. I think some of the advantages are getting that last, like you think you're at 80%, and this other thing you can buy maybe is at 90%. And it's you're not going to be that hard to get from 80 to 90. But I think this is like truly an example of the 80 over 20 rule, that getting that extra distance can be pretty painful and much more so than I think people anticipate.

Noah Waisberg: [00:35:04] But then other advantages to kind of a I always thought at Cairo we had this pretty amazing thing, which was we had most of the world's best law firms using our tech. And so that meant we were getting feedback from all these different law firms and accounting firms and alternative legal service providers and companies, and they were all in tons of different jurisdictions, like in the UK and Canada and the US, like tons in the US, but then also in like India and Australia and Brazil. And they were all seeing sort of like slightly different variations of similar problems and suggest like a good suggestion from a top UK firm might be something that a US firm would face in like six months or a year and they just hadn't faced yet. And because we were getting all this feedback from all these different people, it gave them, like back in the day, it gave all our customers, we think a real advantage of just kind of getting to experience something that a lot of different people were contributing to.

Noah Waisberg: [00:36:10] The other thing that you kind of get to notice is people notice bugs, right? Like just building software is inherently buggy. There's inherently edge cases. And the more sort of diverse a user base you have, the more you can kind of tease out some of those bugs and weird edge case situations and get pretty good at dealing with them. And as a customer, you might never run into these bugs, but you might also just be like three months away from running into these bugs. And if you're dealing with a vendor who has a whole sort of developed set of different users, I think it can, uh, it can really kind of flow pretty well. Um, and then the final thing is the vendor also can just afford, at a certain level of scale, you know, maybe not when they're tiny, but as they get bigger, they're getting money from like 50 different people to solve this problem. And so, like if you're putting in the same amount of money as, like one of those places or maybe even less than one of those places because you're trying to save money by doing it yourself, and they're getting money from like 50 of those places or 100 of those places.

Noah Waisberg: [00:37:15] And, you know, obviously money goes into lots of things, like it doesn't just go into development, it goes into, um, it goes into sales and marketing. It goes into SG&A like you know or sorry, general sales is in the but like goes into administrative [CROSSTALK] totally like money goes into lots of different stuff at a vendor. But like ultimately like if I've got 100 customers and you're building specifically for one customer, the level of investment that I'm probably putting in is probably pretty high-

Isaac Heller: [00:37:45] I think. Yeah. And that's that's like a huge reminder. And of course, if you're at a big company, you think you have more scale than, you know, a software vendor who's sampling hundreds of different customers. But I think, you know, people need to remember that AI benefits a lot from a community, both on the user experience and then also, of course, from the from the training. And essentially building internally can can resemble an echo chamber, unless it's something like super specific and unique to your company that can be easily deployable. You're missing out on like the benefits of a whole community, right?

Noah Waisberg: [00:38:21] Yeah, yeah. Like I think there are situations when it can totally make sense to build-

Isaac Heller: [00:38:26] Give us a couple and try to think, is there one in accounting and audit? Like if you're an auditor or you're an accountant, maybe Big Four or whatever where you could build internally?

Noah Waisberg: [00:38:35] Well, I think- with some of the big four firms specifically, I know that some of them have kind of like their own, uh, audit workflow systems. Right? Like workbenches. And so to the extent you had your own workbench, like question, if you could get a vendor who could sell you like an entire workbench like that might be something that'd be good, but you may not be able to get that vendor. In which case I think some places do have their own that they built out. And at the scale of one of those firms, I think that can make sense. Like they'll sometimes incorporate external technology and it may make sense to incorporate some external components in that, but I think that could make sense.

Noah Waisberg: [00:39:14] The other spot where I think it can make sense is like if you're dealing with a problem that you have that maybe not too many other people have, like Isaac, you mentioned at one of your earlier jobs, like before visual release, you were at a travel company, right? And it maybe there are some special quirks to being at a travel company that, um, that a vendor is just not solving. Right? Like, there might not be, uh, like Expedia owns or whatever, like Hotels.com owns, uh, every single thing out there. And so there's just not a big enough market where a vendor can kind of benefit from customizing to that market.

Noah Waisberg: [00:39:53] And in that specific case, like maybe you use an external vendor for some stuff where you can get it to work across all the stuff that you have, but for stuff that you do that's special and unique, then that's a really great spot to build yourself. And I think the nice thing is that today you may be able to get something sort of working with a lot less effort.

Isaac Heller: [00:40:15] That's awesome. Okay. Um, just maybe last couple questions give us like 2 or 3 things that you're excited about as it relates to applying AI. If you can give us like accounting and audit on one foot and then definitely legal. Um, but just give us a couple of things that you see on the horizon that you're excited about.

Noah Waisberg: [00:40:37] Yeah, I think to me, uh, I think it's super exciting that you can start to get structured information out of contracts. I think that the sort of generation stuff that we built the first time was like just helping you take a unstructured contract and turn it into basically semi-structured information. Like here is the termination for convenience clause, like this contract has a termination for convenience clause. Um, I think a lot of for what it's worth, a lot of the LLM results today are actually kind of ironically also semi-structured. Right. Like if I say to an LLM like across this pool of contracts, you know, tell me which ones I can terminate for convenience. And they say, well, you can terminate lots of them or. But they say it in a different way each time. You know, ironically, that's also semi-structured information.

Noah Waisberg: [00:41:28] But to me, the thing that's really exciting is trying to get the contracts down to more structured information, because if you can get things into structured information, you can do really interesting stuff with it, like for example, trigger an action. Like if I see this specific form of termination for convenience, then I would like to send it in for further review or something like that. And in fact, um, or you can derive richer visualizations, you can put things into tables and say like, I'd like everything that's in group A on category one and Group C on category two. And you know, this type of normalized after this normalized date in category three. And if I can combine those three things together, like maybe I can get much richer insights than I could before. Interestingly, it also opens up possibilities on the precinct side.

Noah Waisberg: [00:42:20] So for example, like now at Zoom, when we sign an NDA to speed things up, we've kind of used our own technology to build like a quick internal NDA sort of review tool, right? Where it just says, like, if the NDA has these three attributes, then it's got to go for human review. But if it doesn't have these attributes, we can just sign it. And so I can imagine something just thinking about what you're describing with accounting and how you only get contracts post-sale. Like I can imagine if a company had some sort of automated checker for their leases or for their contracts and it said like, hey, if it's terminable for convenience, it's got to bump up to a higher level, like, you got to get the cfo's approval or something like that.

Isaac Heller: [00:43:01] Oh, very, very interesting. [CROSSTALK]

Noah Waisberg: [00:43:06] - like you can put like you can save risk. Like you can do risk. If you can break things into categories, you can do risk scoring. Right. If you can do risk scoring you can do that pricing too right. Like you can do it post-sale. But what we've done for our NDAs is we do them like we send them through risk scoring software that we built ourselves, just because we're able to get our results into structured form. We use that to say like if a contract has a risk, then it requires a different level of review, right? But if it's sort of risk free as determined by the software, and there's like a bit of risk, uh, in having the software make the determination. But there's also a big plus because you can do it a lot faster, and it might spot something that a person would miss who was being lazy. Um, it just it's it's something we're we're able to kind of use that structured information to kind of do something we couldn't have done before. So I think I'm excited about structured information in contracts.

Isaac Heller: [00:44:04] That's awesome. And those were some inspirational use cases. So just to to kind of put a bow on it. Like for those of you out there, I hope it inspired a few entrepreneurs or entrepreneurs. You know, if you're sitting at your, your, your big corporate accounting or audit firm. Um, I mean, it was just really cool, you know, to hear the story that that that kind of end node of risk scoring of the kind of the timeline of AI starting with extracting those, you know, 15 or 20 fields, almost like a binary. Yes. No, all the way down to this, like red, yellow, green, like like quantified risk scoring, which I know that the accounting teams really, really appreciate. It's been a really cool journey and I hope it inspires more. I also just saw on your bio that you wrote a book, AI for lawyers, which was on the Wall Street-

Noah Waisberg: [00:44:47] Yeah, yeah. The Wall Street Journal bestseller on the topic.

Isaac Heller: [00:44:50] We've got we've got two book plugs. I think you should check out everyone listening. Should check out Zueva and Kira, of course. And a little more about Noah's story. And hey, maybe we'll do one of these again next year.

Noah Waisberg: [00:45:02] Yeah, this is totally fun. I would happily talk to you again. Isaac.

Isaac Heller: [00:45:06] Okay, awesome. Noah. Thanks for everything. Have a good day.

Noah Waisberg: [00:45:09] You too. Thank you.