Why Most Enterprise AI Projects Fail to Show ROI | HP, ServiceNow & Accenture

Show Notes

The “ROI of AI” has been marketed as a panacea, a near-magical solution to all business problems.

Following that promise, many companies have invested heavily in AI over the past year and are now asking themselves, “What is the return on my AI investment?”

This week on Chain of Thought, Galileo’s CEO, Vikram Chatterji joins Conor Bronsdon to discuss AI's value proposition, from the initial hype to the current search for tangible returns, offering insights into how businesses can identify the right AI use cases to maximize their investment.

Next, we’re joined by a panel of AI experts to discuss the ROI of Enterprise AI, featuring Alex Klug, Head of Product, Data Science & AI at HP; Sriram Palapudi, Sr. Dir, ML Platform Engineering at ServiceNow; and Jay Subrahmonia, Global MD for AI Research & Products at Accenture.

Together, they explore effective implementation strategies, how to measure the returns of AI adoption in the enterprise, and why AI's ROI isn't always just about the bottom line.

Chapters: 00:00 Current State of AI Investments

03:59 Challenges and Solutions in AI Implementation

08:30 Identifying and Prioritizing AI Use Cases

10:53 Ensuring Trust and Explainability in AI

15:29 Measuring ROI and Efficiency Gains

21:10 Panel Discussion Begins

21:54 Trust and Risk Management at HP

23:27 Accenture's Approach to Operationalizing AI

26:06 ServiceNow's Trade-offs and Prioritization

31:17 Measuring the success of AI for customers

36:29 Frameworks and Best Practices

40:57 Conclusion and Final Thoughts

Follow:

Vikram Chatterji: ⁠https://www.linkedin.com/in/vikram-chatterji/ Conor Bronsdon: https://www.linkedin.com/in/conorbronsdon/

Alex Klug: https://www.linkedin.com/in/alex-klug-67ba3655/ Sriram Palapudi: https://www.linkedin.com/in/sriram-palapudi-11294b1/ Jay Subrahmonia: https://www.linkedin.com/in/jayashree-subrahmonia-99963a/

Show notes: Watch all of Productionize 2.0: ⁠⁠https://www.galileo.ai/genai-productionize-2-0⁠⁠

What is Chain of Thought | AI Agents, Infrastructure & Engineering?

AI is reshaping infrastructure, strategy, and entire industries. Host Conor Bronsdon talks to the engineers, founders, and researchers building breakthrough AI systems about what it actually takes to ship AI in production, where the opportunities lie, and how leaders should think about the strategic bets ahead.

Chain of Thought translates technical depth into actionable insights for builders and decision-makers. New episodes bi-weekly.

Conor Bronsdon is an angel investor in AI and dev tools, Head of Technical Ecosystem at Modular, and previously led growth at AI startups Galileo and LinearB.

Introduction and Welcome
Conor: [00:00:00] welcome back to chain of thought. Everyone. I'm Connor Bronson, head of developer awareness for Galileo. And I'm here once again with Vikram Chatterjee, our CEO and co founder. Vikram, great to see you.
Vikram: Thanks, Connor. Great to see you again.
Discussing ROI of AI
Conor: uh, it was fun getting to spend some time last week with you in San Francisco, and really dive into what's happening at AI today as it continues to advance. So too does the conversation that we're having and other companies are having around ROI of AI. and you've talked about this before on this podcast, which is, you know, we need to move to some, Paradigm where, you want to get your dishwasher to actually be activated on its own.
you want real ROI for people. and we've had other folks in the show talk before, like, Brian Raymond at unstructured talk about, the mom test of like, uh, Hey, like, I just want Siri to work, But for companies, you really need to be able to [00:01:00] justify the spend when it comes to AI or else we're just going to be in this arms race that doesn't actually pay back. It's likely to be one of the biggest themes we see in 2025 around AI.
I think you mentioned it in our predictions episode. and we heard from leaders at ServiceNow, HP and Accenture on their approach to, the ROI of AI at our Productionize 2. 0 event we're going to share some of those perspectives here in the second half of this conversation, but first Vikram, let's drill in on your thoughts about the ROI component of AI. what's your take on how leaders today should be thinking about the ROI of their AI programs and investments?
Vikram: Yeah, good topic to talk about, Connor. So I think of it in two ways. One is, sure, what should they be thinking about and how they should be thinking about it? But also, what are they thinking about right now? And where is their mental state today? Like, what and why is that the case?
Current State of AI Investments
Vikram: Galileo, we've been talking to a ton of enterprise customers, but also a lot of these leaders that are in different phases of bringing their AI applications to [00:02:00] bear.
Right. And as a part of that, what we've realized is, um, the, discussion and this topic around the ROI for AI. Which has become a little bit of a, a meme in the world of AI now around like, what is the ROI for AI? And, you're seeing ads for that from a lot of different players in the market.
But all of this is happening because in my head, it's because of an adjustment to the actual value that AI can provide versus what the anticipated value was. And a lot of people overpaid for AI maybe even like eight to 10 months ago. And now they're just wondering like, Hey, is this actually going to pay off for me or not?
So just to unpack that a little bit, the reason why they had to pay a lot for AI was because The model providers that were highest accuracy, like the open AIs of the world, the Anthropics of the world, were much more expensive about 8 to 10 months ago. Now you do have, lower cost alternatives.
It's a lot of these open source models. A lot of our customers are fine tuning those models. A lot of them are not [00:03:00] fine tuning and just using rag with smaller models. And so things have become cheaper. The cost of compute, the cost of these models are much cheaper now than before. but the investments were made for a lot of the enterprises at the beginning of the year.
The budgets were allocated for this at the beginning of the year. So a lot of GPUs were bought, a lot of tools were bought, a lot of expensive consultants were brought in. And so if you think of it from that perspective, right, you're an executive at a large company and you're trying to build out these AI applications, your name is on the line.
You've done, now you've basically spent maybe 50 million dollars on launching a couple of these AI applications, but then you're not seeing the result. And that's kind of what, if you magnify that across hundreds of enterprises that have done exactly the same thing, there's this trough of disillusionment that's kind of come in, which is the natural cycle of things,
This ROI is just not there for this AI thing at all, which is why I find it to be very short sighted, but it's also because of what's happened over the course of the last eight to 10 months.
Challenges and Solutions in AI Implementation
Vikram: [00:04:00] what I keep talking to executives about and, and especially in the enterprise is, Great, like we've all learned that, you know, don't, don't spend your money without knowing exactly what the output is going to be.
We've all also matured as an ecosystem in terms of seeing what works and what doesn't work. And so when you think of the ROI, for anything, including for AI, it all comes down to being really smart on the inputs and the outputs that you want. So if you go back to what the outputs should be, we can now be much smarter about what kinds of use cases can you actually use generative AI for, versus which ones you can't use generative AI for.
So in the enterprise, we often hear about customers coming to us and saying that, look, we have 500 ideas across the enterprise for sales, for marketing, etc. where we can use generative AI. and we're going to be using it for a hundred of them. Instead, I feel like the smarter organizations kind of creating a rubric where they can just use generative AI for maybe four or five, which are higher ROI, which can really the needle, but also, are [00:05:00] built where GenAI can actually help, right?
So that's one, like figuring out those use cases really well based on maybe even the last 10 months of history is super helpful. how do you actually know that it's good enough to scale? Because a lot of the lack of ROI comes from this idea that, hey, this thing is not working.
I put in all of this money and time, but it's just not working. in the enterprise. And the idea of it not working comes down a lot to the guardrails that he put in place. a lot of the, different kinds of, mechanisms for iterating faster that he put in place so that you can actually, be accurate, but it can also adhere to the different rules and laws of the land from a regulatory and compliance perspective.
So those are the pieces that I'm seeing a lot of the more mature organizations do. And some of these organizations actually did this about 8 to 10 months ago when they started out. And those are the ones where we're seeing them being way ahead of the others. and so I think there needs to be a reframing net net of this idea for ROI for of AI.
And I think we're still in the first innings of many. I don't think it's the question mark is not like, is this a [00:06:00] useless, technology, but it's more about great. Now, how do we, How do we be better from here in order to actually generate more ROI?
Conor: Totally. And I know May Habib talked a bit about this in our first episode of Chain of Thought, about this idea that AI needs to be magic. And as you said recently on our 2025 predictions episode, a lot of companies were kind of sold AI as this magic solve for whatever their problem was. And the truth of the matter is it takes more work.
the potential of AI is huge. It's huge.capacity of AI to address different use cases, as you pointed out, could be 500 different use cases at a company. Huge.But it still takes work to define your goals, to set guardrails, to evaluate, train, and observe the success of this, because it doesn't get you to 100 percent off the bat.
You still have to put the work in, like you would of any normal software program.
Vikram: this reminds me of when the BERT model came out and everyone at Google was very excited about this when we talked to people outside of [00:07:00] Google. we were hearing a lot of, you know, BERT is great. But then I asked her a question and it's giving me a shitty response.
And that's because it needed work. It needed to be fine tuned on the highest quality data to make that actually work out. to get that magic value, which can be promised. It just takes a lot of work. And those were data scientists that were doing it versus now we're seeing a lot of software engineers that are leveraging AI.
And I do think the other gap that we need to bridge is how do you work with these? larger language models. How do you build these compound AI systems? How can you tweak them, iterate them, bring in the right guardrails to actually make sure that you can, squeeze out that magic value, which is latent in it.
and then bring it to the other side. And that's to some extent, it's the. think Craig Wiley had said this on the last productionize that we've done. I really liked that statement where he said, it's the science and data science that needs to be brought out. And people have to understand that and software engineers have to kind of learn that in order for us to all [00:08:00] really harness the true power of generative AI.
Conor: well said. And I know one of the things that a lot of software engineering leaders are thinking about today is finding that right use case because parts of their business have gone down this track already and installed, or maybe they've been more hesitant. How would you advise them to think through? identifying the right use case and those first steps to successful, building out of that use case.
Identifying and Prioritizing AI Use Cases
Vikram: Well, yeah, I mean, that for the use case identification for AI and ML in general, that's actually, in my opinion, not changed a lot for the last, decade. It's been three things. One is, uh,are you using humans a lot for this problem right now? Number one. Number two, is it super repeatable or are there way too many edge cases?
And number three is uh, that are, let's say, looking at, uh, documents for
mortgage lending, right? That's something which maybe can, you can use AI for, but the maybe question can be [00:09:00] answered based on, is it a repeatable task? Are these humans looking can be answered based on, is it a repeat? Basically it's doing the same thing where they're like, has wine spilled all over it.
Like They're doing all these things where they're just listing down these 10 or 15 different kinds of items. Where honestly a lot of the labeling companies made a lot of their money because they're just throwing humans at very bespoke specific questions and those humans have to answer those questions. So whenever you see those two pieces come in, it's ripe for generative AI or AI in general to be used a lot, which is why you're looking at low level code generation.
there's some code generation by developers, which is almost like art, right? And we're not talking about that code gen for the most part. When you think code gen, it's like not an application that's being built out from the ground up. It's actually the mundane parts of that. The same is true for conversational AI.
The same is true for contact center AI and summarizations and copy creation, et cetera. But the third one is high If you're automating the contact center, that's huge ROI for the business from an operations cost [00:10:00] perspective. So those three, I feel like have really not changed for when you're thinking about the use case.
That's when it gets into, how do you take those humans in the loop and move them from being the first line of defense to becoming more of the second line of defense where let AI. take up the bulk of the work, but then they become more of the subject matter experts and the quality checkers in the loop to make sure that there are no edge cases that are just seeping through the weeds.
Conor: And we'll hear more about this topic from Alex at HP and Jay from Accenture later in this episode. and another thing they really emphasize is the importance of that trust and explainability piece, something that we talk a lot about here at Galileo when it comes to enterprise AI,
What's your thoughts on how leaders should be thinking through? explainability and trust pieces of AI as they start to execute on these applications.
Ensuring Trust and Explainability in AI
Vikram: think when it comes to explainability of AI, I think number one, it's going to be a bit uncomfortable for a lot of engineering [00:11:00] leaders to come to terms with this because it's there is it is a little bit of a black box. It's not simple if else statements that you're stringing together, which lends itself to a very deterministic software based solution.
It's much more non deterministic than that. but if you can set the right kind of, bedrock and guardrails in place, things can help. Typically, what you've seen is there are four core components to making sure that you're, putting yourself up in the right way. The first one is, uh,Just having deep logging of your systems.
What that means is just making sure that your developers, as they are building other systems, are logging everything. And are doing that in a way such that they can actually track every step of the way as the model is making this decision. As an example, if you're building a RAG based application, then making sure that you know what chunks were retrieved from the retriever?
what the quality of those chunks are what the content is, what the exact response was like every step of the way got to know exactly what happened That's number one Number two is once you have that deep logging like [00:12:00] figuring out what are those right guardrails and metrics and tests and assertions that you want to make in place for your specific use case and Generating that and building that out from the ground up is very important and also taking testing that out to make sure that those are high quality.
Now to test those out, the third thing is you need test sets and data sets, and those need to be very high quality as well. That's classic machine learning. You need at least a set of queries, which are representative of your system, but you also need the ground truth, which is the right highest quality responses for those queries in place.
And this is an, you know, an evergreen part of the ML system where you've got to make sure constantly that those metrics are of high quality, that the test set is of high quality, even once you've productionized your application. So the fourth one though, is human in the loop.
these neural network based applications you always, they're always going to be edge cases. So, you always need there to be some subject matter experts in the loop, on the offline and the online side. And it's only when you get these four things together, the [00:13:00] logging, the metrics, the test set and the humans in the loop in a really cohesive workflow, that's when you can actually sleep at night knowing that, hey, this is trustworthy and I actually know that this thing is gonna work out on the other side.
And if it doesn't, then I actually have the answers because I've, tested it. set all these different shields in place for me to understand exactly what happened.
Conor: How can AI builders effectively communicate the complexity of these aspects and what they need to stakeholders who may expect this immediate magic?
Vikram: there's two parts to this. It depends on who the stakeholders are. we've seen a lot of the enterprises build a lot of internal facing applications, right? Where maybe the, barrier to being considered or barrier to brand harm is actually much higher, because they're just analysts at a bank, but now there are 20, 000 analysts at a bank.
So the ROI is high, but at the same time, the level of forgiveness is also high. You know, you can set some expectations through an email, [00:14:00] but when it gets to, external facing applications, like we have, a lot of early stage companies building out really innovative AI based applications, and then putting this out there for, you know, marketers to build high quality copy for, insurance providers to be able to process a claim faster.
It's all of these incredible use cases where. the, AI application starts to, just give out responses, which are not really high quality, there can be huge harm for that particular application. but that's where I feel like a big part of this comes down to the offline systems, like before you actually put something out there and if drawing what the boundaries are of what that particular application can and cannot do, and not just necessarily educating the, final stakeholder about this, because it's not the user that has to be trained on what the application should be doing, but training the application to not go beyond what it's supposed to do.
You know, as an example, if you have a chat bot for your website, to answer any question about your product, if somebody comes in and asks [00:15:00] about, I don't know, what's the meaning of life, it shouldn't necessarily say it's already do or read this book. it should just say like, I'm sorry, I'm not supposed to help you with that information.
I'm just supposed to help you with this specific information about this specific website. so it's a little bit like teaching the application to be, you know, within realm of what it's supposed to do. so, which is why I always try to flip it on folks who, think about how should I educate my user to know what to ask.
It's actually not on the user at all. They're just, they're just gonna go wild with their questions.
Measuring ROI and Efficiency Gains
Conor: I think this kind of speaks to the practical need to not only have this systemic approach to operationalizing AI and applying it to whatever your desired use cases are, but also on the challenge of measuring that ROI we've been talking about, how can companies, you know, Kind of showcase, not just numbers to say, Hey, this is working or hey, that's delivering value, but also assess efficiency gains and other more intangible benefits.
Vikram: that's a good question about the ROI [00:16:00] piece because, um,in general, I feel like a lot of the, People that are jaded by generative AI, they're looking for a very specific, metric that they can point to. And so you're like, look, this is what is, this is what it's helped with. that's where I've seen like, it, it's not necessarily materializing the form of, um, revenue or profitability, which I've seen a lot of people chasing when it comes to generative AI, but I don't think that's where that's where it really shines.
If you look at most machine learning use cases, we rarely have that, has that actually led to like, here's a net new business that you've built out using ML. so on the one hand in the enterprise, we're not seeing net new businesses being created based on ML, which makes sense because an enterprise, a massive organization, which is like, if you look at a bank, it's got, it's like 30 different organizations within itself, but they're all.
for financial services. And so you won't have a net new AI powered bank being built out. So it's not supposed to generate revenue or profitability. However, if it's geared towards cost savings and operational efficiency, that's where machine learning and AI [00:17:00] thrives.
It's for places where there's a lot of human in the loop, where there's some human work that's necessary that can be done away with. With BERT style models, which we built at Google, there were much lower level human in the loop tasks. Now it's slightly higher level humans and human in the loop tasks.
And now with GPT 4, and models like that, like it's gone. with slightly higher in terms of the kind of human in the loop tasks that it can take on. But those tasks are, if you automate them, that's meant to improve operational efficiencies, right? While on the other hand, you have these startups like Harway and there are so many others across, legal and, healthcare, et cetera, where literally all their millions of dollars in revenue are because of AI.
So they're not going to be the ones complaining about the ROI of AI, but the enterprises are. And that's where I feel like the main coaching would be, you got to count over the course of six to eight months around what was the cost of human capital that you were putting in for this specific [00:18:00] workflow?
How much time were they actually taking to do a certain task? And now how much time is that taking? And, um, How many humans in the loop do you actually need? And that's how you can calculate, the ROI to some extent. That's what we're doing with our enterprise customers too.
Conor: I feel like it's often the More digitally native enterprises that are actually identifying that ROI because they're used to having to kind of extrapolate, and say, Oh, well, look, look at this efficiency gain.
I can reach through this new innovative technology versus here's this direct. We sold more business units this quarter. they're saying, Hey, I just saved a lot of developer hours that now can be put towards innovative work. or I can limit my hiring in this area for a while because we've eliminated this, low level task.
Vikram: No, exactly right. and that's why I think we all have to exercise some patience, but also be very smart about the investments that we make with generative AI. I'm personally very excited about the trough of disillusionment that we've all reached because, uh, I feel like when we were on the other side, everything and anything was [00:19:00] possible, which is nice.
But at the same time, it was for, folks who've been in the world of AI for a long time, a lot of the use cases that were coming out of the enterprise where. They would even come to Galileo this particular application. We'd kind of tell them that, you know, like this is not a great application for generative AI.
And they realized that over time, but now we're seeing a lot of the folks coming in as being much more mature in terms of their needs and asks and what it can and cannot do. So that's typically what happens when you get like a whiplash in terms of like, hey, I invested a bunch of money and it didn't work out, but you get a lot of learnings as, as a result of that.
Conor: Fantastic, Vikram. I really appreciate you taking the time to talk through this with me as we head into our panel with some incredible guests. Do you have any last words for the audience to consider as they think about their decision making with AI applications and finding ROI from AI?
Vikram: the last word would just be that, it's a very exciting time to be alive. In general, I feel like when people have to understand that that it's a very powerful technology to [00:20:00] harness.
And because it's powerful, you can harness it in just the right way for the right kind of use cases. So if you're a technology leader at an enterprise and thinking about the ROI, I would say starting from the basics and first principles around, like, what are those use cases that are repeatable where a lot of your, team and especially human capitalists going into that.
It's repeatable. Those are probably good use cases for you to start working with. and the second part is keep an eye out on what's coming in the future and make sure that you have the bedrock in place, including the talent, the right kind of talent and the team can go a very long way. And there's a big talent gap right now in the market, which is also causing this, feeling of being jaded from where we are today.
Conor: Vikram, thank you so much. Listeners. Stay tuned. After the break, you'll hear from our all star panel. And while I've got you here, if you haven't checked out our YouTube channel already, please give it a visit. Maybe after the show ends or Hit pause and do it right now. We post all of our episodes here.
You can watch this interview and you can also see unique [00:21:00] outtakes and our favorite moments from each episode. They all make it to our YouTube at Run Galileo. Give it a like and subscribe. That way you never miss an episode.
Next up, we've got our enterprise AI panel from productionize 2.
Panel Discussion: Maximizing ROI of Generative AI
Conor: 0. You'll hear from Alex Klug, head of product data science and AI at HP, Sri Ram Palapudi, senior director of ML platform engineering at ServiceNow, and Jay Supermonia, global MD for AI research and products at Accenture, alongside Yash Sheth, co founder and CEO of Galileo as our moderator.
Nice to see you all.and thanks for joining us at Productionize today. how do we maximize the ROI of generative AI in,the enterprise today, so to start the discussion with you, Alex,
you're looking at. You know, trust and trustworthy development and avoiding, you know, the reputation risk in the enterprise. Uh,how is HP thinking about this?
Trust and Risk Management at HP
I mean, trust is so important to HP. We have a great, brand that's been around for a long time and, it's super [00:22:00] valuable to us.
So especially as we look at gen AI internally, it's critical that as we're deploying it, We're deploying solutions that we know are going to, add value to our brand and not have negative impacts. Like the worst thing that can happen is we are in the news about having deployed a solution that has steered our customers wrong or delivered a bad experience.
And in fact, our CDO often tells to me, you know, you guys are building rocket ships. I want a bracking system. And that's the mentality of a lot of our customers as well, is that we're always trying to move fast. The space is moving really fast, but what are the controls in place to ensure that you're delivering a good experience, that you have that accuracy and trust in the models and applications that you're developing?
and that's where we see, the partnership Galileo is being hugely critical because you get that trust by having explainability and visibility in the work. Is happening as may described. It can't just be a magic box where you ask it something and then some magical answer comes out. you have to have an understanding of where that [00:23:00] answer is coming from, and be able to protect against when there's a bad response potentially, or we don't have confidence that answer.
and so that's critical. And that's been a big gap in what I would say. has been preventing HP from deploying AI, a lot of our customers from deploying AI and just generally we're seeing in the market, preventing a lot of CIOs signing off and saying, Hey, I'm ready to adopt this across my organization outside of the one or two pilot use cases that maybe they've done so far today.
Accenture's Approach to Operationalizing AI
Amazing. Yeah. And, Continuing that thought, Jay, at Accenture, you know,we talked earlier as well, how Accenture is, supporting thousands of use cases, globally, when we look at it from the other side of, like, applying this technology in the enterprise.
you know, what's an effective strategy and what are some of the things that, teams should be thinking about early in order to get, to success with their applications and production.
One is, you know, last year was a year of doing a lot of POCs and pilots, and this year definitely is about promoting a lot of them into production. And that's where a lot of the issues around trust and, [00:24:00] concerns around brand, et cetera, are front and center. so we are definitely sort of helping our clients.
you know, take that journey towards operationalizing that principles and doing so with a very systemic and technology led approach, because this cannot be done in a more ad hoc manner. So, it needs a more systemic approach to doing so. And this is where sometimes our own journey internally comes as a good credential and to help.
Sort of done this. And in 2016, as the EOI act was just being formed, you know, we, grounded in our core values and code of ethics. Our responsible AI journey was, established and it's a large sort of CEO sponsored compliance program that has scaled over 70 to thousand people.
And in a nutshell, you know, we started with our governance and principles, but, you know, really mapping that into an AI risk assessment. We all hear about how regulations take a very risk based approach to looking at different use cases and then bringing in a very technology led program with systemic [00:25:00] testing and monitoring and compliance are all key sort of foundational, elements.
Why does this seem so complex and difficult? It's because the compliance rules and enforcements are still in their infancy. They're also rapidly evolving, you know, technology and vulnerabilities and broad scope and, you know, high volume of use cases that we're dealing with.
And this is why sort of, you need. a cross functional team absolutely has to come together across compliance, business and development and take a very step by step journey, towards actually going through this, this entire process. And this is where Accenture, like, companies are helping our clients sort of take that journey, from, establishing their principles to actually putting them into full operations.
Amazing. Yeah. Andand, Sriram, welcome to productionize. And I think we're going from, you know, HP, being, you know, the technology provider, Accenture, We'll see you later. Bye Bye you know, actually helping enterprises implement this and service now, now [00:26:00] actually, implementing a lot of generative AI solutions internally, you know, independently as well.
ServiceNow's Trade-offs and Prioritization
from your perspective, what are some of the, trade offs, that you have to make to kind of prioritize some of the GenAI applications, and the ROI that we get from that. Hey, nice to be here. Thank you for that. ServiceNow, it's a very, very, broad platform.
We have thousands of customers, like lots of other enterprise companies. And so with that, we do have to do a lot of trade offs. We are not at a place today, where we can have for each of our customers for each of their use cases separate models, right? So we have to look at what we can do to optimize across a different set of customers, what kind of models we have.
the end of the day, the infrastructure is also not like, you know, that there is limited infrastructure. So we have to look at the number of models we trade off on the number of models. We trade off on the sizes of those models. We try to bring together different use cases across the broad set of workflows we have and try to [00:27:00] optimize.
And that's where all our trade offs happen. We trade off on the size of the models. And then we obviously have to think about quality, right? So we try to bring all that together, keeping the third dimension of all the infrastructure. So that's the area up. Where all our trade offs happen, that's where we are trying to optimize and we are trying to make sure we get the best value for our customer.
does that involve a lot of, customer back and forth? Like, you know, when you're trying to prioritize certain features, do you have customer advisory boards, uh, that you get data from on what's higher. So we do have a lot of different use cases and obviously ServiceNow is a product that supports, uh,
So we have a lot of priorities that are coming to us from our customers, whether it's requester assist, as in the users of our customers trying to do stuff for themselves or our agents trying to. So, and then we also have a huge developer community. [00:28:00] We have a lot of people who develop solutions for customers.
So there is a myriad of use cases that we have. And we try to prioritize based on where we think the customer will derive the highest value at this point. So that's where there's a lot of back and forth with customers. We have a lot of our internal product and outbound product managers who talk to customers.
And we kind of prioritize based on where we think the maximum value for the customer will be. Amazing. And speaking of value, J at Accenture, I'd love to see. How Accenture thinks of measuring the ROI or the value of a generative AI application when you, pitch and deliver, that value to the enterprise in terms of the cost it would take, also, you know, the, the end ROI, for that application.
Absolutely. And I think, you know, when we look at the journey towards using AI based transformation, it's always starting with lead with value and then, enable with all the responsible AI guardrails. So, we, go very deep into an industry focal. So for [00:29:00] example, we are seeing a lot of, uptake with the regulated industries.
And I think we heard that earlier as well, things like banking and insurance and, you know, life sciences Partly because, you know, they know how to deal with, regulations really well and, you don't know how to sort of look at transformation with that. and then, you know, cross function looking at things like marketing, as a function that allows you to really target, customers at a much, much more personalized level.
But, you take an industry like insurance underwriting, last thing you would think about, from a AI based transformation, but You know, huge, a value that they are seeing in terms of how this is going to help them. And we see over 4 to 5 percent improvement in the ability to predict, risk across different, customers and partly powered by.
Traditional and generative AI, systems. For example, you, you know, underwriters have to read a ton of documents, property documents and, all kinds of financial statements and environmental reports. Great application of using gen AI and LLMs to be able to do that.[00:30:00]
Today they are only able to read things at a great level of detail, probably to about 25 to 30 percent, and here it just expands that scope. The second is to be able to predict at a very individual level what there is. Profile should be again, use of potentially traditional models to be able to do it. as these are coming together, high risk, because they have an impact on the individuals or an enterprise's, risk level, and so it needs to be implemented and deployed with the right set of testing and guardrails so that, you know, things like accuracy and hallucinations and bias and all of this are sort of treated appropriately.
personally also, I can resonate this a lot because just I was with a customer yesterday and, we were doing a value exercise, one of the things, that came up at the C suite level is like, we can only realize the value that we calculate, let's say if it's productivity gains or it's new customer revenue, if the behavior of the system.
Is as expected. And so investing in the responsible AI efforts, [00:31:00] having good governance, but also having good guardrails, uh, that you can, define becomes very important. Sriram, Alex, anything to add on this topic? I know, Sriram, you covered a little bit of the customer.
value creation perspective as well. And on the measurement side and Alex, from your perspective as well. Yeah. I mean, definitely. I think from an ROI standpoint, it's. Increasingly more important because so many companies have invested so much. And as we mentioned, I think we raised the, you know, the question earlier about how many companies are seeing that ROI, be able to understand not only, how you are getting to that ROI, what's the effort it takes.
Because I think everyone assumes that gen AI or AI in general is some magic bullet that can solve all problems and deliver returns, but don't think about the cost of getting there. But then number two, how do you measure that? And especially how do you measure that in a world that's completely new.
It's like a new systems. and then number three, like so many of our customers today, like, because of gen AI and the changes that this has [00:32:00] done just at a macro level and expectations that that is all part of your strategy, I mean, we expect that everyone kind of knows what's the latest and greatest, but so many companies.
They hadn't done AI up to this point and now they're expected to have this comprehensive strategy that's going to redefine their business, where they're just trying to get their foot in the door and what to do and take it very incrementally. So it's been a challenge for sure. for us, I mean, at the end of the day, we are trying to So we are trying to measure the success of AI for our customers.
And the best way we figured out for ourselves is to use data for that, right? So we, like I said, humans are using our stuff. So we are trying to see how much of deflection we cost. I mean, how many tickets were created in the case of ITSM. We have lots of other use cases, but we use measures like. How many people actually derive benefit of it?
How many people could actually solve their problems on their own? I mean, this is what customers are wanting from us, right? So that's one very obvious way for us to measure success and ROI [00:33:00] on AI. There are lots of other somewhat intangible benefits as well. I mean, you know, Jay was talking about underwriters looking a lot of content.
We have the same problem. Tickets can become very big. And it can become a very big cognitive burden on people to look at all that stuff. I mean, it, you know, ticket after ticket after ticket, it can actually dull you down. So a lot of some of our capabilities like summarization and all that help bring all that a little down.
So there's a lot of. time savings, which is obviously measurable, but then the cognitive burden on the person is also another value. So these are all the different things that we use to measure return on AI investments by our customers. Jay, I guess a popular question from the listeners as well, and as a relevant topic as well is like, Sharing some stats about POCs and applications in production as well as, I think when we, talked earlier, you had an interesting stat about CEOs, that I'm sure everyone here would love to hear from.
so when we [00:34:00] talk to CEOs, you know, first and foremost, across the world, you know, you ask them, do you have your responsibility principles documented? And usually you have a very resounding yes, you know, over 95%. But then when you ask them, have they put Operationalize this, you know, that number sort of dropped significantly and, you know, well below 10%.
and that's why sort of, we start with benchmarking our clients in terms of where they are. We've come up with this sort of stage 1 through stage 4, based on our some partnership with Stanford in terms of posture versus fully pioneer posture is where there is. Some level of basic infrastructure in place, some awareness to all the way to fully operationalizing.
So that's point number one. We also hear stats that say, you know, while there are a lot of POC that have been done only less than 10 percent have actually been put into production and partly, you know, It's because of not having the right sort of governance and the, testing and the monitoring in place to be able to make that shift.[00:35:00]
And so that is why I think we all here are here to not just help our clients build these use cases, but also help them. with that transition. Also at Accenture, we work with a lot of global companies and the regulations are just, growing and so being able and, and evolving. So how do you sort of keep track of those is also, a non trivial task for people.
Yeah. And, you know, earlier we were talking, with Greg from Databricks about, expectations, you know, at, from, from leadership and given, you know, how our POCs are very exciting, the use cases are transformational, we want to see them in production very quickly. Sometimes we feel like, Oh, you know, the POC is working so well, but actually productionizing it.
You know, we've expected to, uh, launch in like three months or six months, but, you know, the reality is not right. Like I've heard that the reality is a lot long. It takes a lot longer and would love to, you know, get an absolutely.
And also this is where the risk based framework really helps, You know, there are, some that are very high risk, but there are, much lower risk that [00:36:00] can definitely go through.
So not being a bottleneck, but every use case has to go through an elaborate sort of check before it even sees some light of the day is also, an issue. So taking a very risk based approach really helps a great deal. And, uh, you know, lastly, I think, one thing that I'd love to end with is the trend I'm seeing from all three of y'all is that, responsible AI safety guardrails, trustworthiness is important. You know, otherwise we can't generate the value that we are hoping for.
Frameworks and best practices
Is there a framework or are there some best practices that.
Do you all have seen work that we can actually help our listeners to, uh, to adapt towards and learnings from each 1 of you would be great.
we realized here is that we need to have kind of process where there are people making sure we are doing the right thing and we seek the right approval.
So we have a lot of controls in place. We have dedicated teams that engage with every team that works on AI [00:37:00] and Gen AI in particular here. What kind of data sets we are using, how are we building our models? Are we able to. Take out data sets, which kind of have an issue. You see, we are a SaaS platform.
We have access to a lot of customer data. but there are very, very strict controls on what we can do and what we cannot do. And there is a fully and well defined process. So I think at the end of the day, it's very, very important to define your process and make sure that everybody who's working on it is aware of that process.
You know, having something, but people not being aware. of it. So we drill into our teams, the importance of what data you use and what are the consequences of using something like that. And we have a lot of checkpoints in the middle. We have lots of people who can kind of know what's going on and with whom you have to seek approval before you do anything, whether it is open source data, whether it is internal data.
And we have processes to talk to our customers. We have contracts with our customers for them to sign. to allow [00:38:00] us to use the data for specific things, whether it's evaluation, whether it's training. So it's all very process driven. And we have a full team dedicated for that. I mean, whether you're a small company or a big company, having that process and having people dedicated to prevent you from causing any issues, I think is key.
Alex, you want to go? I'd echo exactly what Saurabh said, but then also add like understanding what that outcome is that you're trying to get. Is really critical in how you're going to get there, mapping that out into your process. I know, like looking at the comments around ROI, people brought some great points.
Like ROI isn't necessarily always just financially driven, right? There's a lot of like intangible ROIs that are a lot harder to measure, but you can, by understanding what that objective and that processes develop the KPIs to do it. You know, it's a great example from our team developing software solutions is like, how are we using AI?
To accelerate the velocity of developing like new features where maybe that feature has some positive financial return down the line, [00:39:00] because it's going to drive higher adoption or higher attention into our product. But in the short term, it's around like, how is the team refocusing on like the more important decisions?
Removing that cognitive load as people were talking about, much trickier to do, but when you're structuring that way, you have to understand that from the get go, like, what's that objective? How am I building my process? And how is that level of trust and support needed to get there? Um, because I, I, at the same time too, like, depending on what your use cases, that level of trust that you need also varies very drastically.
So like our financial customers, super high level of trust needed some of the more creative marketing approaches. maybe, maybe lower levels of trust, depending on, you know, what those use cases are.
Jay to, to end it with you, like, what's the expert playbook? I mean, again, a lot from our own Accenture's internal experience of rolling this out to 700, 000 employees.
No, it's a step by step journey. Don't expect perfection and maybe never will. This cross functional team, you have to bring in legal compliance, data [00:40:00] scientists, infosec, procurement, HR, to be sort of with you in to look, to know what good looks like. Set up effective governance. Take a risk based approach and really leverage existing processes, technology systems and controls.
the existing systems where they are versus instituting a whole new process to deal with this. Yes. It's kind of extending, you know, I've, I've seen a lot of the, of our large customers say like, we have an MLOps playbook. It was almost like, you know, we were reaching some level of maturity with, predictive and traditional machine learning systems pipelines.
Everything has gone for a toss, but it's not exactly that. Like, you know, it's about extending the existing processes and the structures around us, to adapt to the GenAI workflows, which is a great thought to end with, and this was very enlightening, I'm sure for all of us, thank you all for, for joining today's talk
Conclusion and Final Thoughts
Conor: That's it for this week. Thanks so much for listening and [00:41:00] make sure to subscribe to Chain of Thought wherever you get your podcasts. We'll see you next week.