Pondering AI

Chris Marshall contemplates the interplay of AI sovereignty, agentic AI, cybersecurity and quantum computing and why resilience, not AI, is the top enterprise concern.  
 
Kimberly and Chris discuss AI Sovereignty as a going concern in an a multipolar world; what is on the horizon for agentic AI, cybersecurity, and quantum; resiliency as the end and AI as a means; whether full AI autonomy should ever be the goal; if boards and executive decision makers have their head in the sand; how accountability and people are the lynchpin for innovation and sustainable ROI with AI; and if AI can provide guardrails for AI. 
 
Dr. Chris L Marshall is the VP of Worldwide AI Strategies at IDC where he leads Enterprise AI strategy research worldwide, A former partner and executive at companies such as IBM, KPMG, Oracle, FIS, and UBS, Chris’s mission is to translate innovative technologies into industry insights and business value for the digital economy. 
 
Related Resources: 
A transcript of this episode is here. 

Creators and Guests

Host
Kimberly Nevala
Strategic advisor at SAS
Guest
Dr. Chris L. Marshall
Vice President, IDC Asia/Pacific

What is Pondering AI?

How is the use of artificial intelligence (AI) shaping our human experience?

Kimberly Nevala ponders the reality of AI with a diverse group of innovators, advocates and data scientists. Ethics and uncertainty. Automation and art. Work, politics and culture. In real life and online. Contemplate AI’s impact, for better and worse.

All presentations represent the opinions of the presenter and do not represent the position or the opinion of SAS.

KIMBERLY NEVALA: Welcome to Pondering AI, brought to you by SAS. I'm your host, Kimberly Nevala.

In this episode, we're going to be pondering the evolving story of AI, both globally and organizationally, with Dr. Chris Marshall. Chris is a vice president of IDC Asia/Pacific, where he oversees research into data, analytics, AI, and sustainability. Welcome back to the show, Chris.

CHRIS MARSHALL: Thank you very much. Pleasure to be here.

KIMBERLY NEVALA: Yes, absolutely. So it did occur to me, as I was prepping for this, that recent - or I guess it was not so recent, maybe 2024 - degree in analytic philosophy has probably served you well over the last year or two here. There has been certainly no shortage of AI news, complete with often breathless headlines and prognostications, as per usual. So I'm wondering from your seat at the table there, what have you seen as the most consequential narratives that have been developing this year? Maybe we'll just take that at a macro level to start.

CHRIS MARSHALL: Good question. I'm sitting in Singapore right now. And I can't help but not pay attention to the move to sovereignty issues and concern about sovereignty issues. I think that's a very big topic in Europe and certainly in Asia. And actually, strangely enough, in the Americas; I'll add an S to that word, not just America.

And I think that that's important because suddenly countries realize that AI is a competitive enabler. It has national economic implications. It has geopolitical implications. And I think in the world that we're in, which is clearly a multi-polar, fragmented world that we're moving towards, it's not going to be acceptable to have the decisions of a few very large companies in the US or China or wherever basically determine the fate of national economies and national corporation strategies. So I think that's probably the biggest single thing.

Obviously, there's a host of things around the technology itself that have really started to change everything. I mean, the shift from gen AI to agentic is happening thing, definitely. Although, frankly, the talk is still way ahead of the reality in most parts of the world, I suspect, and in most organizations. And that's kind of part of the problem, I think, because we're seeing the gap between the people who are really taking it very seriously indeed and those that aren't; which is most of us, frankly. And that's a problem. And I think it's a problem for a host of reasons: technical reasons, social reasons, economic reasons. And I think that's something that I would worry about.

At the same time as those two big themes, I suppose I'd be worrying about the security aspects. The post-Mythos kind of story. I think that's added a little bit more to the sovereignty question as well because it puts pressure on national sovereignty. And it makes people feel like they're under siege. It's every man against every man. You mentioned the philosophy discussion. Well, Hobbes wrote about exactly this: a state of nature where every man against every man. And I think that's exactly where we seem to be heading in this world at the moment. But that's my high-level $0.02 worth.

KIMBERLY NEVALA: So when it comes to the sovereignty question, how is or is that narrative developing differently? And is the priority being placed on that different as you look geographically across? And is that starting to converge at all? Or is it really starting to diverge now under all these other pressures?

CHRIS MARSHALL: There is a difference, but it's a little insidious. But I mean, I think it's more to do-- it's less to do with actually where you are sitting. I mean, in one sense, it doesn't matter whether you're sitting in KL, London, or Nairobi or New York or London, whatever. It doesn't matter.

What matters is if you're in a very large organization that's got a lot of technology and is making investments in AI versus a much smaller, much more regional, locally based organization. And I think then the sovereignty question becomes actually quite paramount. So it's really a question of maturity than more location.

There's a locational element to it, though. And I think the location element is usually driven by regulation more than anything else. I mean take where I am in South Asia. We have lots of regional restrictions around data privacy, and data security, and where we can move data to, and all the rest of it. And that has huge implications for AI for lots of reasons, whether it's training or inference.

And of course, as we're all - I won't say all, but certainly the larger companies, more mature organizations - are moving to a very hybrid environment where it's essentially everything everywhere. You can have your training, your inference, on the edge, on the cloud, whatever. And there's all sorts of issues about managing that portfolio of AI-related tasks in near real time. Particularly from a resilience perspective. Maybe we should come back to that point a little later.

I think that's much more difficult in some parts of the world than it is and others. In the US, for example, it's really that difficult, relatively speaking. But if you're talking about South America, Europe - even Europe is not so bad in many ways, certainly the EU. UK is slightly more complicated. But the rest of the world, it's very complicated.

And I think that has in many ways stopped that move to hybrid computing, hybrid cloud, if you like. Hybrid AI really, more correctly. And I think that's kind of slowed down the development in some ways of AI-as-a- service kind of attitudes outside of the US. And inevitably, I think that's one of the reasons that the US has done so well.

Now, China is its own beast for many, many reasons. So I'm not going to talk too much about that. And India to some extent is a little bit its own beast as well. But they're the only two economies that have got the scale and sophistication, maybe, to make a little dent in the sort of hegemony that the US currently has in the world and certainly where AI is concerned.

KIMBERLY NEVALA: And I mean, do you see any cracks in that hegemony? Or is this the state that we are in? Is there any indications that that's going to start breaking down? You mentioned that folks are, in some areas, perhaps not developing, or maybe it's not adopting, AI due to some of these constraints or concerns. How do you see that playing out moving forward?

CHRIS MARSHALL: I don't think it's a luxury that people can decide not to take the journey. I think the journey is pretty well the same for everybody. The question is, what stage of the journey are you on? So I don't think it's the case that we can say, well, we're not doing that. We're not going to go towards a hybrid compute world, hybrid cloud, hybrid models, hybrid security. Hybrid everything, really, in one sense. Simply because we're just all too interdependent.

And the only, maybe, argument against that is China. And China, again, where I am sitting here in Singapore, we're very much between the US and China in terms of the technology. And the choice is either or. So I don't think it's a question of deciding not to take the journey. I think it's a question of where on the journey we happen to be and the challenges we inevitably face as a result of being on that journey at that stage.

That's true, I mean, we mentioned the discussion about sovereignty and the concerns about sovereignty. In many ways, it's not a regional concern. It's very much a maturity concern. And I think that's borne out by the data. We've seen a lot of data that says exactly that, a recent survey that we did as part for SAS in fact. And I was a bit surprised by it, quite honestly.

I was a bit surprised that so many people in North America were concerned about sovereignty too. I mean, I'm sure friends across the border might have a slightly different view. But I mean, something like 78% viewed sovereignty as a major concern in North America, which, again, higher than I would have expected. Obviously, APAC and Latin America and Middle East, particularly the Middle East, it was an even higher concern. But still, that's a pretty big number.

So I think it points to the fact that we're living in a much more multipolar world, where technology is concerned. And we can't ignore that. And therefore people's attitudes about where data lives, where models are sourced, which vendors they use become inevitably more multipolar too.

So those are the things that you've got to think about. And that means, I think, the established vendors - like SAS, all the big hyperscalers, lots of companies, not just American ones - they've got to think about flexibility and, again, this hybrid model of technology. And I think that's going to be increasingly important.

Another thing we're seeing is the gap between the sort of closed models and the open-source models is getting smaller all the time. I mean, I don't know what it is now. It's probably a matter of months. The idea that somehow the closed-source models are better. In some sense, they probably are in a pure performance perspective. But the flexibility and security issues arguably that they imply maybe counteract that somewhat.

So I don't think it's quite so obvious that we used to, a year or two ago, we'd talk about the model of the month kind of thing and which model is on the leaderboard. And I think we've kind of gone beyond that a little bit. And again, it's more horses for courses. Perhaps a better model for thinking about this but, again, that leads to other problems. So, anyway.

KIMBERLY NEVALA: So what are some of the problems that the Horses for Courses race we are now running--

CHRIS MARSHALL: Well, security is probably the obvious one. In fact, if I go to the average, say, financial service institution, bank or whatever, and ask what their biggest AI-related problem, they'll probably say security and security management. Again, post Mythos, they feel vulnerable. And correctly, they should feel vulnerable.

And I add to that a few concerns about quantum computing, which are sort of never quite on the horizon but just always slightly beyond it. But the fear is still very real. And the fear is actually what's driving a lot of quantum-related investments right now. It's more about quantum ransom and stuff like that.

So I think that story is making a difference. And it's causing a lot of organizations to be wary of going too far from their comfort zone. So they're going to the vendors. They know the vendors they trust. They're going to sometimes local vendors, sometimes international.

I think it's not realistic entirely to take sovereignty too seriously because where do you stop? I mean, chips and forget it. I mean, it's not going to happen. So sovereignty is a bit of a chimera. It never really realizes. It's always something in the distance, a bit like quantum computing, for that matter. So I think those are the things that are on people's mind, especially in the wake of the challenges that we face from a regional and local perspective.

KIMBERLY NEVALA: And you had mentioned some of the gaps between, I think, the realities and the perception of the agentic AI trend in particular. And I want to come back and talk about agentic AI specifically. But the last time you and I had spoken, you had also kind of posed in passing this question of - and it was really around the perceptions of - how important and how front and center does AI need to be in the background of other geopolitical concerns? And you had actually just said, how big a deal is AI really?

And we didn't have the chance at that time to talk about it. And given everything that has been going on over the past year and the fact that we obviously solidly sit in highly technically capable environments and organizations ourselves: how big a deal is AI? And how much shading or differences in priority do we see there that we may just not be aware of--

CHRIS MARSHALL: It's certainly true.

KIMBERLY NEVALA: --in this technological bubble?

CHRIS MARSHALL: I mean, we are in a little bit of a bubble. And everybody in this business tends to think of everything in terms of that one lens.

But I mean, I don't think you need to be a booster of the technology to realize this is pretty damned important. This is at least as important as the internet, if not more. Again, we can debate this as big as this, as big as that. But it's clearly a generational change. I mean, we can define it any way we like.

We can just look at - we were chatting before the call - the average consulting firm now has got, I don't know, maybe three or four people, maybe five, per partner. Now, when I was a partner 400 years ago, it used to be that you'd have about 30 people, maybe 40 people, per partner. Now, that tells you something about how a very knowledge-intensive organization, like a consulting firm, has to change in the wake of AI.

I mean, it used to be the fresh MBAs going to a consulting firm would spend half their time, if not more, doing either on-the-job project work or PowerPoints and development of BD kind of efforts. Now the average partner can write a good PowerPoint presentation to give to an executive presentation in a matter of 20 minutes, something like that. Claude helps enormously with these things. So suddenly the need to have all this backup stuff is just not there.

And of course, what does that mean? Well, that means that suddenly there isn't the pool of resources that are needed to develop the skills to become directors or become partners or whatever in that organization. So suddenly that does change the very nature of what the organization means. It becomes top heavy. I think a lot of organizations see that their structures are becoming - not sure what the right phrase is - sort of diamond shaped rather than sort of triangular. And that is something that's not, in every knowledge area we're seeing exactly that, and that's not a little thing. That's a very big thing because that's got economic, social ramifications that cut across the board.

So I think that's very real. And I think boards' executives are still coming to terms with that. I don't think they've quite cottoned on to this because that's not the world they were trained for. That's not the world they grew up in. And I think a lot of boards, frankly, have got their heads in the sand on this. I don't think they quite know what the implications of it are. They don't know how they're going to react and change and respond.

I think there's a bit of a mindless - frankly, it is a bit mindless - response, which is just to get rid of headcount. And one of the uncanny things is that those organizations that we see, who just really prioritize headcount reduction or automation, the ones that usually consistently do worse in terms of their overall success with AI. And that's important because, it means that what we should be doing, instead of saying, well, I could save this headcount 20% in my IT department because of Codex and all the various tools that we have to do coding quicker, for example. They're missing the boat.

And much better are people who think about, how do we use AI for actually transforming the nature of the business itself? Because they're asking the right questions. And eventually the people who just take a cost-cutting story will be beaten by the people who take the transformational story. And we're seeing that in the numbers to some extent. The returns on AI projects are really coming from transformational projects rather than from cost-cutting automation projects.

And to some extent, that maps quite well, not perfectly, but somewhat, to the GenAI/agentic AI story. It tends to be that, and again, I'm painting with a broad brush here, GenAI tends to be a bit of a productivity focus. And agentic tends to be, by its nature, more enterprise oriented across process, et cetera. And I think that the trick is going to be to marry the best of both.

In fact, we were talking about hybrid. I think hybrid everything is really one of the themes in many ways of where we're going. It's not simply AI for every problem. It's not simply agentic for every problem. It's nothing like that at all. It's really about figuring out when do we need deterministic RPA, for example. And well, for example, maybe that's when you've got guardrails, when you've got rules that cannot be broken. When does it make sense to have GenAI? Well, probably in those situations where knowledge management is key. There's a relatively small number of users interacting with the system. And they're doing something that's high value, blah, blah. Agentic, on the other hand, is usually when it's more about an enterprise process and essentially a transformational one.

So I think those sort of things, I think managers are still - well, executives, really - are still trying to figure out. And I think they in many ways aren't ready for that yet. And obviously the ones that are leading are. And the ones that aren't, aren't.

KIMBERLY NEVALA: So when it comes to things and minding the gaps, either because they're getting there now and starting to adopt this agentic AI - these kinds of systems - or trying to look to do that. Or they're projecting into the future. What are some of the other, I'll say, factors or frontiers that boards need to be looking at?

CHRIS MARSHALL: I think if you went to the average bank. And again, sorry, I'm a bit of an ex-banker, so I tend to look at the world from a banking/financial services perspective. But in many ways, financial services is the vanguard of a lot of the changes anyway in corporate world. So what JPMorgan does is something that a smaller company would do eventually.

So if you ask the average bank, they will say, the most important thing for me right now is probably resilience. They won't say it's AI because AI is a means, not an end. It's a thing that you spend money on to do something. Resilience, on the other hand, is an end in itself. Because if my ATM network goes down, I'm in a bit of trouble. If my payment platforms fail, I'm in a lot of trouble. And it might not just be customer trouble. It might be regulatory trouble. It might be all sorts of problems that come on the back of that. So resilience is a hot topic. And I think that is in many ways the underlying theme.

And again, it goes back to some of the sovereignty and geopolitics and all the rest of it that we talked about. The resilience story is almost tailor made for agentic AI. And that's why I think it's a match, a problem, and a solution coming closer together. I mean, banks, financial services, insurance, have always spent a lot of time thinking about risk management. But now, with resilience, they're thinking a little bit more broadly. They're thinking about not so much understanding the nature of the exposure but also saying, well, OK, imagine this bad thing is going to happen anyway. How do we respond?

So it's more sort of response management and the fact that if it happens, if you are resilient organization, you just bounce straight through it. It's not a big deal. So it's more your response to the exposure rather than the exposure itself. And I think, also, you've got regulatory burdens in several parts of the world, DORA, in Europe, for example, that are pushing to do banks, financial institutions, to really do exactly this, to make sure they can respond and deal with the sort of threats and challenges that they face.

Of course, one of the biggest threats that everybody faces right now is cybersecurity. So cybersecurity is sort of, again, the coalface of AI and agentic AI in many ways; because that's where they're seeing the biggest immediate threats. And of course, if you actually were to look at the number of - I'm not sure the current numbers, but I suspect it's of the order of 30% - of the current threats are nationally directed. Which has all sorts of interesting implications.

So resilience becomes a corporate and a national issue. And I think technology, agentic especially, is almost the only way to think about resilience. It has to obviously be balanced with the human component. And this is perhaps one of the biggest weaknesses that AI has at the moment is accountability, I suppose you'd call it. We don't know when an operational system cuts in to deal with a particular problem whether it's doing it for the right reasons or whether it's making a mistake, who's responsible for it, who's accountable for it.

So the agentic AI story is still a little bit problematic for exactly that reason. It lacks that transparency. It lacks that explainability. It lacks that accountability, really. I mean, we've talked about human in the loop. OK, which human? Which loop? What point in the loop? These are the questions you've got to be asking yourself. And these are not ones that we can give a carte blanche answer to. It's a big, complicated question that we've got to think through carefully.

Again, it goes back to the earlier comment we made about naive approaches to automation. I mean, one thing I'm not a big fan of - I know I probably am upsetting a few clients when I say this - I'm not fond of phrases like the autonomous organization. I mean, autonomy is a mistake usually. Because if it's really autonomous, there's no humans in the loop at all. And how can you have accountability if there's no humans in the loop anywhere? I mean, are we going to a world where you've got a CEO whose job it is to make gazillions of dollars, but the moment something goes wrong, he's fired? Is that the world that we want? Is that a good world? Is that a good world from an economic perspective? I don't think so.

I think accountability is at the heart of how organizations have to work in a changing and dynamic world. And I think that's one of the things that we're still struggling with, I think in the AI world, is figuring out which bits still need people to be accountable for them. And obviously agentic, in theory, could do an awful lot of things on its own. But do we want it to? And that's a bigger and a more deeper question.

KIMBERLY NEVALA: And you and I have spoken before, too, about this issue. Of even if we said we wanted to have most of them, humans, are on the loop, not in the loop or part of the loop. And we want them-- Are we actually building the capacity and the muscle so folks know how to be on the loop? So in this diamond-shaped organization, we've got a lot of folks in the middle who've developed experience over time. But if that funnel is going away, how does that work?

But also, are we developing folks with enough experience, enough intuition to be able to be the instigators of the next opportunity? Or are we all going to just operate as it is today but maybe better or faster? I'm not sure.

CHRIS MARSHALL: Whenever people ask this sort of question, I always think of the Matrix again. And it's not an accident. The old guy's sitting in the cave, basically, with the tubes. And they're being used to basically run the world.

I can't help but think of that whenever I think of this because it does suggest-- I mean, it's a metaphor, obviously. But it does suggest that limitation that the technology, clearly, we're becoming dependent on the technology. And the technology is, by its very nature, an aggregation of all the experiences we've had in the past. That's just the nature of how AI works. It really is little more than a glorified FAQ system, an aggregate FAQ system.

I'm going to be shot for saying that, but I think it's true. And I think that means that it's not able to have that sort of flexibility and creativity to think out of the box, to manage difficult, conflicting problems with quite complex trade-offs. Things where people are really going to get hurt. And I think we're already seeing the reluctance, particularly in health care, for example, to take the models quite so seriously as they would, say, in financial services, for example. And I think that tells us a lot about our future.

KIMBERLY NEVALA: And in some ways, when we're really leaning into pushing these decisions down into systems - where very consequential decisions get made that have really big implications in terms of risk and liability - are now getting, in some cases, almost hidden. Or just sort of sinking lower in the organization where they're no longer visible to some extent.

So they're just kind of-- it's not that decisions aren't being made, but they're being made where? How aware or how critical is this view? You talked about the agentic, or the analytic system loop, for boards to be really cognizant and cautious. Not just about is it on the loop or in the loop, but which loops are we actually using these for.

CHRIS MARSHALL: Again, sorry to go back to financial services. I don't mean to do this, but it's true. I mean, I used to work at a large Swiss bank that was rumbled as, other banks for that matter, by the individual actions of particular traders, for example. And they're quite junior people. I mean, they're in their 20s, 30s, whatever. They used to be. And that world, where you've got a few people with the ability and the leverage and the capital and the technology and resources to make consequential decisions for the organization as a whole, that's only going to get more.

So if, first of all, the total number of people in the organization decreases, which it probably will, and the sort of locus of power, if you like, gets closer to-- it leaves the senior management. Because the senior management don't really know what the hell is going on in many cases. And again, that's partly an argument for transparency, particularly of agentic AI. Because without that transparency, what the devil is going on? So the temptation to just have agents do their stuff maybe controlled by local managers or local operators is kind of silly. Because unless you've got the transparency at the very top about what the agents are doing, you're completely beholden to them. You're in a very vulnerable position.

And frankly, I don't think the senior executive team quite appreciate this yet. We're just seeing a little hint of this with things like token economics. With token economics, people are starting to realize there's a side effect of all this use of, particularly, GenAI. Probably that they're finding it's much more expensive than they thought. CFOs are asking, well, are we really making any money out of this GenAI project? if it's suddenly found that it cost $300,000 last month to do something that we did in for $30,000 a month before.

So I think the token economics story is a hint of things to come and the realization that agents have knock-on effects beyond their narrow operational area. And that means if you're going to deal with that problem, first of all, you've got to have visibility, which, frankly, is mixed realistically at the moment. And you've also got to understand why it did what it did. Which is sort of an explainability rather than just transparency question. Which is also extremely weak, frankly, for the current AI systems that we have.

I mean, explainability is probably one of the single biggest weaknesses of the technology inherently at the moment. A lot of people say, the lack of explainability is the reason - certainly in health care, for example, even in manufacturing - the lack of explainability is one of the big reasons we're not deploying this for whatever reason. And the fact that management doesn't trust it is usually because it's not explainable.

They can't go to the regulator, their board, or their shareholders and say, well, why did you do such a stupid thing? I don't know, because that's what the agent did. And again, in a relatively porous environment where you've got cybersecurity threats coming out of the washroom, it's difficult. It's very difficult to guarantee that things are being done for the right reasons.

And again, unless you've got that transparency, that governance, which is-- I think we talked about this last time. In our view, the governance aspects of AI are probably the single biggest barrier to actually getting genuine-- I mean, you can get short-term value out of AI without that governance, quite honestly. But they're short term, and they're local. If you're starting to talk about enterprise value from AI, particularly when you move to agentic, especially when you move to agentic, you go beyond GenAI, you need to have that governance layer.

And it needs to be an active governance layer. It needs to be, it needs to be about accountability. It needs to be about explainability. It needs to be about auditability and all these different things that make up a true governance capability. It's not just a policy document, which is kind of trivial. So that's one of the things that we're seeing still getting in the way of deployment of AI at scale. And again, another financial service example; if you've got a risk, eventually it's going to be realized. And ignoring that risk for long enough is ultimately completely doomed because if it's a risk, eventually it will happen.

KIMBERLY NEVALA: It is interesting. I think there's a tendency for us to talk a little bit - and you had referenced the study that SAS had recently commissioned from IDC - and it was really looking at this calibration of trust and trustworthiness between the systems and how are those. And there's a tendency for us to say, well, we just need to - people just need to, we need to have our folks or our employees or our customers, trust the AI system.

And that seems to, sometimes, I think it's nice, kind of trips off the tongue. But it overlooks that component of the systems really need to be trustworthy. It's not just a question of telling somebody to trust the system. And one of the interesting things, and you alluded to this actually in your comments there, was that one of the takeaways from the study this year was that perceptions of trust in AI had actually gone down a little bit while investments in maybe trustworthy AI -so we'll use that broadly, and we'll come back to what those components are -had gone up.

And there might be a propensity to say, well, that seems illogical. But in some ways I look at that and think that that may actually be a sign, in fact, of maturity. Because knowing when you can and cannot trust it is the first step in knowing when, where, and how to deploy it.

CHRIS MARSHALL: Lately, I agree with that. I mean, it's a race between two things. In the early days, the trust, the expectations of AI - and again, let's talk GenAI for the sake of argument - very high. The trustworthiness really was way down here. As experience lets us understand a little bit better what GenAI for example can and cannot do, inevitably the expectations come down a little bit. And at the same time, we are making the investments in data quality, data management, governance, explainability, transparency, et cetera. So the trustworthiness aspect is increasing. So in some sense, the gap is narrowing.

The problem here is that first of all, the number of deployments we're doing is increasing dramatically. So it's not simply that I've got one shot of making a mistake. I've now got 50 shots of making a mistake. So the consequence of that gap is actually getting bigger. Even though it's a small gap, and it's decreased a little bit, as all technologies tend to do, by the way. Every technology starts off-- in the old days, you didn't need, there was no speed limit for driving a car, for example, in the US or the UK. And you could drive as fast as you liked. And people thought this is cool; we don't need any speed limits. Let's carry on. Then, of course, people started getting killed. And suddenly the reaction to that was to have speed limits, 70 miles an hour or whatever. And the same thing is happening with AI. I mean, EU AI Act is perhaps a more extreme version. But there's other similar requirements across the world that do somewhat similar things. And they're the equivalent of the speed limit.

And what's interesting, I think, is that although the gap is getting smaller - and again, let's speak about GenAI for a moment, I'll go to agentic in a second. The GenAI story is getting more-- commoditized is too strong, but certainly a more mature environment for operating for GenAI. Success rates are a lot higher for GenAI than they were two years ago. Agentic is different.

Agentic is quite different because the expectations, in fact, I think it's borne out by the data that - I haven't got the number in front of me - the trustworthiness of agentic, the trustworthiness of the technology is actually even lower than it was for GenAI. The expectations associated with agentic are much higher. So again, the gap is bigger. The number of deployments is fewer. So the actual impact of that gap is relatively small in a world where the average company is, I don't know, a dozen agents, for example, maybe less.

So what is happening is that our expectations are being transferred to agentic AI as a technology. We're making some of the same investments to support agentic AI's trustworthiness. But because of their scale and their magnitude, potential magnitude at least, any mistakes get magnified and become a much bigger deal than they are for GenAI, where the average GenAI project is used by so many people at a time. So a mistake is just not that significant usually. Slow somebody down a little bit, but that's not a big deal. With agentic, it becomes an enterprise issue. So even though the gap is relatively smaller, the risk actually is bigger.

KIMBERLY NEVALA: And I think even the-- And again, we have to, I think, acknowledge that we're dealing with this as a single point in a study. But I think the perceived trust that people reported in agentic AI was, I don't know, on the order of 67% or something. And GenAI was 77%. And we could probably argue, depending on the explicit context and application that even - you know - a) I don't think that the comparison between GenAI and agentic AI is, it's apples to oranges in some sense. Or in a big sense, I think, even though some of the underlying componentry in agentic is certainly based on generative, but--

CHRIS MARSHALL: I don't think we'll be able to answer that question for another year because in a year's time, as you say, it'll be sort of a hybrid mix of technologies that I don't know if it's GenAI. Is this agentic? Maybe a little bit. Is this entirely deterministic? No, there's a bit of all three in each of these agents, for example. So I think that question is probably a question we won't be able to ask anymore because it's just too hard to tell.

KIMBERLY NEVALA: And far from looking at any of those numbers, even with all of those caveats, I looked at those numbers and they gave me some level of optimism. Because, again, the question isn't to have-- that everyone just needs to trust it uncritically and adopt it uncritically. The question, as you said, is to actually make sure that you've got the right balance between your trust and how trustworthy it is. And so a little more healthy skepticism, knowing what we know about the systems and what the limitations are--

CHRIS MARSHALL: The human element. There's the human element that sort of surrounds the technology, the overrides, for example. So again, financial service example, human overrides are not bad things. Sometimes it makes sense. You get a loan application. There's something funny about it. And therefore, human overrides it because it doesn't fit with the standard model, blah, blah, blah.

Overrides are not necessarily bad. They're expensive, no question about it. But there's a reason that you have human overrides there. So one thing that we did ask was how many times respondents overrode AI in its various forms. And one thing we tend to find is that a very small percentage never override, very small, less than a percent. On the other hand, something like 80% override AI in more than 10% of the cases.

Now I think, and we can debate this, that high level of overrides is actually a good thing because it means that people are thinking. And they're understanding, well, this model doesn't quite work in this situation. I'm not quite getting it right here. Or maybe the inputs aren't quite right or incomplete or whatever. So a high override rate is a bit of a measure of trust in my view. Because what it says is that the human failsafe layer, if you want to call it that, is there. And it's acting, and it's operating.

And that's where resilience starts to come in my view because it's that flexibility of mind that allows you to respond to the mindless automation. I can't remember. Sorry, I'm going into popular culture here. I always think of The Simpsons when Homer is playing on the nuclear power station, and something goes wrong, alarms in the background. And he just ignores it and drinks his tea or whatever. And it's a bit like that.

So I think it's that organizational muscle, if you like, to respond intelligently and creatively. That's got to be there. And unless you build that in into the process and don't think of it as sort of an afterthought and hope it will happen, which is sometimes the case, agentic is never going to be as successful as it should be. And I think that's got to be aligned very much with the level of trust we have in the technology and its ultimate trustworthiness.

So I think the gap-- in short, the gap is decreasing between the two. There's still a gap. The implications of that gap are getting only more important. And obviously, there's differences by country and region and what have you.

But thinking about trustworthiness, a bit like an employee. A new employee comes. You don't have entirely a great deal of trust in the person. So you give them relatively easy tasks. Slowly, they develop skills and capabilities. You start to trust them more because you have evidence that they are trustworthy.

You continue doing that. And of course, you give them bigger tasks. And eventually you start to believe that they are genuinely trustworthy if they've done the right things consistently. And they're worthy of additional trust. And maybe you promote them. And you do all sorts of good things.

So I think the same attitude should be there for agents. That's not how software engineers tend to think, though. They tend to think about a solution for all time in one place. And yet that's not what AI does. AI is inherently dynamic and changing. And I don't think we quite know how to incorporate that dynamic nature of AI into how we use the technology. And that's a struggle, I think, that we're still going through.

KIMBERLY NEVALA: Well, and you wonder if it goes back to, you had mentioned this vision, perhaps ill conceived, at least in my opinion, of this fully autonomous enterprise. And so there's this expectation that for all of these things to be successful, need to be 100% coverage. They need to do 100% of something or cover 100% of a process or 100% of the cases.

And we've been in the governance game for a very long time. And one of the things that we used to tell people is, it's not a question of, do things get escalated or not? If nothing's getting escalated, you probably have a problem you don't know about. If everything's getting escalated, we have a problem that we don't understand why and where people can't make decisions. So we kind of know what-- we probably know what the problem is. So it's, are the right things getting escalated for the right reasons? And that's what we're looking at.

CHRIS MARSHALL: But that's more of an art than it is a science. It's still very much, I mean, that's an organizational design question rather than a technology question. And again, in the loop, on the loop, by the loop, these all kind of ways of avoiding, in one sense, avoiding the real question. Of, OK, you say that on the loop, what does that really mean? I mean, do they look at the outputs once a day? Or do they get an alert if this happens or that happens? What does that mean exactly? And I think that conversation organizations are still struggling with frankly. But the right way to think of it is very much in terms of how you would deal with a human being.

KIMBERLY NEVALA: And we talked about this last time. We continue to talk about this. And the question I think I'd asked you last time was, are we finally seeing that a-ha moment for organizations when it comes to the investments they need to make - and what you called all the boring things - in governance, in sustainability, so that they can actually get value out of these. They put the infrastructure, both the human and the technological and the data infrastructure in place, to do this.

The research continues to tell us that the organizations that are farther ahead, that are leaders in delivering value with these technologies, do make sort of stepwise changes. Or, likewise, those that are adopting are also making these likewise investments in governance, in management, et cetera, et cetera. It still seems to me, though, that when we look at that in our organizations--

CHRIS MARSHALL: I think it's still problematic.

KIMBERLY NEVALA: --it's still not there.

CHRIS MARSHALL: These technologies, there seems to be multiple stages. They tend to start as infrastructure investments of some sort. I mean, we saw, even now, I suspect the bulk of AI spend is probably on chips and infrastructure if you actually just look at the pure dollar numbers. And that's actually gone up over the last year. So it may actually be even increased a little bit simply because of cost increases.

But then the shift goes from infrastructure to technology or software or applications. And that's probably a good thing. And I think we're in that direction. We're certainly in that ballpark. But then the question becomes, how do we organizationally make use of the technology? And that's an investment ultimately in processes and people and skills and all sorts of stuff like that. So it cannot end with the software.

And we're still not thinking that way. We're still seeing CEOs of whichever company brag, which I think is quite shameless, that they have cut x percent of their headcount as a result of using this tool or that tool. And I think that's completely the wrong perspective to this. They should be bragging about the improved performance or improved service level or quality of service that they're delivering or the fact that they're doing things faster, quicker. But not the fact that they're doing things with less. In itself, that's not really an interesting thing to talk about or useful thing to talk about.

I think that really gets to the people aspect of the problem, which I think is still where the heart of this still lives. And that's why the trust element is so important is because that people aspect needs to trust the technology before they can effectively deploy it. So I mean, I still think that for the most part, companies view this not as a people problem, which in my view is a mistake. But we're seeing change. And it takes time to happen. But who knows?

And I think the only way we'll actually see real things happen on this is if there's some major problem, major disaster. And again, it goes back to the resilience question. If there's a problem hidden in the woodwork, if you like - maybe I should say the metalwork - that means that it's going to come out eventually. And when it does, it's going to be painful. And there'll be a huge reaction to it. And I think you saw that with the Mythos thing. I think that's an example of this.

I think the emphasis, again, in financial services on cyber security, cyber resilience, again, is pointing to exactly this. It would take very little for a lot of these systems to be unhinged. And I think that's not a good thing.

KIMBERLY NEVALA: We could spend a whole separate hour, and maybe we should one of these days, about that particular occurrence and what went on there. Everything around Mythos, around the narratives that have spun up since, then extended and related to that. About-- I sort of lump these in the same category to some extent about agents going rogue. It's fascinating to me--

CHRIS MARSHALL: I mean, there's a wonderful--

KIMBERLY NEVALA: --in some sense because I'm like, if these weren't these what we're calling frontier foundational systems. And they've got such this aura around them, I'm like, in any other circumstance, a company putting some of these things out or bragging about that, we would be denigrating them for their very poor, poor engineering and cybersecurity practices, their software engineering practices, their operational practices, and/or arresting them for hacking. For developing that and being proud of it. So there's so many things that are mixed up into that component.

But at the end of the day, as you said, there's also just the practical reality that--

CHRIS MARSHALL: I think so.

KIMBERLY NEVALA: --these systems are a lot more vulnerable.

CHRIS MARSHALL: To some extent, I think we're looking, again, AI as a hammer. And everything is solved with AI. The reality is, for some of these things, you need hard and fast deterministic rules about what works and what is not allowed to work.

I mean, most of the time - I can't think of any exceptions - it doesn't make sense for guardrails to be AI driven. It makes sense for them to be deterministic. If they're a guardrail, if they're a rule that has to be enforced. Whatever the rule happens to be, it should be a deterministic rule. Theres no ifs, ands, or buts about it. If it's important enough, you make sure it's deterministic. There's no question about it. So I think that-- because one of the problems is once you start making it AI driven, the guardrails themselves, the rules that prevent certain actions, then AI can game it.

In fact, we're already seeing the sort of Mythos story posted. Even people talk about agents communicating with each other to get around the system, to play games with the system, to fool the system, to learn from past mistakes, past agents' actions and see what worked and get around them. And it gets a nightmarish situation, in my view. And again, part of the reason for that, I think, is the fact that these are non-deterministic systems that are able to fix other non-deterministic systems.

So I think that we've got to realize these are increasingly very smart actors. And we have to treat them not with kid gloves. We can't treat them with, well, sometimes you're allowed to be smart. And sometimes you're not. No, these are the rules of the game. And they have to be forced - enforced, I should say.

KIMBERLY NEVALA: Now, I looked down, and I realized we're closing in on an hour. And I've been jotting notes as we go. And if I asked all the questions, we would be here for 10, as is always the case.

But perhaps because I'm just always nosy, I think last time we closed out with letting folks know that you had reintroduced yourself to the stage. And today I learned that you are also an artist who paints. So want to give you the opportunities for any final words or observations for the audience. And any ways in which those artistic endeavors provide any insight into how we might think about moving forward.

CHRIS MARSHALL: It's a silly example but thank you. I appreciate the comment. They're all available for $300,000 a piece, so just call me. No, seriously. I think it's so important that as a human being, all of us, we need to develop those skills. And I have this conversation with my children on a regular basis. I'm sure many of you do.

The way to remain viable in this sort of environment is to figure out those things that are uniquely human and those things that are uniquely you. What's important to you? What are you personally passionate about and enjoy and excited about? And those are the very things that AI will never be able to touch.

And I think that emotional engagement, I mean, emotions in one sense are the secret weapon that human beings have, technology will never have. And you've got to leverage that because it's those emotional elements that really drive, I think, everything in one's career and one's personal life and the rest.

Because it's that human humanity that drives what's going to work and what's not going to work and what adds value beyond mere intelligence, which is increasingly a bit of a commodity. It's more about emotion and wisdom about how that intelligence is used that you want to develop. And art and writing and all sorts of good things, talking to people, these are all good ways to do exactly that.

KIMBERLY NEVALA: Excellent. Well, wise words and a very nice note to end on. So I will, as always, restrain myself from continuing on. And thank you so much for your time, insights, and--

CHRIS MARSHALL: Thank you so much.

KIMBERLY NEVALA: --patience in--

CHRIS MARSHALL: It was a lovely chat.

KIMBERLY NEVALA: --exploring all of these avenues with me.

So to continue learning from thinkers, doers, and advocates, such as Chris, you can find us wherever you listen to podcasts and also on YouTube.

SPEAKER: This has been a SAS podcast.