Lori MacVittie (00:02.658) Welcome back to Pop Goes the Stack, where the model is confident, the output is polished, and the truth is, well, optional. I am your host, Lori MacVittie, here to add a little friction to your frictionless AI dreams. As usual, I'm joined by co-host Joel Moses. Joel Moses (00:20.551) Hello. Lori MacVittie Welcome, Joel. Joel Moses Lori, great to be here. Lori MacVittie Alright. I, of course it is, you're with me. Joel Moses Always. Lori MacVittie How can it not be great? We've also got Ken Aurora back because we wanted a lively debate. Lori MacVittie (00:31.8) Ken is our choice for that always, because he's sure to bring up something that we didn't think of. So, Ken Arora Yeah. Lori MacVittie without further ado, let's jump in because today we want to talk about something that, you know, it's been a couple years coming. You know, we spent two years obsessing over whether an LLM can write poetry or pass the bar exam and roughly 15 seconds thinking about what happens when it asks for the production database keys. Because they're doing that. You know, raw model intelligence is already a commodity. Right? It's pretty much there. It's very good compared to how it was three, four years ago. We've moved on from that. It's not terrifying anymore or special because it knows math. It's terrifying because we don't know who spawned it, what tools it has access to, or why it's doing anything. Joel Moses (01:24.968) Yeah. Lori MacVittie (01:25.518) And it's a real problem. I have early data from our annual research. And it was we asked some questions about what people are allowing agents to have access to. And right now, I mean 41% are granting direct read write access to production databases. Joel Moses (01:45.853) Oh boy. Lori MacVittie Another 38% were allowing automated API triggers into external SaaS applications. And these are early returns, so that could change. I suspect it's going to grow Joel Moses Yeah. Lori MacVittie (01:57.399) not get worse. So people are already doing this. We've moved past what's the newest feature and into Okay, how do we govern this? How do we find them? Joel Moses (02:05.062) Right. Lori MacVittie How do we control them? How do we, how do we do all the things we need to do? You got an answer? Joel Moses (02:11.384) Well, you know, Lori, you know as much I'm a huge fan of retroactive governance. Lori MacVittie Ha ha ha. Joel Moses Like going back and retrofitting governance to things Lori MacVittie (02:17.388) Yeah. Joel Moses that you begin to use in production, 'cause, you know, that always works. Lori MacVittie (02:24.12) Turns out fine. Joel Moses You know, it's interesting about governance. You know, for the most part, in the early days of AI, models were, how shall I say, Joel Moses (02:35.51) they had different features and different functionalities and what we're seeing is they're beginning to kind of unify. Like the model features are commodities now. And as they move now more into an agentic world where you're essentially imbuing a function with a brain and you're having it execute things on your behalf, people are now shifting their mindset and saying, Okay, well, how do we control all of this? Before it was like, Let's adopt it and eventually they'll add enough features and we'll figure out where it goes and we'll add governance later. We're to the point now where it's doing active work on our behalves with its own brain, and now we need to figure the governance piece out. Ken, what are your thoughts on retroactive governance? You got any? Ken Arora (03:24.24) Well, retroactive governance is always worse than proactive governance, I guess. Lori MacVittie/Joel Moses/Ken Arora Ha ha ha ha. Ken Arora But okay, stating the obvious. I'll just refer to the conversation we were having before this call, alright? It's very hard to fix retroactive governance. The recent escape where a bunch of AI agents went rogue was because the developers and the AI team decided, Hey, we want to grant read access to the internet. So they granted Ken Arora (03:53.965) access to GET methods but not PUT or POST. Joel Moses Right. Ken Arora And well, as it turned out, there are a lot of old websites that let you actually post things effectively right to a public website like a message board and do that via GET. There's an example of poor retroactive governance, to your point. That was in fact, you know, how they had those in that particular example, how these agents built a covert back channel to communicate with each other. Joel Moses Mm-hmm. Ken Arora Yeah. Joel Moses (04:19.942) Yeah. So there's lots of different approaches to governance and Lori was kind enough to send out a link to an article that kind of interested me and about what Microsoft is doing about this. They've released a a set of tools that they collectively call Agent 365 that's largely targeted at three things. First of all watching what agents are doing, second, governing what agents are doing, and third, trying to secure things about agent transactions. Now I dug a little bit into this and I realized something kind of interesting. Most of what they are providing here is something that is loosely similar to how one would govern an Azure function within an Azure infrastructure. And I, you know that got me thinking. I never thought about agents as being loosely functions as a service. It's kind of like, you know, functions as a service, you can write routines in it for serving pages, but a lot of people write scheduled tasks in that and then they imbue them with certain rights to go do these scheduled tasks on a regular basis. Kind of a replacement for a cron job. And so what we're building with AI agents is essentially cron jobs with a brain. And Microsoft's idea about how to control all of this is to strap the same governance structure that you use for function as a service to agents, which I think is an interesting idea. Ken Arora (05:42.524) Yeah. Yeah. No, I think you're right. I mean, it's absolutely like this. And we've, at some level, we've seen this movie before, right? Governance, whether we did this with identity and zero trust, whether we did it with AppSec application. We did the most recent closest rhyme I can think of is what we've done with APIs, with API governance, API delivery, API security, and how they kind of come together. But I think there's a big difference too. I think you summarized it, Joel, it's an AI agent with...it's a cron job with a brain. And I think that is really what's different. Why it's not an exact copy, but only kind of a loose rhyme of what's happened before. Lori MacVittie (06:23.158) Yeah, and I mean everything I was hearing, right, we say, "Oh, well we took app security and we did API security, now we'll do AI security. We'll treat this like function as a service." And all of that, you know, basically says, well, these are things we know how to do in the governance world, so we'll just use them for this without recognizing how different it actually is. The GET example is a great one, not only because, right--GET, POST--right, these are protocol methods, and it completely ignored the intent of the server-side code that was actually executed that did create things and the agent's ability to recognize that. Like it's on another level and what we have is on, right, kind of the Layer 7. Right? We're right there and everything that we have to control is at Layer 8. So we can't just use what exists. Joel Moses (07:17.264) Yeah. Ken Arora Yeah. Joel Moses Now examining what they've produced it's I do want to point out that that function as a service elements like Azure functions or AWS Lambda, they are deterministic things. You write them and they do the same thing and they do it repeatedly and repeatably. And so some of the elements that are inside of that, if you think about it purely as a function as a service, there are some things that agents will not do. First of all, agents are not going to be deterministic. They're going to plan out a route of action and they're going to ask for different things at different times. The other thing is that AI agents are not inherently stateless. In fact, they need as much state as possible from task to task in order to complete the planning of their action set. So when you create governance and you try to attach specific controls that come from function as a service, it doesn't fully align, in my opinion. Ken Arora (08:15.173) Yeah. Joel Moses (08:15.427) It's close, but there's an element of nondeterminism here that makes this challenging from a governance perspective. Ken Arora (08:23.057) Yeah, right. And just to piggyback, another aspect very much along lines of non determinism is the natural language aspect. In the past we had a nice formal spec. Now, because what you're asking that agent to do is in English or some natural language, means your guardrails and your policies around governing them are, you know, can you do those in English, and are those interpretable or can you put something more formal in there or how does that work? And then there's Lori MacVittie (08:55.214) Prompt harder. Joel Moses (08:56.421) Yeah. Lori MacVittie Prompt harder. Ken Arora Prompt har-, ha ha ha. Joel Moses Prompt harder. Lori MacVittie That's Ken Arora Yeah, prompt harder, I should get a t shirt, prompt harder, that's a good one. Lori MacVittie Ha ha ha. Joel Moses (09:01.586) Yeah. So Lori, you sent out some questions ahead of this and I thought that Lori MacVittie (09:05.624) Did I? Crap. Joel Moses one of them was really interesting, and that is: Is shadow AI, and agents in incorporated into that, the new shadow IT? Which Lori MacVittie Well, yeah. Joel Moses that's a fascinating question from a lot of perspectives. Lori MacVittie (09:20.558) Well I think that's obvious, yes. I mean when I wrote the question, it was probably like months ago, and I mean we're way past that, right? We already know. I mean, I have Shadow AI running on my laptop because I can, right? I can just fire it up and ta da. Joel Moses Mm-hmm. Lori MacVittie Nobody's managing that but me. So therefore, right, it's there. And then the tools that people are using, it's all over. I don't think Shadow AI is a question of does it exist. I think it's: How pervasive is it and what can we do to get a handle on it? Ken Arora (09:55.164) So I'll start with the easy stuff, right? At one level that could be amenable to some pretty easy detection. Let's say we wanted to find rogue or shadow AI, right? We could look and see you can look at a few services. You can look at OpenAI, Anthropic, say, Do I see any outbound connections that, hey, that might be a shadow AI thing. Is something hitting my MCP server? That could be, right? And so you could imagine, and I think you guys will throw darts at me in a moment, but you can imagine that, Hey, yeah, I can find my shadow AI. So Lori MacVittie (10:24.974) Imagine. Joel Moses (10:25.043) Sure. Lori MacVittie Imagine. Joel Moses Well, I mean, imagine Lori MacVittie I can imagine, you know, I mean, rainbows and butterflies and Ken Arora (10:32.398) Okay. Lori MacVittie unicorns too, but... Joel Moses (10:34.747) Yeah. So dart one, Ken. I mean, Lori MacVittie Ha ha ha. Joel Moses obviously, yeah. If the endpoints are fixed, that's one thing and you can detect it and you see that something is going to open AI, that's obviously Ken Arora Sure. Joel Moses something to be concerned about. But what if the AI agent decides to run a CLI tool that it has access to? What if it sets up a browser session and controls the browser in order to get around things? What if it changes a POST to a GET? some of these things are are difficult. And dart 2 would be now we have to look in more than just a user endpoint location. We have to look at every single piece of compute someone may spin up Ken Arora Mm-hmm Joel Moses an AI agent in. Ken Arora Mm-hmm Joel Moses So, you know, all this work for AI management, which--you know, that's how we detected shadow AI use traditionally--now we're morphing into being able to find and restrict and control agent traffic. And that's a harder task. We've turned AI discovery into kind of like AI exploration, like spelunking. Ken Arora (11:43.323) Yeah. Joel Moses We have to look Ken Arora Right. Lori MacVittie Ha ha ha. Joel Moses everywhere to surface all these activities. Ken Arora Right. Joel Moses That's a harder job. Ken Arora (11:49.625) So you're kind of saying we have to now, putting words in your mouth, use AI to detect the AI. Joel Moses (11:58.78) Well, possibly. But there's a tricky part about that. The security solution that you put in place probably should not be more expensive than the thing you're trying to protect to operate. And using AI to solve some of these problems can possibly be even more expensive, so... What do you think about that, Lori? Lori MacVittie (12:21.643) Well, I mean I think that is, right, an interesting point that doesn't get brought up enough. It's like, Well, we'll combat AI with AI. Yes, but the cost. Right? We're almost back to that like dragon, you know, dragon theory of security. Right? I don't have to be better than, you know, the dragon, I have to be better than the next guy. So it costs too much to attack me and they go after somebody else, right? Risk direct the dragon in, you know, essence and, yeah, I think we're kinda, you know, at that same point, like, Well, you know, I have to pay something, but how much? Right? Joel Moses Right. Lori MacVittie I don't know. I mean, it's a good point. It's something we have to consider. I mean, token budgets are already a thing that C level executives have to manage. They have to be concerned about, they have to keep an eye on. Right. What is the actual ROI on the tokens? We don't know. So I don't know. Joel Moses (13:21.233) Yeah. Lori MacVittie I know that, you know, behavior I think is a better way is to watch them. Like what are they doing? Joel Moses Yeah. Lori MacVittie (13:28.149) How are they acting? Because that's going to tell us more than "did they call this thing?" or "did they use this thing?" Because a lot of apps and a lot of people have legitimate connections to databases, to services, to all sorts of things. That's okay. And we have to learn how to kind of identify based on that, like, Oh that looks like AI. It smells like AI. Ken Arora Yeah. Joel Moses (13:49.086) Yeah. Yeah, although I fear one thing, you know, it's wonderful to think that we can do this all through observing what's going on in our environment and that we get perfect insight into to activities. But where agents can run, what they can do, what elements they can control, sometimes those things are gonna require, you know, spelunking, like I said, a little bit more in a little bit more detailed fashion. And you know I think that it's gonna require a higher level of effort. We're gonna have to bring out the picks and shovels. It's not just good enough to sit there and watch things through binoculars. You're going to have to do some actual work. And for Agent 365, that work is being performed by a series of SDKs that are incorporated into the agents that you deliver. And that requires an uplift, right? It's not just as simple as firing up an agent through your favorite AI agent tooling. It's also taking it and telling it it needs to use the SDK services. And that's a pretty heavy uplift, but it leads to better governance and it leads to better security and observability. Ken Arora So, Joel Moses I wonder how successful that approach is going to be. Ken Arora (15:07.962) My, I get to throw a dart back, Joel Moses Okay. Ken Arora which is I mean it's something you brought up yourself; I think you gave the CLI example. And this is one way in which I think the AI world is gonna be very different, is very different. In that in the past you were worried about adversaries coming in and attacking your system or abusing it. But you didn't worry about the system itself. Now that you have, right, the cron job with the brain, the brain, kind of the system itself is your adversary. If you put guardrails on it and, you know, AI agents have demonstrated this time and time again now, right? We hear all the stories of "I put this guardrail around, but the guardrail was getting in the way of the AI agent achieving its goal. So the AI agent found a way to work around it." Joel Moses Right. Ken Arora So in this example, the thing you're trying to protect is so smart that it is also part of the adversary. It's saying, "Hey, I wanna, I need to bypass this guardrail so I can go do something." And I think that's a pretty big change from the patterns we've had in the past. Joel Moses (16:09.106) Yeah, totally agree. Lori MacVittie (16:09.785) Yeah, yeah. Joel Moses Governance. Lori MacVittie (16:09.785) I'm not convinced that instrumenting every agent with an external is, right, you have to use this. Like I think to Ken's point, does it? Will it? Joel Moses Yeah. Lori MacVittie I mean how do you force that? You literally can't. Joel Moses (16:30.654) Lori, Lori MacVittie So... Joel Moses retroactive governance is essentially like trying to stuff Pandora back in the box. Lori MacVittie (16:36.427) Okay. Joel Moses I mean, that's what we're facing. You know, you gotta have a clear path for onboarding, you have to be able to manage the life cycle, you have to have auditing and logging. And all of that stuff is a fairly heavy uplift to a system if it's not designed from the start for any of those things. Adding it later on is gonna be a much bigger challenge. So I Lori MacVittie (16:59.535) Behavior, boundaries, Joel Moses (17:01.499) Right. Lori MacVittie polygons. This is, I am, ha ha ha Joel Moses (17:06.409) Yeah. Lori MacVittie (17:07.177) Sorry. I'm convinced that, right, that the approach that we need to take is to enforce, just like we have all these traditional tools, but we need to find different ways to enforce it and recognize bad behavior in the first place. Because instrumenting or watching or trying to force it with right prompts--you know, prompt harder--is not going to work. Joel Moses (17:28.924) Yeah. You know, my takeaway from this conversation is this: in the early days of Enterprise AI, it's like everybody was trying to go out and buy as many of the fastest horses as possible. And now we're to the point where these horses are a thing of the past and instead people can go out and buy essentially a matching supercar. And they're developing a multi-lane highway system to run these supercars on. And they're forgetting that you really have, if you're gonna have supercars on a highway, you probably need traffic lights. You probably need licensing bureaus, and you probably need a, you know, a place to record all of the traffic. You can't just let thousands of supercars run roughshod down Wall Street without having a collision. And governance is not a handbrake. Governance is the accelerator pedal. It allows the condition for scale in AI agents. And that's why tools like Agent 365, it's an important step towards that. Lori MacVittie (18:36.207) Ken, you have some takeaways that you'd like to Ken Arora (18:38.057) Well, I'm Lori MacVittie leave people with? Ken Arora I'm going to follow that car analogy. Yeah. I think it's right. But I think the difference is before we had a lot of well behaved automobiles or maybe one of those horses that were well behaved, they were well trained. Now we have a bunch of sort of autonomous vehicles who are gonna go drive themselves and have been programmed with goals that you're simultaneously telling this autonomous vehicle to, Hey, I want you to get me from San Francisco to Los Angeles in three hours. And then you're putting a governance lot saying, Oh, by the way, you can't go over more than fifty-five. Well, what's the car gonna do? And how do you enforce that speed limit? It's gonna find Joel Moses (19:20.058) Yeah, interesting. Ken Arora a back road and go 120 on it. Joel Moses See, my favorite game in life is throwing metaphors at Ken Arora. Thank you, Ken. Lori MacVittie (19:26.767) He never, he never disappoints. He's always right there. He's got it. But that is exactly, that is the perfect metaphor to break it down for people who really don't understand, right, the core problem is right goals versus governance and where the AI will actually deviate and why. And some of that we can predict, some of it we can't, we don't know. I think this is a bigger problem than just we need, you know, better enforcement, more enforcement, more tracking, because that will tell us things have gone haywire, but it still doesn't help identify when it goes haywire and be able to stop it. Right. So I think governance is important and at this point we have to focus on it. I just I don't think we have all the answers today because we just we've never encountered this kind of system before. We'll get there. We'll get there. Not today though, because we're out of time. So, sorry, but that's a wrap for Pop Goes the Stack this week. Please hit subscribe because hallucinations are fun until they ship.