Subscribe
Share
Share
Embed
AI is no longer a lab tool; it’s showing up in pipelines, production systems, and the places where “seemed like a good idea” becomes a 2 a.m. incident. In this episode of Pop Goes the Stack, Lori MacVittie and Joel Moses are joined by John Willis, known for his work on DevOps and Deming, to separate what’s genuinely new about AI from what looks like the same organizational patterns repeating under a new label.
John frames the shift in two parts. First, the human side: every major technology transition triggers the same dynamics, and there’s a century of first principles from Deming and others that still apply. Second, the operational side: AI introduces a different kind of authority into the delivery loop. DevOps optimized for speed with reasonably deterministic pipelines. AI pushes systems into probabilistic behavior, where correctness is no longer guaranteed 100% of the time and audits can’t pretend “this will never happen.”
The conversation gets practical about what that means for enterprise teams adopting agents. The real questions aren’t whether tools use MCP or a CLI, but what authority an agent has: read-only, write/mutate, or execute. From there, you need boundaries, containment, escalation policies, kill switches, stronger logging, replayability, and the ability to justify decisions after the fact.
The main takeaway is permission to slow down. Step back, define what risk you’re willing to accept at each stage, and build guardrails that match that risk. AI isn’t going away, but “move fast” without a risk model is just handing operational authority to a very smart script and hoping it behaves.
Creators and Guests
Host
Host
Lori MacVittie
Distinguished Engineer and Chief Evangelist at F5, Lori has more than 25 years of industry experience spanning application development, IT architecture, and network and systems' operation. She co-authored the CADD profile for ANSI NCITS 320-1998 and is a prolific author with books spanning security, cloud, and enterprise architecture.
Guest
What is Pop Goes the Stack?
Explore the evolving world of application delivery and security. Each episode will dive into technologies shaping the future of operations, analyze emerging trends, and discuss the impacts of innovations on the tech stack.
Lori MacVittie (00:05.04)
Welcome back to Pop Goes the Stack, the podcast that treats tech trends like change requests. Bold idea, unclear impact, definitely someone else's problem. I'm Lori MacVittie. Let's take a look at what's going on today.
Today, Joel and I are joined by John Willis of DevOps and Deming fame. Hi, John.
John Willis (00:26.15)
Hey, great to be here. Yep, it's been a long time, Lori.
Lori MacVittie (00:29.159)
It has been a long time, so I'm glad that you could come. When I saw Deming in the subject line, I'm like, that is John, and we have to have him on for a discussion, right?
John Willis (00:37.902)
That's awesome, yeah.
Lori MacVittie (00:41.07)
So of late, you're the author of A.I. CIO and Rebels of Reason, and you bring both a historian's perspective and real world experience to A.I., which is helpful because right now most people are just yelling about it, making it up.
Joel Moses (00:58.397)
Yeah, no kidding.
Lori MacVittie (00:59.21)
AI isn't living in the labs anymore. We know that. It's landing smack in the middle of everything, including DevOps. So pipelines, automation, production systems; it's all in the places where seemed like a good idea at the time turns into a 2 a.m. incident. So, John has a knack for separating what's actually new from what's just history repeating itself. Right, patterns.
And that matters because as AI shifts from tool to participant, a lot of what we think we know about delivery and operations starts to become murky. So today we're going to dig into what happens when DevOps meets autonomy. What works? What doesn't? Why your pipeline might have opinions. So let's kick it off.
Joel Moses (01:47.218)
Oh, yeah.
Lori MacVittie
Joel, I know you had a great question to kick us off. Why don't you?
Joel Moses (01:49.01)
Well, I mean, look, we've got John Willis here who went
Lori MacVittie (01:52.696)
I know.
Joel Moses
from early cloud evangelist to early containerization advocate to DevOps pioneer to now AI historian, which is like the tech equivalent of going from discovering fire to creating nuclear reactors. So, so you've seen and done a lot of things over the years, John. And I want to start with DevOps in particular.
Joel Moses (02:11.657)
We're tracking the rise of something a lot of people are calling AIOps, but it looks to me like an enhanced tool set for DevOps. What is different about DevOps now that AI has risen and is newly capable to perform autonomous actions?
John Willis (02:28.109)
Yeah, I think, you know, there's two ways to look at this. One is that there are patterns, right? And, you know, I was mentioning earlier, there was a, we're at DevOps days in Atlanta and some of the young people were talking about how new AI is and how confusing it is and how everything's terrible. And then they pointed to like you DevOps folks, you know, you figured it out. And I was like, no, not 15 years ago we didn't.
You know, so one way to look at it is there are patterns. You know you mentioned Deming, Lori, right? You know, there are things that are first principles of how we do things. How we do, you know, how we apply critical thinking. And one of the problems with any of these new technology shifts, and AI being an interesting and very complex shift, it is, you know, that in the end of the day, these are repeating patterns of things that we know we have a hundred years of people like Deming and Ackoff and Senge and just a ton of resources about organizational dynamics.
So that's one way to look at it, right? Is that like, if we step back and say, how do we apply first principles to the technology? Cause it's not about technology, it's about the humans, right? And all those things they're, you know, the thing I found early in my career, which was, we thought we were inventing DevOps, but we were really is reinventing what Toyota did, which was reinventing what Deming did when he went over to Japan, right?
And so that's one, is like understanding first principles, critical thinking, and the organizational dynamics around how humans deal with any technology. Then there is this operational authority shift where we've moved from sort of determinism--and the world was never really truly deterministic, but let's just for sake of argument,
Lori MacVittie (04:20.102)
Let's pretend.
John Willis
that the way we delivered software in the DevOps age is that we delivered software very fast. The DevOps was a catch-up mechanism, but it was reasonably deterministic in terms of how we created the pipeline, right. The STLC, right. We did our sort of testing, we did our SAST, our DAST, all the things we did, and then we learned how to do audit properly. So we called it DevOps Automated Governance.
John Willis (04:49.886)
Now we live in a world of probabilistic non-determinism. And so this is the first principle is that the question now when, you know, if an organization wants to use or an organization is bounded by regulatory controls, right, whether it's inference or agent based, you can't just put your head in the sand and say yes or no. Cause we saw that in cloud, right? Cloud, no cloud. No cloud. Wednesday at 4 p.m., yes, cloud. Right?
You know, like we can't do that with AI and agents, right? Like, you know, we have to have a pragmatic approach. And then the question then is, are you willing to accept some type of risk?
Joel Moses (05:33.545)
Mm-hmm.
John Willis
Because we could live in that deterministic world. And again, it was sort of a falsehood anyway, but like we could say, this can never ever, ever happen.
John Willis (05:43.625)
And so what we would do is we'd say, here's all the things that to make sure this never ever, ever happens. We'd have regulatory control. We'd have findings. But now, it will happen. Right?
Joel Moses (06:00.185)
Mm-hmm.
John Willis
A sort of an answer, a regulatory, an answer that is regulated, just plain old inference. I'm not even talking about agents. There's going to be some probabilistics where the answer may not, may be correct 99 out of a hundred times.
John Willis (06:013.025)
But it's going, there's going to be some percentage of risk. So the reality check then is, is an organization if you're going to go into AI and you're going to have a chat bot that is going to give regulated answers, then are you being realistic about the actual risk profile that you're willing to accept?
Joel Moses (06:33.811)
Mm-hmm.
John Willis
And so this, this changes everything about how we think about audit.
John Willis (06:40.046)
Cause audit isn't like, it can never do this, you showed this evidence and we accept that you did this. This is, this will probably happen. Are you willing to accept it less than 5%? Right. And then when you get into agents, it's a whole new layer because inference by itself is dangerous, right? The Air Canada example, right. That, you know, where it gave the wrong answer to somebody who like wanted to get a refund for his mom's funeral. There was a big lawsuit front page of the wall street journal.
In the end, Air Canada tried to say it was AI. And well, sorry folks, it's never gonna be AI. It's gonna be some corporate or legal entity. And then agents are a whole nother ballgame. So inference is important by itself because if you're gonna give answers, there's gonna be a risk that the answers may not be correct. Now there's tons of tools to work that out. But now in agents, now you get a whole nother level because now you're sort of maybe mutating the production system or you're executing multi-systems.
And so there's just a whole lot of new way of thinking, but starting foundation with the question that you have to ask is, are you willing to accept risk? Because if you're not willing to accept the risk, then you just can't use these tools.
Joel Moses (08:00.563)
So if I,
Lori MacVittie (08:01.027)
I like that.
Joel Moses
Yeah.
Lori MacVittie
I was just, the notion that I heard there is when my son started driving, right, we live in Wisconsin.
John Willis (08:09.121)
Mm-hmm. Mm-hmm.
Lori MacVittie
You are going to hit a deer eventually. If you're not willing to take that risk,
John Willis
That's right. Good one.
Lori MacVittie
do not drive.
John Willis
That's right.
Lori MacVittie
Really that's you have to approach it that way. So you accept upfront.
John Willis (08:21.837)
Yeah. And unfortunately, leadership, like we, I mean, it's part of that whole shift in thinking. Like the early day DevOps cloud started to sort of break down some of the never ever, we won't ever ever, we can never go to cloud, we can never deploy, you know, we can't, we must have sort of like air gapped, all these things. We've deteriorated that over the years, but now we're at a fundamental shift of, no, no, now you have to deal.
Leadership, you can't put your head in the sand and say, you know, this can't happen. Because it will, you know, like your deer example,
Joel Moses
Yeah.
John Willis
it's going to happen. I mean, if you're going to embark on inference, which is what LLMs do,
Joel Moses
Yeah.
John Willis
then there is a risk. And then your job then is to calibrate and try to safely mitigate that risk using a ton of tools like evaluations and guardrails and those things.
Joel Moses (09:16.573)
Right. So let's go back to the DevOps topic for just a second, because I'm really curious about one of the things that you said there. It seems to me that DevOps typical practice is to try to shorten the loop between human interaction and system action. So you create kind of a lifecycle which is gated through humans, but shortens the loop as much as possible so that you can deliver quicker.
It seems to me that AI, in the way that you're describing it, is a choice as to how much authority you grant to the system part of that loop. Is that correct?
John Willis (09:49.483)
Yeah, yeah. And you know, there's this notion of what, you know, HITL, right, the human in the loop. And then there's this notion of what human on the loop, right? And so I think it's, there's a learning curve, right? Like, and that's the other thing too, right? That what people are not doing, you know, I sort of joke that, I know you guys cover a lot about MCP and stuff, and I think it's an important conversation to have, but I think when I walk into large organizations now, the most critical sort of audit question is, you know, should we do CLI or MCP?
And like, those are the wrong questions, right? The real questions are if we're going to use agents, are we even scoping out the sort of the scope of like, is it an agent that does read-only? Is an agent that does write and mutates or is it an agent that executes?
Lori MacVittie
Boundaries.
John Willis
And then like, let's have that discussion as just being binary.
John Willis (10:48.942)
Agents, yes, agents, no. So if it's read, maybe we would learn in this sort of HITL to HOTL, you know, it sounds kind of weird, is that like we get better and better at learning about the, you know, it's like the remediation systems of the last decade, right? Early on, there was a lot of sort of pushback on how much you'd let a remediation system. But we learned over time that we can do auto remediation for incidents, calls, and we've added more authority.
Well, again, it's not like next Wednesday, we do agents that execute MCP that make, you know, billion dollar requests accidentally or delete whole databases, which actually seems to happen every day now. It's like, how do we work through learning that authority, calibrating? Okay, we've gotten better at this. This seems more consistent. Let's move on to the next level. Right. And I think it's the same feedback loop, right?
Well, it's a feedback loop, but it's a feedback loop on operational authority shifts.
Joel Moses (11:50.043)
It seems to me
Lori MacVittie (11:50.936043)
Well, you're actually talking about setting up boundaries
Joel Moses (11:58.287)
Yeah.
Lori MacVittie
in which the agent can operate. You can
John Willis
That's right.
Lori MacVittie
read only. And then if you try to violate that, if that agent says, "yeah, but I really want to write," you could just, right, you could put controls in to go, "Hey, you've exceeded your authority. I'm going to stop it." Right?
Joel Moses (12:13.288)
Yeah.
Lori MacVittie
The boundaries around what you're trying to do and you put those in place before to help mitigate the risk. As you were saying, that's one of the tools you can use.
John Willis (12:21.751)
Yeah, no, I think and then you start thinking about atomic nature of a what is the atomic nature of an agent? And then I think containment becomes an interesting discussion. You know, you have to get to sort of principles of like let's just not talk about we're going to do agents or not going to do agents. Or we're going to do AI, we're not going do AI. Let's talk about, like it's the same problem we had with cloud, right?
Everybody ran to the cloud. And then the first thing they realized, which was, you know, we probably should have had a data classification program, you know, before we ran it. So like, we need to think about like, what is our classification? What is our scope? And so I think that working through all that, then you get to a point where once you have that straightened out, now you can start thinking about containment, right? And what is the best containment level? So if I'm going to have read or write authority, you know, maybe it's WASM.
You know, maybe it's eBPF. I'm working with a professor who's doing intelligent traffic systems
Joel Moses (13:24.218)
Mm-hmm.
John Willis
and he's trying to run agents at the edge. And so he's exploring with eBPF. I think there's a lot of interesting ways that we can get to containment once we have an overarching discussion about what actually we want to do with AI. Duh, right?
Joel Moses (13:43.825)
Right. So, the two steps are first of all, figuring out what authority to grant, which to me sounds a little bit like how many rights are we giving to our DevOps intern? It's the same basic decision process that you go through when you're trying to figure out, you know, whether to trust and grant authority to someone. This is just something that you trust and then grant authority to. And that trust level will of course change over time.
And then you have to think about, okay, what if things go completely wrong with that process? How do I contain that process? How do I contain the blast radius of potential erroneous change within that loop? So I totally get what you're saying there.
John Willis (14:25.899)
Yeah, no. I was just going to say that, you know, that those are some new things we need to think about, particularly with agents is like, do we have escalation policies in an agent? Do we have kill switches? Right. These are things that new ways of thinking about, you know, and there's many more: new types of metrics that we need to think about, logging, you know, replay, traceability, explainability.
And we overuse that, but like, if we're going to have agents make multitask decisions that have consequences, we better be able--from an audit perspective--explain why it made those decisions.
Joel Moses (15:04.795)
So John, I want to ask you a historical question if I could. I did read your book Rebels of Reason, which is great
John Willis (15:09.854)
Oh, cool. Awesome, thank you.
Joel Moses
a rundown of how AI got to where it is on the backs of lots of unsung work. I want to ask this question, because you've seen it all. You've done a little bit of everything in your career. What feels new in artificial intelligence but actually isn't?
Lori MacVittie (15:27.365)
Ooh.
John Willis (15:29.931)
Yeah, that's a really good question. I mean, I think that one of the things that I think... So, you know, I wrote the book because I was just fascinated with storytelling, right? And when I wrote my Deming book, my goal was to write like, you know, sort of storytelling nonfiction. You know, like how would Michael Lewis write a book about Deming? And then I was like, okay, what's next? Okay, AI. How would Michael Lewis write a book?
But since then, I've thought a lot about what did I learn from writing that book? And one is the patterns of, if you go back to, you know, spend way too much time, but if you go back to, I mean, AI starts with formulated logic from Aristotle, right? And then it turns into Boolean algebra. And then it's Ada Lovelace turning into a program, and so on and so forth, right? And then you get into neural networks and all that.
And I think one part of that is like, you know, there's a lot of stuff going on with Yann LeCun right now. He's in my book, right? And he's got this JEPA and Fei-Fei Li has a new world. And so there's this, you know, there's a lot of people thinking LLMs are going to hit a peak, right? And I think understanding the history of how we got to, you know, deep neural networks will help us better understand where we go next. The other thing too is it is how humans react to technology.
Right? And so that's a secondary conversation. Like if you think about, you know, IBM's sort of Watson and some of the early, even the AlphaGo stuff, right? There is always this human element. So as I think about the book that I wrote, Rebels of Reason, it is this combination of this interesting technology of, at the end of the day, it's bits and bytes and Boolean logic, right?
The deep neural networks are just firing of Boolean networks, right? But then there's also, there's a sort of underlying story of how humans interact with those. I think that
Joel Moses (17:35.753)
Mm-hmm.
John Willis
as you read the story, there's always this like, what will they think of next? Or this has got to be the end all be all, right? From a robot in the 30s to the deep blue to all these things that every time we get to it, we think, wow, this is crazy. What is going to do?
John Willis (17:54.615)
So we're in one of those right now. We're in those
Joel Moses (17:57.022)
Yeah.
John Willis
and it'll change. There'll be some new patterns. Yeah.
Lori MacVittie (17:58.523)
Yeah. Yeah.
Joel Moses (18:00.679)
Yeah, it seems to me that human attachment to technology is the thing that is not different about AI. We become very attached to the technologies that are either patterned to behave or look like us or interact with us, right? People get very attached. And I think that that is not new about AI. I think that's been a constant presence.
John Willis (18:21.741)
Yeah, and that's story of technology in general, right? Like right now we're in this fluorescence of like, oh my god, this is like new, it's different. And like at some point we're gonna look back and like, oh AI was just like the computer age and the cloud age. And that will happen.
Lori MacVittie (18:39.344)
It's just new, yeah.
Joel Moses (18:40.859)
Interesting.
John Willis
It will happen. It's just a new way of technology. Yeah, that's right.
Lori MacVittie (18:39.344)
It's a new way.
John Willis
Yeah, that's right.
Lori MacVittie
Yeah, it's a new way to do it. And part of it is we fear change.
John Willis (18:47.149)
That's right.
Lori MacVittie
The other part is we embrace change, right? So there's that mix, right? So yeah,
John Willis
Yeah, yeah. Paradox. Paradox, yeah, yeah, yeah, totally.
Lori MacVittie
and we're getting out of time here, but I do want to have takeaways, right? What should someone listening, if they're in the enterprise and they're thinking about agents or they already have one running, what should they...
Lori MacVittie (19:07.431)
What should they be thinking about? What should they be doing actively before, I don't know, it orders pizza for the whole world? Cool.
John Willis (19:14.469)
Yeah, I think if you're in sort of a role, you're sort of somebody who has influence to a CIO, right? And maybe CEO, depending. But in the large regulated corporations, right, that's our first line of defense is in theory, the CIO should actually have a handle on all these things.
And so if you're somebody who's like an influencer or, you know, I'm trying to think of the terms that people use, but like a sergeant of arms or whatever that reports like. Go to the coffee pot right now, drink coffee, take a couple of deep breaths and go back and like literally try to force your organization to sort of like, let's step back for a minute.
You know, I mean, it's the common sense thing that's the critical thinking is, you know, what are the things that we know are common to all technology transformations? And like work your way out of that as opposed to being in this firefight of, well these people are using Cursor and we got a contract with Co-pilot and they shouldn't use agents and what happened to the database that got deleted? I mean, yeah, those are all important things, but they confuse what we know are the things that at organizational levels, especially in a regulated business.
We need step back. What are first principles? Are we doing critical thinking? What do we know about organizational design and discovery from hundreds of years of knowledge? And it all sounds cliche, but it's not. When we do it right, that's what happens.
Joel Moses (21:05.757)
Yeah. Yeah. I think my takeaway from this discussion is that it is safe to pause, step back, and consider things for a moment in the context of the practices you already hold and the ones that you want to improve upon. That it's tempting to kind of go and try to pursue things because they're the next hottest thing, but they may not be the right thing necessarily for your organization, which is what a CIO has got to analyze, I would guess.
John Willis (21:33.441)
Yeah, no, I mean I think, you know, it's that firefight. Getting out of that firefight, and it's hard. We all know this is hard, right? Like it is easy, you know, you've got a CEO screaming and hollering that their competitors are using AI, how come we're not? Where, you know, we've got vendors coming in and promising the world, right? These are like all these things. And the true leaders step back, you know, they go to the coffee pot, they take a couple of deep breaths and they pause for a minute to try to figure out like what is sort of common to all this that happens all the time, right?
Joel Moses (22:07.699)
Mm-hmm.
Lori MacVittie (22:09.839)
I like that. I think the takeaway, right, your talk about risk, right. There is risk. Decide what risk and
John Willis (22:16.652)
Mm-hmm.
Lori MacVittie
how much you're willing to accept at which step. You know, step back, go, I'm willing to take this risk right now, and then move forward. But also consider that you are handing authority to basically a really smart script.
John Willis (22:31.597)
That's right.
Lori MacVittie
So set up some boundaries and make sure that you can actually manage that risk as you move forward. AI is not going anywhere.
Lori MacVittie (22:39.001)
You can step back, have some coffee, right?
John Willis (22:40.717)
That's right, that's right.
Lori MacVittie
Take a deep breath and then move forward safely and quickly. If you do it, do it safely. Yeah?
John Willis (22:47.885)
I was asked on a podcast recently, what would be your recommendation to a 90 day plan to a CIO for AI? My recommendation is not have a 90 day plan.
Joel Moses (22:59.721)
Hahaha
John Willis
Like that's ridiculous, right? I don't know, is it a week? Is it like 300 days? I don't know. Let's figure it out.
Lori MacVittie (23:12.635)
I like that, I like that. I wish we could talk more, but that is a wrap for Pop Goes the Stack. Subscribe now so you're ready when the next can't miss innovation shows up in your architecture diagram.