Subscribe
Share
Share
Embed
Remember when APIs were quiet little endpoints that waited politely for humans to click buttons? Yeah, that’s over. Now you’ve got swarms of LLM agents duct-taping tools together like caffeinated interns on Red Bull, firing off recursive calls at 3 a.m., and cheerfully melting your infrastructure while insisting everything is “working as intended.” Observability dashboards are screaming, rate limits are sobbing in the corner, and your security model still thinks it’s guarding humans instead of self-directed toolchains with the attention span of a squirrel and root access. Welcome to the new game: not keeping the stack up, but keeping it from eating itself.
In this episode of Pop Goes the Stack, F5's Lori MacVittie and returning guest Connor Hicks discuss the rapid adoption of MCP and the risks of going too fast without considering security, governance, and supply chain pitfalls. Listen now to take control of MCP tools and AI agents before they take over.
And after you've listened to the episode, check out our WebAssembly Unleashed podcast: https://youtube.com/playlist?list=PLyqga7AXMtPNV1zr2aTWEegep0FQU6Qvj&si=YZkHT7VeqfrANeZO
Creators and Guests
Host
Lori MacVittie
Distinguished Engineer and Chief Evangelist at F5, Lori has more than 25 years of industry experience spanning application development, IT architecture, and network and systems' operation. She co-authored the CADD profile for ANSI NCITS 320-1998 and is a prolific author with books spanning security, cloud, and enterprise architecture.
Guest
Connor Hicks
Connor has been working in cloud security for over a decade, starting with building enterprise security products at 1Password. He then founded a company called Suborbital focused on securing untrusted code in the cloud with WebAssembly which was acquired by F5 in 2023, where he now works on AI security as an Architect.
What is Pop Goes the Stack?
Explore the evolving world of application delivery and security. Each episode will dive into technologies shaping the future of operations, analyze emerging trends, and discuss the impacts of innovations on the tech stack.
00:00:05:21 - 00:00:36:06
Lori MacVittie
Hi everyone. This is Pop Goes to Stack, the podcast that side eyes every game changing feature like it's a 2 a.m. hotfix. I am Lori MacVittie. Proceed with caution because I am alone in hosting again as Joel is out. We're not sure why. I'm pretty sure it's not training. We asked him about that. He's not AI, so. My dog is back here, so, you know, he may want to interject
00:00:36:09 - 00:00:43:26
Lori MacVittie
just to, you know, keep things lively. But in the meantime, we brought Connor Hicks on again. Hi, Connor.
00:00:44:03 - 00:00:46:27
Connor Hicks
Hello. I'll be my, I'll do my best Joel impression.
00:00:47:00 - 00:01:14:04
Lori MacVittie
All right. Awesome. Awesome. Because we're going to talk about MCP. Right. Okay. Model context protocol. I think it actually stands for other things too, so we have to specify that. The style guide says so. So these are enabling tools. And it's a really exciting time. Everyone's excited by being able to do this. I know our latest research says that people are adopting MCP pretty quickly.
00:01:14:06 - 00:01:57:03
Lori MacVittie
Some might say too quickly. Because as we know, GB hackers found a nice list of unvetted MCP servers that are sneaking into supply chains, they're exfiltrating creds, and they are gnawing at the very trust we hand them. So we wanted to talk about MCP tools. Tools gone wild in the enterprise. Because this is a real risk, especially the pace of adoption around AI is so fast that we are just grabbing things without reading any of the instructions or, you know, considering the risks and just putting them into place because, gosh, they're cool.
00:01:57:05 - 00:02:19:07
Connor Hicks
Yeah. No kidding. I mean, first off, all I have to say is who could have seen this coming? All of us.
Lori MacVittie
Yeah.
Connor Hicks
There is such, there's such an excitement factor around MCP and, you know, connecting AI agents to every which thing that you do that people are leaping before they think. And it needs to slow down.
00:02:19:09 - 00:02:42:08
Lori MacVittie
It it needs to really slow down. It's not like we haven't seen this kind of thing before. Now I'm old. That's that's what the gray hair is, is, is from, really. That and four teenagers, but yeah. You'll you'll learn about that, Connor. You'll learn about that. But, you know, in the early days of the web, we had static pages
00:02:42:08 - 00:03:09:10
Lori MacVittie
and it was, I mean, it was cool. I mean, come on, it was awesome. But we really wanted things more dynamic, you know, dynamic clocks, countdowns, data, and CGI. Right? The the Common Gateway Interface allowed us to actually execute external code and give us things like database stuff. Right? Or just go out and retrieve something or run a tool. A tool. Hmmm,
00:03:09:12 - 00:03:35:06
Lori MacVittie
as you said, who could have seen this coming? Well, people who've been around a while might have thought, huh, this looks similar. So we we've seen this before where we we embrace the capability because it is it it's it accelerates so much innovation. But there are risks that we need to watch out for, especially with MCP because we're only, you know, 18 months or so into this.
00:03:35:08 - 00:04:05:14
Connor Hicks
Yeah, totally. I mean, I thought that we had a bit of a supply chain security revolution over the past couple of years, but the last few months is making me think that we might, might have forgotten a couple of things we learned from that, from that wave. Because, you know, just because the tool is out there, just because, you know, somebody has developed an MCP server to give your AI agent access to X, Y, and Z doesn't mean that you should just immediately pip install it and run with it.
00:04:05:21 - 00:04:08:19
Connor Hicks
You have to do a little bit of due diligence first.
00:04:08:21 - 00:04:30:15
Lori MacVittie
Oh, absolutely. I mean, every every company should have a process that says, "yes, it's okay to run this software," "no it's not." I know we have to go through considerable, right, evaluations of both the provider and the tool itself to make sure it's safe to be running in our environments, because we don't want things like this to happen.
00:04:30:15 - 00:04:42:19
Lori MacVittie
So you would think there'd be some kind of, dare I say the word, governance?
Connor Hicks
Oh.
Lori MacVittie
Yeah, I I'm sorry. I'll let myself out. Is that?
00:04:42:21 - 00:05:01:20
Connor Hicks
Now, it's one of, it's an engineers, one of one of an engineer's least favorite words. But I think when it comes to building secure and stable enterprise systems, it has to be a word in your vocabulary, or else you're going to get in big trouble. And there's kind of two ways to approach connecting, you know, your APIs and your tools to, to AI agents.
00:05:01:20 - 00:05:20:11
Connor Hicks
You can either say, oh hey, there's a shiny open source one on GitHub that I can just grab and pull into my system, or you can build your own. You can do your own, you know, blood, sweat, and tears to build these tools specific, you know, endpoints that are compatible with MCP and make sure that you do it all yourself.
00:05:20:11 - 00:05:37:02
Connor Hicks
So there's, you know, the question I think at hand is, do you take every API that you need access to and wrap it, you know, in a bespoke MCP endpoint yourself? Or is there a safe way to embrace the open source community that's coming out around these things?
00:05:37:05 - 00:06:08:08
Lori MacVittie
Well, and I, I like that thought because inside of that is the notion that maybe wrapping APIs in MCP, very specifically choosing which ones agents will have access to is actually a security and governance, right, method of making sure that they don't have too much access, that they can't do things they shouldn't be doing, and really controlling it while still enabling them to help you do automation or, you know, do things faster, whatever it is you're trying to achieve.
00:06:08:13 - 00:06:24:24
Lori MacVittie
So I think that's really important is to look at it, this is one of those times when security really is like an enabler. Like if you do it right, you're going to enable people to actually experiment and use MCP and agents to do new things without exposing yourself to risk.
00:06:24:27 - 00:06:43:21
Connor Hicks
That's right. I mean, principle of least privilege has existed for eons. And so I'm not quite sure why we aren't adhering to that with the world of MCP. Now, I can think of a couple of examples where somebody has built, you know, a wonderful, and they've put a lot of hard work in, you know, even if it's not malicious.
00:06:43:21 - 00:07:03:27
Connor Hicks
right, even if an MCP server that you pull off of GitHub is not malicious, it still tries to expose everything. Like take GitHub for example. You might have ten different tools and, you know, resources that you can access through the, you know, GitHub MCP server. But the application that you're building or the use case that you have might only need two or three of them.
00:07:03:27 - 00:07:10:09
Connor Hicks
And so why do you risk installing an MCP server that exposes ten when you only need three?
00:07:10:11 - 00:07:40:27
Lori MacVittie
Yeah. Least privilege should also be called least access when it relates to, right, APIs. You only need what you need. You don't need need everything.
Connor Hicks
That's right.
Lori MacVittie
And I think that's, it's, I'm thinking of like NPM poisoning, right, package poisoning where you know your your images just go out and they just pull whatever package, right, is out there. And people have actually used that to, you know, inject into the supply chain poison packages that then can run rampant in your environment.
00:07:41:00 - 00:08:14:03
Lori MacVittie
And these tools, you know, MCP servers, if you're just grabbing them willy nilly and going, that looks good, can potentially do the same thing if you don't, you know, where's your code review? Did you look through the code that you just deployed? That should be at least at a minimum, if you're going to trust it and make sure that it's not, you know, got some kind of a trigger or a redirect where it's, you know, going out and it's pulling, you know, data from poison pool, tool, pools—pools and tools very closely, right—from, poison wells, as it were then.
00:08:14:05 - 00:08:20:00
Lori MacVittie
And, you know, or running tools, it shouldn't. So I think that's really important.
00:08:20:00 - 00:08:49:12
Connor Hicks
100%. And, you know, we sometimes give these tools our credentials because we want them to do magical AI agent things for us, not realizing the implications that intentionally giving that access has. Even, even if the tool, you know, is not 100% malicious, you can still accidentally provision maybe an API token that has more permissions than you want it to have, and so that MCP server ends up allowing things that shouldn't happen. And AI
00:08:49:13 - 00:09:18:14
Connor Hicks
agents are squirrely, right? They will come up with, you know, solutions to problems in ways that you didn't expect them to. So maybe I write, you know, an AI agent to summarize my GitHub issues and, you know, write a plan for me for my day to go and tackle the issues that I have available. But maybe in an agents, you know, AI brain it thinks, well, if I just write all the code and commit it all to main, then I've done your work for you
00:09:18:14 - 00:09:26:27
Connor Hicks
and so that's the easiest way to organize your day. But if I don't give them permission to push to main in the first place, then we can't make that mistake.
00:09:27:00 - 00:09:49:26
Lori MacVittie
The the danger of telling it, "you are a helpful assistant." You know, maybe, wow. It was it really tried to live up to that, that system prompt there. The, you kind of, like, triggered a thought. Like, a lot of systems have, you know, and I could do this on most things, once I log in, I can automatically provision an API token.
00:09:49:28 - 00:10:13:26
Lori MacVittie
So, it is entirely possible that an agent could go, "oh, I can just do that," when it wasn't supposed to. So yes, giving it your creds and and having it go, "Oh I'll just log in. Oh I can do it that way. Let me get, oh." Right? And now suddenly it's like elevated its privilege through proper processes that exist and are absolutely valid.
00:10:13:28 - 00:10:16:17
Lori MacVittie
But it wasn't supposed to. But it didn't know that.
00:10:16:19 - 00:10:33:16
Connor Hicks
That's right. No, it's a little bit of a terrifying thought. I don't want to be installing an agent or an MCP server that is meant to help me, you know, perform some software development tasks and accidentally have it believe that it is now the GitHub admin for my whole organization.
00:10:33:18 - 00:10:58:13
Lori MacVittie
It, well yeah, it could it might, it might think that. Yeah. I mean there's a there's a prompt injection thing I've heard about that could, yeah I, right. I mean there's, there's so many ways that this could go wrong. It just it boggles the mind really when you think about it. It's, MCP is sitting, it's effectively acting as a proxy for any tool or system that you give it access to.
00:10:58:15 - 00:11:21:08
Lori MacVittie
And that's going to allow AI agents to use it in ways that you never imagined. You almost have to think of, if I gave a toddler this toy, right, what is, what is that toddler going to do? We think we know. But you know, I've seen sticks, right? I mean, that's the the typical case, right? You give a kid a stick: it's it's a sword,
00:11:21:08 - 00:11:42:05
Lori MacVittie
it's a magic wand, it's it's a laser, it's a light saber. You don't know what they're going to come up with. And tools are in their infancy right now. Right? This this notion, this MCP. And, you know, we kind of have to treat it that way that, you know, what could it possibly do that you know, I would never think to do.
00:11:42:08 - 00:11:56:22
Connor Hicks
It sounds like you're advocating for toddler proofing all of our APIs.
Lori MacVittie
I, absolutely.
Connor Hicks
Maybe that's the mental approach we should all have to AI agents. Treat them like a toddler with a stick and hope that they don't poke themselves in the eye.
00:11:56:24 - 00:12:13:05
Lori MacVittie
Yes. Well, isn't that part of the security mindset? Is, right, is is think about how could this be abused? Not how will it, but how could it? Because 'how will it' we know, right, some of those things. I mean, you're going to get this kind of an attack. People are going to do this. They're going to do that.
00:12:13:05 - 00:12:36:02
Lori MacVittie
They're going to. Okay, we know that. But how could they? Are there are there scenarios that we haven't considered? And then how viable are they? Because it's, you know, sure, you can imagine a lot of, you know, crazy things, but only some of them are viable. Those are the ones you really have to focus on and go after first, because they're more likely than the the crazy ones, if you will.
00:12:36:04 - 00:12:57:22
Connor Hicks
That's right. I mean, so when I write, you know, any piece of software, it's inevitable that there's going to be a, I'm just speaking for myself, there's going to be a bug here and there. I don't know about you, but there's going to be a bug here or there when I write software, and there's a whole class of software development techniques that you can use to limit the blast radius of a bug that might come up in software.
00:12:57:25 - 00:13:40:03
Connor Hicks
But we also don't have to be thinking of how do we limit the blast radius of like a misbehaving toddler, essentially? And does that mean that we have to go and just tightly restrict the, you know, access and the blast radius of all the potential APIs that these toddlers could access? I think that's the only way that we can do it today safely until we have better tooling, better, you know, supply chain around MCP, better security vulnerability, you know, stacks in the enterprise to to really look at these. Because we have plenty of tooling to look at Python modules and Go packages and Rust crates and scan them for vulnerabilities.
00:13:40:03 - 00:13:48:17
Connor Hicks
But MCP servers connected to a non-deterministic, nondeterministic LLM? I don't think we have enterprise scanning tools for that yet.
00:13:48:20 - 00:14:16:13
Lori MacVittie
I, yeah, I don't I don't think so. Right? I mean, it's kind of like scanning, you know, dynamically generated on the fly things. You're, you can't you can't necessarily prepare for that again, unless you're using your imagination and thinking ahead. But we we do need to think ahead. And some of it is, you know, we, the idea of having an MCP server have access to different systems and such is so that we can build workflows.
00:14:16:18 - 00:14:51:26
Lori MacVittie
And I think it's important to remember that in in most cases when we talk about 'we're going to put them into production, they're going to help us' it's so that they can do tasks that are essentially, you know, components of a larger business process. And maybe instead of just putting them in and saying, "okay, figure it out," you know, we we map that process and we say, "okay, this process requires these things, so they can have access to these systems," and really limit it as we're starting to build it out. Because in doing so, we'll get a better understanding of how they work, how, maybe how they think even.
00:14:52:02 - 00:15:01:21
Lori MacVittie
Right. What is, you know, they they keep going to this other system. Why? Right. Maybe maybe you learn something; maybe you find out you need to tighten your controls.
00:15:01:24 - 00:15:13:08
Connor Hicks
Right. You know what? This is reminding me I, in a previous life, I worked very deeply on WebAssembly. I think this is a topic that has come up on, you know, several F5 podcasts. I think we might even have a whole podcast about it.
00:15:13:08 - 00:15:15:07
Lori MacVittie
I think maybe.
00:15:15:09 - 00:15:41:23
Connor Hicks
But it reminds me of WebAssembly quite a lot, because one of the great features of WebAssembly is being able to completely lock down the execution environment of the code that you're writing. And so it begs the question, do we need some kind of similar paradigm for agents and tools? Do we need a deny by default environment that is maybe something that's created by the industry to help us constrain the blast radius of these agents?
00:15:41:25 - 00:15:58:05
Lori MacVittie
Well, isn't that the kind of the intent of zero trust? Is you start with zero trust, like actually zero trust, and then we only give you what you need as you need it. And then always go back and default to that 'no, you don't have anything' and kind of build it from there.
00:15:58:07 - 00:16:12:25
Connor Hicks
Yes. If only zero trust was the default in enterprises. I think we might get there one day, but I think it's still there's still, you know, some amount of work that needs to be done for really all the enterprises in the, in this universe to adopt those practices, I think.
00:16:12:25 - 00:16:32:23
Lori MacVittie
That's true. But I, I actually thought you were going to say if only zero trust were real. I mean, that's that's where I thought you were going and you, you didn't. So now I'm, I'm not sure whether that's me or you. So, but it's it's very hard. It's very difficult because it's a complete change in mindset from how we've traditionally operated.
00:16:32:23 - 00:16:57:27
Lori MacVittie
Because most environments, way back in the day were completely open because they were completely contained. Right? They didn't have to worry about it. You either had access to a system or you didn't. It wasn't at the object level, which we know from things like OWASP, right. Object level authorization is actually very important. So you didn't have that.
00:16:57:27 - 00:17:37:05
Lori MacVittie
So we're actually building from a completely open and trying to reduce down. And we know that the human condition is that when you take away access and privileges, they get kind of frustrated by it. Especially when they don't expect it or they needed access to do their job. So it's a very difficult process to navigate in terms of like cultural mindset in an organization, as well as just putting the tools in place, because a lot of organizations have big chunks that were never built with this kind of a technology in place or even in mind. Like, would we ever need that?
00:17:37:05 - 00:17:42:08
Lori MacVittie
No, just keep it open. So it's very difficult. It's very hard.
00:17:42:10 - 00:18:01:00
Connor Hicks
It is. You, I mean, you're reminding me of, you know, I'm trying to eat healthier, so I hide my snacks so that I can't accidentally go and eat more than I should. You know what I mean? It's that kind of thing where you are used to something working in a certain way, and then adapting is is hard. It's just not our, it's not our nature, I suppose.
00:18:01:03 - 00:18:10:14
Connor Hicks
So, I mean, what are the takeaways? What what do we think is important for people to consider when they're trying to adopt MCP and tools?
00:18:10:16 - 00:18:35:08
Lori MacVittie
Wow. I, Connor's playing host. I like this. Well, Connor, you know, I think, now what, I mean, one of the takeaways is like, 'whoa, okay, slow down for a second.' Yes, MCP is incredible. Yes, you should adopt it, but you should also be aware of the risks and approach it more strategically with security in mind. Like that's the big takeaway.
00:18:35:08 - 00:18:53:09
Lori MacVittie
Like these do not come risk free. And if you just let them run willy nilly, they are going to wreak havoc across all of your infrastructure, your environments, period. So be careful with it. Yes, do it, but do it carefully.
00:18:53:11 - 00:19:17:12
Connor Hicks
That's right. I would also say, let's not forget the last, however many decades of supply chain and access management best practices. Just because you put a shiny bow on it that says MCP server here, does not mean that you should forget all of the things that we've learned about making secure and, you know, safe enterprise software.
00:19:17:14 - 00:19:48:27
Lori MacVittie
Absolutely, absolutely. And maybe a third one would be: have a policy around the use of these public MCP servers that are out there on the internet offering you access. Because you don't have any visibility into what they're actually doing, what tools they're calling, what they're collecting, and how they're behaving. So, be more thoughtful about what public MCP servers you allow to interact with your internal systems
00:19:48:27 - 00:19:52:08
Lori MacVittie
and, you know, the people that are using AI.
00:19:52:10 - 00:19:58:25
Connor Hicks
Absolutely. Just because an AI agent promises to make your life easier doesn't mean it's not also going to rob you blind.
00:19:58:27 - 00:20:04:18
Lori MacVittie
Wow. We are terribly cynic cynical today. I mean, just, we are but,
00:20:04:25 - 00:20:07:12
Connor Hicks
That's the security professional in me.
00:20:07:15 - 00:20:27:23
Lori MacVittie
It's going, that's the security side. Well, before we get even more cynical and decide to find the red button and just, like, turn everything off. Yeah, let's let's wrap. And we'll ask people, hey, subscribe now, stash your backups, absolutely, and join us next time for more production grade chaos.
00:20:27:26 - 00:20:30:19
Connor Hicks
Just don't let the agent stash your backup, cause it will lose it.
Lori MacVittie
True.