Subscribe
Share
Share
Embed
Agent skills were introduced less than six months ago, and they’ve already graduated from “handy configuration” to “supply chain artifact.” In this episode of Pop Goes the Stack, Lori MacVittie talks with security expert Peter Scheffler about why skills, often packaged as YAML, are becoming portable, shareable, and dynamically loadable in ways that attract attackers fast, including skill poisoning and repository-based compromise. The fact that there’s already an OWASP Top 10 focused specifically on agentic skills tells you how quickly this risk surface is forming.
Peter breaks down what “skills” really are: anything from a narrow tool instruction to a broad workflow like “prepare for a podcast.” Skills can be created by humans, generated by agents, and even expanded by tools that add more skills, which creates a compounding trust problem. Once skills can be modified, composed, and distributed, you need provenance, signatures, hashing, and an approval process, but simply copying the traditional package ecosystem isn’t a silver bullet because supply chain compromise is already a reality.
The conversation pivots to what actually helps: least agency. Define what actions an agent is allowed to take, and constrain execution at multiple layers, not just in a system prompt. System prompts are guidance, not enforcement, and relying on them alone is asking to get burned. Then assume unintended action, treat all external content as untrusted input, and focus on stopping unsafe actions at the boundary rather than trying to prevent the agent from ever attempting them.
Finally, Peter stresses observability. If agents can make their own calls, you must log agent-to-agent interactions, tool usage, and skill loading, because you’ll need forensic data when something goes wrong. For enterprises, the practical starting point is clear: follow emerging frameworks (OWASP, NIST), standardize which agents and skills are allowed, store approved skills in a controlled registry, enforce authentication and authorization to access them, and be ready to collect a lot more telemetry than you’re used to.
Creators and Guests
Host
Lori MacVittie
Distinguished Engineer and Chief Evangelist at F5, Lori has more than 25 years of industry experience spanning application development, IT architecture, and network and systems' operation. She co-authored the CADD profile for ANSI NCITS 320-1998 and is a prolific author with books spanning security, cloud, and enterprise architecture.
Guest
Peter Scheffler
Peter Scheffler is a Senior Solutions Architect at F5, focused on API security, post-quantum cryptography, and next-gen app delivery. Known for translating complex tech into memorable stories, he blends deep expertise with real-world demos and a bit of humor to help teams secure and scale modern applications.
What is Pop Goes the Stack?
Explore the evolving world of application delivery and security. Each episode will dive into technologies shaping the future of operations, analyze emerging trends, and discuss the impacts of innovations on the tech stack.
Lori MacVittie (00:03.08)
Welcome back to Pop Goes the Stack, where compliance is a calendar reminder and security is what you add right after launch. Yeah, that's how we do that. I'm Lori MacVittie and today we want to talk about consequences since no one else scheduled them. Yeah. Yeah, consequences from agent skills. Remember agent skills were introduced not even six months ago, I'm pretty sure. It's been, we're on a tight, tight tech evolution schedule here.
And so agent skills are basically plugins for AI agents. They're packaged in YAML because apparently we've learned nothing in the last 20 years. But to be fair, YAML isn't the problem. I mean, it's a problem because people really don't like it, but it's not the problem. The problem is that skills are becoming portable, they're shareable, and they're dynamically loadable.
Which means they're no longer just configuration files, they're supply chain artifacts. There's that word. That should sound familiar. So, they're already attracting attention from attackers. There's already an OWASP Top 10 that's out there, specifically focused on just agents. It's agent skills. Which is crazy cause when's the last time a configuration artifact, right? That's the whole top 10.
Like APIs, broad. Web, broad. No, agent skills, like this file, we have an entire top 10. So, you know, I mean people have already been mucking about with agent skills, poisoning them, right? Because they're downloaded. They're something that you can get out of a repository or a GitHub repo. Right, some kind of centralized storage where you can just grab them and then you can use them in real time. So, we wanted to dig into agent skills and to do that we brought Peter Scheffler. Welcome.
Peter Scheffler (02:08.875)
Thanks, Lori. Thanks for having me.
Lori MacVittie (02:10.723)
I'm excited you're here.
Peter Scheffler
It's been a while.
Lori MacVittie
Yeah, it has. So I'm glad you actually came back. I was
Peter Scheffler (02:15.943)
I told you,
Lori MacVittie
kind of worried maybe we
Peter Scheffler
I would always say yes. Thank you. Yeah.
Lori MacVittie
Ha ha ha. That is awesome. Cause I know that for like the last two months of your life you've been digging into agents and skills and just really getting into it from a security perspective, which is kind of your forte here. And so, you know, let's talk about you know agent skills. Let's talk about both like what can they
Lori MacVittie (02:40.243)
do, like what kind of skills are we talking about? And then, right, what are those security risks and what are we going to do about them?
Peter Scheffler (02:47.245)
So a skill is anything, really. I mean it, like you described it
Lori MacVittie (02:50.995)
That's
Peter Scheffler
well with it's a YAML file, but that YAML file could be anything. That could be how to use a program. And here's, you know, here's where the, you know, access the program and those kind of things in that skill file. But it also could be something more broader, like it could be how Peter prepare for a podcast. And it could be the things that I expect the tools to use, and it might explain, you know,
Peter Scheffler (03:17.121)
here's the things, pull out some points you want to talk about, blah blah blah. These are the things you should be looking for. This is the format Peter wants it to be presented to at the end. So it doesn't have to be an atomic thing, like maybe I have a skill for SketchUp to do drawings or something like that. That's a skill. But a skill could also be, you know, prepare for article writing or prepare for a podcast and be much broader.
And then it defines all the things that are going to go on inside of that and the output formats that you'd want. That's great because it allows you to sort of say, okay, here's how I want you to work, here's how I want my tool system and my tool--I don't know, infrastructure or you know whatever you want to call the the entire environment--to work together. The problem is it could, you can throw anything in there and
Lori MacVittie
Yeah.
Peter Scheffler
and a tool could add its own skills. So, you know, it could say, "Hey, I need this skill to go do this. So I'm just gonna go add another skill," and sort of build and build and build and build and build, right? So I mean I don't write my skill files. I ask an agent to write my skill files, right?
Lori MacVittie
Well, yeah.
Peter Scheffler
So, what could possibly go wrong in that, right? So, yeah.
Lori MacVittie (04:28.317)
No, nothing. Noth-, I mean it sounds like we took the early idea of tools, remember? You know, if the word like, you know, map or weather or this comes back, then call this tool
Peter Scheffler
Yep.
Lori MacVittie
and a tool was basically a function that called out to something else and did it. And now we've just abstracted that and said, you know, we've got all these great languages and capabilities where we can just execute code dynamically and do things and kind of build on the fly. Let's leverage that and define it in this agent skills, right, MD, right?
Peter Scheffler (05:08.886)
Yeah, right.
Lori MacVittie
In YAML. And then we'll just do that dynamically and that will let us expand. Which on the surface sounds like an awesome, awesome idea. I mean, it's not new. I, you know, I always hear this and I'm like, I think DLLs. And what did we get from that? We got DLL hell.
Peter Scheffler (05:27.746)
Yeah.
Lori MacVittie (05:27.951)
And we didn't want to go back there, but here we go with agent skills. I mean, same kind of premise, right?
Peter Scheffler
Yep.
Lori MacVittie
And you the problem is if you're writing, I mean, how do you verify it? How do you, you know, if it's written dynamically, how do you verify it? I, can you put code in it? What if there's things hidden in it? Like I mean nothing, it's not that it's YAML, it's that it's writable, dynamically loadable, executable. And you have kind of no visibility into what's going on in the moment. Right, I think that's the scary part.
Peter Scheffler (06:02.145)
Yeah. That is the scary part. And as that ecosystem, if that's the word I was looking for before, sort of grows and I want to share a skill with somebody else, or I'm in an enterprise and how do I share
Lori MacVittie
Well, yeah.
Peter Scheffler
those skills, right? And who validates those that they are and once they're validated, how do you track that? So there's a growing need now for, you know, an agent manifest that says, you know, here's what it is, here's who wrote it, it's signed. There's providence to say where it came from and its life and all those things. And then there's hashing of the files that are in there, and then there's some sort of ecosystem.
And then you have to package all that up, and now you have to push that out. So now do you need some sort of repository or vault where you're storing these? And it quickly becomes, you know, the whole SBOM problem of how do I make sure that the libraries that I've been using in my Node.js libraries from five years ago or 10 years ago, you know, how do I know that those have been validated and checked and that that the company allows me to use those?
Same thing goes for those skills. I could write a skill file and use it, and maybe I test it and I've got some, you know, wherewithal to do it correctly. But if I share that with somebody or an agent goes and makes a change to that file, that should invalidate that skill. Like we shouldn't allow it to be used anymore. Maybe it's right. Maybe there's nothing malicious in it, but we need to validate that too. So it becomes a validation and checks and balances process.
Lori MacVittie (07:39.967)
Yeah, I, there were a couple of things in there that were intriguing to me. I mean, when you were describing, you know, signed, hashed, you know, certified, all of this. I'm like, oh, so you mean like packages.
Peter Scheffler (07:52.93)
Yep.
Lori MacVittie
Like we're talking about like we need MPM. You know, we need to be able to apt-get our, you know, agent skills. However, you know, then you look at that and you go, Yeah, but that's not any more secure. I mean, none, we've done that for
Peter Scheffler (08:08.258)
Yeah.
Lori MacVittie (08:08.084)
here we go back to the, right, 20 years. We've done that and the supply chain is compromised. Not will
Peter Scheffler
Yeah.
Lori MacVittie
be, is today, right now. You don't know. Right? There's
Peter Scheffler
Right.
Lori MacVittie
just no way to know. So we're talking about, well, let's just do the same thing with agent skills. And it kind of, it frightens me because like that's probably not gonna work any better than what we have.
Lori MacVittie (08:34.664)
But you also mentioned something in there you said, you know, put it in the vault. And I'm like, wait, we don't currently store artifacts like that in vaults, right? That's for credentials and secrets and things like that.
Peter Scheffler (08:46.997)
Yeah, but you could
Lori MacVittie
But
Peter Scheffler
but it could be a registry or something like that too, right? So
Lori MacVittie (08:50.334)
Right.
Peter Scheffler
you could use, yeah. So then you're,
Lori MacVittie
Could you?
Peter Scheffler
if you- Yeah. If you could store those as a say a tarball or something like that and then have that stored in the registry, then there's a way to ship that and have some validation against it. Again, you're relying on the on, ehhh, it's a trust and a trust and a trust and a trust.
Peter Scheffler (09:13.225)
It's still not perfect, but if we just rely on these YAML files in of themselves, it's gonna come down
Lori MacVittie
Bad. Yeah.
Peter Scheffler
crashing quickly. Yeah, really bad. So
Lori MacVittie (09:21.694)
Yeah. Yeah. Don't do that. So number one, just don't. Don't do that. Like don't trust that. Cause that's like taking code from me. Just don't. Don't trust it. Don't, I wouldn't. I wouldn't trust my own code these days. So don't trust YAML. But if we, well, if we can't do like, you know, vault, I mean, is there a way to restrict the agent? And you've been digging into this, so I ask you: to restrict the agent from what tools it can use?
Or skills that it can load? Like can we back up and say, hey, maybe it's not that all these skills exist, it's that the agent just right unilaterally, you know, uses things and whatever. Can we restrict it that way?
Peter Scheffler (10:04.671)
So there is a hierarchy, right? So you can set
Lori MacVittie (10:06.986)
Okay.
Peter Scheffler
a hierarchy in your environment so that you can say, you're not allowed to do this. You can only do this with like--sorry, let me, I just I even had a little chart--so, you know,
Lori MacVittie
Ahhh.
Peter Scheffler
maybe there's several, there's, so I don't know if I've come up with this, but I've stumbled across this of it's like it's least agency, right? So we wanna kee-, so we got least trust, now we have least agency, right? So we wanna
Lori MacVittie (10:29.204)
Yeah.
Peter Scheffler
Yeah, so there's the buzzword.
Lori MacVittie (10:29.204)
No, love it.
Peter Scheffler (10:34.525)
Yeah. But we want to be able to say, okay, it can answer something, and cite those answers so that we can give a level of confidence on that. It can retrieve things, it could analyze things, it can draft things and modify things. So those are certain agency things that we can define in the overall, not just the skill system, but in the application that's calling those skills, right? Cause you're always gonna have maybe Claude Cowork or you've written your own environment.
You know, there's a bunch of different ways that you could use these things. And so you define that and say, Okay, these are the things that you can do without human interaction. You can't execute. You can't ever execute anything. So that the top level system is gonna put those in. Now, again, you have to make sure that the system isn't trusting input, because you know, I was saying to you earlier, one of the things that, oh, you know, while I was looking at having it read my emails, someone could send a prompt in an email. And I might not even see the prompt
Lori MacVittie
Peter Scheffler
in the text, it could be, yeah, you would do that. Yes.
Lori MacVittie (11:38.228)
I would do that.
Peter Scheffler
I know I would do that to somebody. Just, not even thinking about it, I would just do it. I would just, my fingers would drop on the keyboard and I would do it. But we would want to make sure that it says don't trust any incoming human-generated or user-generated content, whatever that's going to be. That could be pictures, that could be YAML files, that could be emails, whatever it's going to be. Like never take an instruction from those.
Peter Scheffler (11:59.158)
And if you define those high enough hopefully the system should be able to protect them. Now it can't always protect you from those like that's not, there's no surefire way to not but you have to be able to define those kind of actions in that least agency sort of thing. And then you need to define who's asking what, what's the action that's being asked, what skills are being used--to your point earlier, you can define these are the skills you're allowed to use, these are the skills you're not allowed to use.
So there's certain ways you could do that. And again, a skill can be a tool or it could be an overall action kind of thing that you want to define. But then you need to define authorization, right? So we typically want to rely on something like OAuth or MTLS or something like that that's going to give us some sort of authorization level, you know, and be able to track those. And audibility and traceability is really important.
One of the things that I was looking at Claude Cowork and the tools that it, the system it works. I mean, it's great if you're in the Claude ecosystem, but it's not great if you're stepping outside the Claude ecosystem. And one of the things when I was designing my, I called it the chef of staff,
Lori MacVittie (13:16.414)
Ha ha ha.
Peter Scheffler
the, ha ha ha, yes, I'm it's, come on, I'm a
Lori MacVittie
I love it. I love it.
Peter Scheffler
I'm an IT nerd pun. So, but when I was designing it, I kind of at the beginning said, well, I don't want to use cloud based LLMs.
Peter Scheffler (13:28.541)
I wanted to have something I could run locally. And one of the people I was talking to at the time, they were like, well, why? And I said, well, it's more secure. And they're like, hmm, is it? And I went, you know what? And I started thinking about that. It's not necessarily more secure. It's more secure because I can control it, but that's not necessarily more secure. So I started to think, okay, now there's certain cases where you want to use cloud-based or frontier models.
Peter Scheffler (13:57.624)
And there's times where you want to use your own models. And a lot of my customers and in, you know, in the space I'm in, they all want to run everything on-prem because it's "more secure." Well, is it more secure? You still need the authorization, you still need data checks, you still need
Lori MacVittie (14:16.03)
Yeah. Yeah.
Peter Scheffler
system...all those things need to be in place. Yes, maybe you're not sharing the inference and the prompts with somebody externally.
Peter Scheffler (14:25.313)
But that's really the only level of security that you've got. Like anything else you have to
Lori MacVittie (14:29.062)
It's more privacy.
Peter Scheffler
build around it. It's a privacy
Lori MacVittie
Yeah. It's more
Peter Scheffler
It's a privacy n-...
Lori MacVittie
about, right, it's privacy and control. I mean,
Peter Scheffler (14:34.658)
Yes.
Lori MacVittie
there's valid reasons like using, sorry, like using cloud stuff, you know, tell me about my fish tank. Great. You know, tell me about this thing, I have a weird question, you know, who sang this song? But, you know, divulging like deeply personal information or right critical--right, we know that already--corporate
Peter Scheffler
Mm-hmm.
Lori MacVittie
information does not go
Lori MacVittie (14:58.064)
in a shared anything. Whether it's a shared, you know, LLM or a shared app or a shared drive, right? We just, we don't. It's not for
Peter Scheffler
Right.
Lori MacVittie
sharing, right? So, sharing is not always caring. Sometimes it's
Peter Scheffler
Ha ha ha.
Lori MacVittie
ba-, see I can do it too.
Peter Scheffler (15:15.437)
Yeah, you can.
Lori MacVittie
Ha ha. Well I'm, so you were talking about like these restrictions, right? That you can put in there. I, they're in, you put those in that agent skills file, correct? In the definition?
Peter Scheffler (15:26.647)
So you can put them in the agent skills file, but they also they're also higher up in the application stack too. Like you want to make sure that the LLM that or the application that's calling those has those defined as part of their system prompt or the actual configuration, right?
Lori MacVittie (15:43.41)
Ahhh, okay.
Peter Scheffler
So you wanna be able to protect them at that level and then you still want to protect them at the skills level. Like you have to have them all the way through. Because it,
Peter Scheffler (15:55.51)
again, to me it's there's too many opportunities for it to get broken.
Lori MacVittie (16:00.085)
So we're going back and putting the boundaries into the application logic, right, kind of as a yeah, you don't, you know, it can't do this. I mean,
Peter Scheffler
Right.
Lori MacVittie
because then I heard you say system prompt and I'm like, it'll lie. It'll ignore it.
Peter Scheffler (16:15.501)
It will.
Lori Macvittie
Like if you think, if you are relying on system prompts in general, this is just good advice. If you are relying on that, you're gonna get bitten
Peter Scheffler (16:25.185)
Yep. Yeah.
Lori MacVittie
eventually. Like it's gonna turn back on you.
Lori MacVittie (16:28.014)
99 times it can actually listen to your, you know, your system prompt that says, don't do this. But the one time it's like, that's the only way I can do this thing, it's gonna do it. It does not care. It's the biggest honey badger that has ever honey badgered. Like it's just, it's
Peter Scheffler (16:45.099)
Right. Yeah.
Lori MacVittie
they're scary.
Peter Scheffler
Yeah.
Lori MacVittie
That's scary.
Peter Scheffler (16:45.099)
Would another way to do would that be to have a referee model that's in there as well that's doing refereeing on the on the prompt? I mean, that's layering security on top of security. I mean, I can see how you can do things like prompt injection protection, but does that help you with-- and I don't know the answer.
Lori MacVittie (17:04.872)
Yeah.
Peter Scheffler
I'm posing this question, so...
Lori MacVittie (17:04.872)
Yeah, I, no, it's a it's a good question. And, I mean, ultimately I think this is the same problem we have with things like DDoS or right any kind of external attack. Right? You are not stopping them. Those attacks are gonna be launched whether you have protections in place to prevent the damage from them or not.
Peter Scheffler
Right.
Lori MacVittie
And that's kind of the, right, the approach we have to start taking with these things and go, Look, they are going to lie, they're going to cheat, they're going to do bad things. So how do we, right, stop that action? So if it loads a tool it wasn't supposed to and it tries to execute it, maybe there need to be boundaries in place or safeguards in place that are like, hey, hey, hey, uh, you're not allowed to like cross the road there. I'm sorry, go back. Right?
And just stop it. Is to stop pretending we can use system prompts and other mechanisms to prevent the invocation and just stop the action, right? And just be like, nope, you're done.
Peter Scheffler (18:07.287)
Yeah. The other, I think the other side of it too is observability, right?
Lori MacVittie (18:11.593)
Yeah. Yeah.
Peter Scheffler
So, you know, as assume the breach and assume that you need
Lori MacVittie (18:15.657)
Yeah.
Peter Scheffler
you need some sort of, you know, forensic data to be able to do that. So being able to pull information out and be able to say back, okay, you know what, we had a breach. I mean I had a very minor breach this week where, you know, someone's personal email was on an email, right? And you know, and I said to the customer, Hey, your personal email is on here.
Peter Scheffler (18:37.003)
Right. Did you know that? And they're like, Oh yeah, no, no, no. It's fine. It like they removed it because it was their email. So you have to say the things happen, they happen. So let's make sure that we call them out, that we have observability, and that we make sure that we correct them as soon as possible. So if we have observability and we have the systems in place to say, Hey, you know what? This data was touched, this you know, these things were impacted, you can then react to them as quickly as possible.
Doesn't, even if it's minutes, seconds, hours, days, whatever it's gonna be, you can still react to them if you've got observability in place. So...
Lori MacVittie (19:14.044)
Awesome, awesome plug. I, you know, I think and enterprises right now have to be going, ahhh, what do I, what do I do? What do I do? So maybe let's, you know, bring it back to the like two different things here, because I'm hearing two different threads. But the first one, right? I mean agent skills. Like, what should an enterprise do who's thinking about, yeah, but these things are cool and they're helping us and we really want to leverage them because they're becoming the standard already?
So what's the best approach to that for an enterprise to stay secure?
Peter Scheffler (19:45.166)
So, you mentioned earlier, you know, the OWASP agentic rule set. NIST has a framework for agentic tool sets as well. So definitely become aware of what those are and understand what the industry is saying we should be doing. Being able to define the agents that people are allowed to use, those skills that people are allowed to use, define those skills as the ones that are allowed to be used inside the organization. And then have a repository where we've got some process.
You know, maybe there's a you've got a registry where those are stored and that's where people can access them. And then you've got authorization authentication that's allowing people to access those files and pull those files out. And then you're, like we were just saying, making sure that you've got some sort of logging of the interactions. Like these agents are making their own calls. So you're not just logging the prompt and the persons asking the system itself.
You need to log the agents and understand what the agents are saying to each other. And so get ready for a lot of disk space
Lori MacVittie (21:01.054)
Ha ha ha.
Peter Scheffler
to store these logs. But without it, you're
Peter Scheffler (21:05.077)
setting yourself up for disaster. It's gonna happen, but you need to be able to look back and say, okay, we can solve this by you know mitigating this or, you know, or at least letting people know that there was some sort of data breach when these systems work. It's a learning process. You mentioned earlier, this is just months old.
Lori MacVittie (21:22.676)
Yeah.
Peter Scheffler
I mean, we've got MCP, we've got A2A security, those are all things that are being put in play right now and the world is changing and shifting underneath our feet every week.
Lori MacVittie (21:33.055)
It is. It is. I mean, I like your advice and I think enterprises should take that away, right? Assume, it's not just assume breach now, it's assume, right, action unintended, right? Or action unauthorized. Assume it's going to do something it wasn't supposed to do, whether it's pulling from YAML or it just decided to. Right? That's, we know now at this point that's how they work. So assume, right, action unintended. And then the other piece, right? Least agency, I think, right, least privilege we already understand. Least agency, I think, will be it's a great way to describe what we're trying to do. Like you know, the fewest actions you could possibly take is the best option, not hey, the world's your oyster, because we don't want that. We already
Peter Scheffler (22:22.721)
Well, it's a token waste too.
Lori MacVittie
know that would go bad. Yeah. Yeah. Yeah.
Peter Scheffler (22:22.721)
Well, it's a token waste too. I mean
Lori MacVittie
Yeah.
Peter Scheffler
there's a cost to like do, right?
Lori MacVittie
Yeah.
Peter Scheffler
So if you,
Lori MacVittie
Yeah.
Peter Scheffler
that, just by limiting what they can do can limit the cost and we're all seeing the cost going through
Lori MacVittie (22:36.104)
Wait, wait,
Peter Scheffler
the roof with, so...
Lori MacVittie (22:36.104)
so security is actually improving costs?
Peter Scheffler
Ha ha ha.
Lori MacVittie
Like, wait a minute.
Peter Scheffler (22:40.077)
I did not say that.
Lori MacVittie
That's, okay,
Peter Scheffler
But I'd love to say that.
Lori MacVittie
that's what, we want to be able to say that, we're thinking, but we're not ready to say that, unfortunately, this time. And now we're out of time. That's a wrap for this episode of Pop Goes the Stack. But hey, please subscribe before we circle back becomes we notify affected users.