Join in on weekly podcasts that aim to illuminate how AI transforms cybersecurity—exploring emerging threats, tools, and trends—while equipping viewers with knowledge they can use practically (e.g., for secure coding or business risk mitigation).
Hey, everybody. Welcome to this week's episode of AI Security Ops. Today, we are gonna break down some agentic terminology for you. Lately, you've probably been hearing a lot about agentic AI and different terms are being thrown around, and you're probably drowning in that vocabulary that is out there. Lots of different terms that are thrown around like they're interchangeable, but they're not.
Brian Fehrman:And so today, we're gonna go ahead and we're gonna untangle those concepts, talk about what each one actually does, and what it actually means for you. So before we get started, though, let's go ahead and let's talk about our sponsors. Black Hills Information Security, if you or your organization are in need of any kind of security services, whether that be AI driven pen testing, traditional external test, internal test, assumed compromise, physical test, wireless test, SOC monitoring services. Anything that you can think of security related, we can help you out with. Check us out at blackhillsinfosec.com.
Brian Fehrman:Additionally, we have the training branch, which is anti siphon training, where many of our consultants take the knowledge from their day to day activities. They package it up in an easy to digest and affordable format for you to consume to help you level up in your daily role or maybe, get your foot in the door on that, new position that you've been looking for. So check them out at antisiphontraining.com. So let's let's kick this off. So think I we have a bit of an analogy to start out with here.
Brian Fehrman:Yeah? In terms of agent picks or agents, we should say, which is you can kinda think of it like a new employee. And so with that, we have a a couple different terms that we can introduce for that. So a prompt can be kind of like the job description of that new employee, how they should you know, a different task that they they should perform. Skills might be the different procedures that they should follow in completing that those tasks.
Brian Fehrman:Tools might be, say, the systems they can log into or just different resources that they have at their disposal to accomplish the task. And then memory might be a notebook that they have where they can jot down different experiences, data that they've gathered that they can use in future tasks. And then we have the concept of sub agents, which are basically like coworkers or interns that they can delegate tasks to. So let's first dive into prompts. What actually is a prompt outside of the analogy?
Derek Banks:I'm gonna let Bronwen take this one because she has an entire, like, training thing on prompts, if I recall
Bronwen Aker:a prompt is input. A prompt is instructions that you're providing to the LLM. It's how you're performing a query to either extract information or to provide, direction in terms of what it is you want the LLM to achieve or perform or or research. So the what makes a prompt different from programming that most people are used to is because it can go in so many different ways. Whereas when you're doing traditional programming, you do a certain type of thing.
Bronwen Aker:A query will pull data out of, a database. You'll have other types of things that will involve stored procedures, or you may have other computational things that are going on are going on. And it's very difficult to provide this kind of instruction. This is why, software development is such a specialized field of study. But with a prompt, it shifts the emphasis.
Bronwen Aker:You're no longer telling, the machine exactly what to do and in in exactly what order. Instead, you are giving human like instructions. You're simply providing words to, a machine to tell it what it is you want it to do or achieve once you hit that submit button. Did I miss anything important?
Derek Banks:I think I would add that the, there's two, like like, subgenres of prompts. Right? Yeah. Then you have the system prompt where, you know, making it, you know, you you may have seen like you're a helpful assistant, you do this, that, and other blah, blah, I I kinda look at that as like your job description, and then the prompt itself, you know, kinda deviating a little bit from the analogy, which I think is a great analogy, by the way.
Bronwen Aker:It
Derek Banks:is. And so I agree. Your your day to day prompt that gets, you know, you have your system prompt with every time you put input into an LLM, you get that system prompt and then append, you know, prepended to your your your prompt input. Your prompt input is the task that you want to do, and what you were getting at, Bronwen, is that that's usually described in plain English. And as Brian has pointed out on previous episodes, for decades, we were trying to use plain English to tell computers to do the stuff and have the computer recognize that task.
Derek Banks:And it's only been here in the last couple of years that that's been, like, a thing. Right? And so, yeah, that's I think that's where I would without going into tokenization and a bunch of stuff that are are, you know, only a portion of our audience probably really cares about, I think that pretty much sums it up.
Brian Fehrman:Yeah. Yeah. I agree with that.
Bronwen Aker:And the differentiation that that you nailed about the the system prompt versus the user prompt, the system prompt helps establish the environment, And then the user prompt is the individual transaction,
Derek Banks:I guess. Yeah. And important for security stuff is a lot of guardrails. Some guardrails guardrails to keep the model from doing certain things and doing certain behaviors, are are oftentimes in the system prompt as a layer of defense. Now, I'm sure at this point, most folks have heard about prompt injection, and that is the security issue with prompts where, the LLM itself cannot distinguish between what is a good input and bad input, so to speak, or input with malicious intent.
Derek Banks:As far as the LLM is concerned, it's really just input and output. It sees a token stream. And so, it is a mistake to think that either the system prompt guardrails or internal guardrails are fail safe kind of things.
Brian Fehrman:Cool. Well, let's let's move on to talk about skills. So alright. Skills. And how do they how do these differ from prompts?
Brian Fehrman:You know, I would almost say to an extent that, I mean, skills are kind of like it's kind of like really fancy prompts that you store.
Derek Banks:Yeah. That's that's like a trick question. Right? Because it's turtles all the way down. Some point Yeah.
Derek Banks:It's all really prompt. Right? But yeah. So differentiating what a skill is, I would say that that is typically going to be what we would, as a human, consider to be more programmatic. Right?
Derek Banks:Like, it's going to be something that you're going to use time and time again. So for example, in the world of pen testing, and I'm not saying that we do this, I'm saying that this is an example. Right? You could have a write pen test report as a skill that takes in a certain type of data and out on the other side comes a report with the same formatting and everything every time. Now I'm being very vague about that on purpose, but that prompt is probably or that skill rather is probably a collection of something like type Script, maybe some Python.
Derek Banks:There's probably some YAML involved depending on your harness, right, which is another word that is in the word soup of vocabulary here. But you're right, Brian. The skill is I would look at that as sort of like maybe like a meta prompt.
Bronwen Aker:I think I think I have the perfect analogy for it. Alright. A skill is a recipe. I like that. Think about what think about what a recipe a recipe is a series of instructions, and they may be simple or they may be complex.
Bronwen Aker:And They're usually the same. Outcome
Derek Banks:Every time.
Bronwen Aker:Yeah. But the outcome of successfully performing a recipe is a defined product of some sort. And and I think the same applies to a skill. Do this, then
Derek Banks:do that, then do
Bronwen Aker:this other thing, and output it in this format.
Derek Banks:And then some some harnesses I've noticed. I've been playing with Hermes here recently, and I'll just quickly define a harness as a collection of all the stuff that we're talking about that interfaces with your large language model. So that's where, like, the end, you know, extra bonus word of the day. Right?
Bronwen Aker:Mhmm.
Derek Banks:And so I I think that really the the name of the game is to make a skill be something that is repeatable. And something that Yes. Hermes does that's kinda neat, is it'll start self making skills based on how like, it's self learning. It's like so I've been working with her recently. He's like, I think you need a skill for this.
Derek Banks:You keep doing this. I'm like, go for it. Let's make a skill. See There we go.
Bronwen Aker:I have to nag Claude to make skills for me. So no, I I really am looking forward to getting homies.
Brian Fehrman:Yeah. And go going along with the with the recipe analogy, which I think is fantastic. I I think it plays really well into the the nondeterminism as well too. Right? Because we have the the set of instructions, and ideally, like, those instructions would be followed exactly, and we would get the the same output each time.
Brian Fehrman:But it's not always the case. Right? I mean, like, I don't know when I make stuff, like, maybe I forget an ingredient or maybe, like, I leave it on the stove for too long or, you know, there's always, like, these weird probabilistic things that that random things that come up, and, you know, it's no different with, with the skills too that most of the time, we're gonna get the same thing out, but there is a little bit of room for, nondeterminism as well cooked into that.
Derek Banks:Yeah. I think it's a good point too.
Bronwen Aker:Cooks that are similar skill level, you give them the same recipe, there are gonna be slight differences because what they bring to the party. And in the same way, the different models and the different supporting stuff would bring different things to the party when you run a skill.
Derek Banks:Yep. Well, if you are someone that took Bronwen's class or you've taken some class on prompt engineering, you probably have already realized that you could use it like a chatbot and just put your question in, you get results. Right? Or you can be more specific in what you want and give roles and like, I want you to do this, this, this, and this. And back when I still use chatbots, I really don't so much anymore.
Derek Banks:I would be very detailed and specific with instructions on what I wanted. But moving to a coding agent or an agent, I'm that you know, it's not all about code now. It's about tools and everything. You know, moving to to that, like, the whole, like, game was to make things more deterministic in my opinion. Right?
Derek Banks:To make it more repeatable, to put more of a framework around the token stream that you're putting into the the large language model to get the same or very similar result each time. But you are correct, Brian. There still is a little bit of variance in there from time to time. So
Brian Fehrman:So I think that mentioning the agent portion of that or the yeah. Basically, the tooling portion leads us right into the next terminology vocab word, which is tools. So let's talk a little bit about about tools. So what's different to saying, say, a tool and a
Derek Banks:skill? Yeah.
Bronwen Aker:The tool is called bioskill.
Derek Banks:Yeah. That's what exactly what I was gonna say is, yeah, I think that tools are something that you you that the model will I'll use the word choose, which, you know, is probably not the right word, but words are problematic. The model chooses based on your input a course of action that involves using a tool that it has access to. Let's say that you had a tool that, did database looks up in lookups in a customer database when you asked, you know, when you put in, tell me about customer x y z, somewhere in that system prompt, there's probably a say, hey. Here's your list of tools.
Derek Banks:And when somebody asks about this thing, use this tool. Right? And and so then the model goes, hey, you know, I have access to curl or bash or your my database lookup tool, and I'm going to use that to get more context, which is a word we haven't used yet, but to get more context around, what my task is. And so I would say that the tool is what generally, what we would still consider to be a tool that a human could use, like the most common tool used, by my agents these days is curl. And so, and so basically curling a website and bringing back, you know, the data, it uses that tool when I say, hey, go off this website and get data, it'll bring it back and it'll, you know, then use that that data from the tool as part of its prompt context.
Brian Fehrman:Yeah. Yeah. That makes sense. So basically, it's it's presented with a problem, and it has all these different options that it can choose from to help solve that problem similar to you're working on a project at home, and it's like, oh, I've got a screw that needs a fastened. So then you're like, okay.
Brian Fehrman:Well, what do I got in my toolbox? Oh, look. I've got I've got a screwdriver. This will be the right tool for that. You pick it out and, go for it.
Derek Banks:Not not to the hammer. Right?
Bronwen Aker:Or or well I
Brian Fehrman:mean, use the tools you got.
Bronwen Aker:Yeah. We also have the cooking analogy. So it could be that, you know, alright. I've got the recipe. Now I need an I got the recipe.
Bronwen Aker:I've got the ingredients. Now I need to combine them. Oh, I need a mixer over here. Oh, I need, you know, this type of knife over here. Oh, I need to do this.
Bronwen Aker:The tools are just that. The tools for accomplishing small tasks as part of the larger task, whatever that may be.
Derek Banks:Yep. I like that one. Is a tricky one. Mhmm. Like, this is one that even until here recently, I was making it more complicated than I think it really really is.
Bronwen Aker:You make things more complicated?
Brian Fehrman:So
Derek Banks:anybody wanna take a stab at it, or you want me to do it?
Bronwen Aker:I'm curious to hear your stab at it.
Derek Banks:Alright. So memory, just like, we'll use the cooking analogy. Right? Memory is, essentially not the memory of the chef or the LLM where they can or that they remember to do things. Because the LLM does not have the capacity to remember your previous input.
Derek Banks:Right? And so it has to have some mechanism to know what you've done in the past. And so back on the chatbot analogy, you know, I used to, you know, talking about chatbots, I used to describe, you know, your context window, you have your system prompt, and then you have, you know, the response or your system prompt, your prompt, and then the response, and then your next input actually sends stuff. Well, that same thing is still true with agents where it's keeping all this thing, you know, you have a context window, which have gotten quite large here recently that that's basically all the the input and output that's going, you know, in into the LLM. Well, memory is sort of like a side thing to that.
Derek Banks:Typically, it's text files that are kept in, you know, a a folder in in your, you know, harness directory in quad. And the harness needs to know, hey, I have memory here, and I can go look up and see. And so memory to me is basically important things that I've done that the harness was like, Or or I said, hey, commit this to memory. And then in the future, when I say, hey, use the, you know, the VPS out in Digital Ocean that we've been using by this name, it might go and check and say, oh, I've got this memory. Or one that I did do here recently that was in memory, was like, it kept forgetting how to, like, do pull requests at at GitHub.
Derek Banks:It would figure it out each time. I'm like, why? And so I finally said, commit to memory. This is how we do this. And then now it just it does that.
Derek Banks:It does it does the same, like, thing every time now. Now, should I have a skill instead of a memory? Well, I mean, I think it all just depends. Right? But to me, this is just a a kind of repository of previous context that the model can pull from.
Derek Banks:And so that's why I kinda I was confused, like memory. I get why we call it that, but consider it's not like computer memory. It's not like, you know, people memory. It's like kind of a new kind of memory. Look at it as a series of of a a series of texts that the LLM can remember from.
Derek Banks:Maybe that's why they called it memory.
Bronwen Aker:Well, in if if I can jump in here, there's there's a been been doing a lot of stuff with with Claude lately, and also, as I was telling Brian earlier, had an interesting experience with my roommate in terms of memory and how much the LLMs retain about us. If we roll back to the analogy of the LLM as an intern, a really good intern will take notes of what the preferences and and likes and dislikes are of the mentor who is guiding them and and will also make notes of how to perform specific tasks. So I think of the memory as the intern's notes of important details that they need in order to do whatever. Now one of the other things that can be done, though, is that you can force load or force feed the memory of an LLM. And this is where I I I should probably not derail this conversation, but my my roommate has been fighting with the VA for for benefits.
Bronwen Aker:And I've tried to counsel him about what to share and what not to share, but, you know, he's not a geek. He's not a techie. And he did what any normal person would do, and every single form that he that he needed to review, he shared. And we got to talking about something, and I asked him to run a prompt through perplexity asking it what it knew about him, and the scales fell from his eyes. Now this is a little bit different from what happens normally in a memory.
Bronwen Aker:And and one of the things about memory is that we as humans do have some degree of agency over what an LLM does and does not remember. The first way that we control what an LLM remembers is by what it is that we share with that LLM either through what we enter in or what we upload to it for processing and analysis. The the other thing is that we can request and require, hey, add this to memory. I still have an ongoing battle with all of my LLMs about inline dashes, and I'm losing it, but I keep You fighting the good
Derek Banks:wanna know how I won that battle? Like, it actually
Bronwen Aker:You did it.
Derek Banks:Yeah. So I have a I have a personal context portfolio that I created, but I'm listening to the AI AI Daily podcast. So shout out to the AI Daily podcast. I'm I'm definitely a listener. I really like it.
Derek Banks:And they had a they have projects, and one of them was a personal context portfolio. This is months ago I did it. And so basically, I used their personal context portfolio, and their one of the questions was like, what is a hard no? Right? And I was like, emojis and em dashes all over the place.
Derek Banks:Right? And now almost every pass that I have something created at the end, there's a hook and, well, now life OS. I have, like,
Bronwen Aker:12 entries in memory with Claude.
Derek Banks:Yeah.
Bronwen Aker:And he's still getting freaking
Derek Banks:Yeah. Know, I need a hook, which is not a we'll we'll leave hook for another episode. You need a hook that calls, like, the right thing every single time. But, anyway, it's recalling from memory that Derek doesn't like em dashes all over the place. Right?
Derek Banks:Because I'm gonna circle
Bronwen Aker:back with you later. Definitely, we are gonna talk.
Derek Banks:Yeah. But that that personal context portfolio I share with, you know, some of the harnesses, like my main, like, clawed harness, and then I've been use I've been trying to move to, a docker setup again. But, like, it's basically meant to be portable, so I don't have to explain myself. I don't have to explain that I'm a pen tester working on this, like, all the time kind of thing. And so yeah.
Derek Banks:But all of that is memory. It's essentially, again, just files that are gonna get put in your token stream as part of what ultimately is still just a prompt that's going into the LLM. Because the LLM doesn't have the features that we're talking about. Like, you know, for simplicity's sake, like when Bronwen's saying like, hey, uploaded to the LLM. Yes.
Derek Banks:But not really. It's really the harness that's around that LLM, and it's just that our language is imperfect describing these things. And even in the show notes, like, of the words was teaches. And it's like, well, I like it. I think it more shows than teaches.
Derek Banks:Right? And so we changed it. And so but and I think that we you know, it's all of this is all of this AI stuff is analogous to the human brain in such a way, but it's not the same. And we use the same words, and sometimes it bothers me. Actually, it's just one.
Derek Banks:Just when people say they train an LLM, like, okay. That's one word that I'm gonna draw the line in the sand that you may need to mean what you mean that you know for that one.
Brian Fehrman:Alright. So that
Derek Banks:You wanted to sub agents. Right? Yes.
Brian Fehrman:Yeah. So let's let's go on to the last one here, sub agents. So sub agent is basically it's agent that can have all the things that we've discussed here kind of in its own little compartment if it wants. It can share these things with other agents as well. But when I say things, I mean, prompt skills, memory tools.
Brian Fehrman:Each sub agent can have its own set of these different items to accomplish a more specific task versus being generalized. And so, typically, what you would have is you have an orchestrator agent, some kind of an orchestrator that is, we'll say, kind of like the project manager that's delegating out all the tasks to the different sub agents. And so it's passing off just the information that each sub agent needs to accomplish its its goal, and then the sub agent takes that that granular information or, focused information, and then it can call upon its own skills, its own memory, its own system prompts, its own tools so that it is doing, just the tasks that it needs to do with the idea that if you have, more focus on getting these specific tasks done, you're likely going to get better results than having something that's much more general and broad across the the spectrum, if you will. So little, little experts, at at their own little tasks that we can delegate out to.
Derek Banks:That whole strategy is so effective. There is, and again, shout out to Daniel Measler and what was Pi and is now Life OS. But there's a research still in there that I use all the time. And it you know, Claude is the orchestrator, and I have Anthropic and or sorry, OpenAI and Google keys, Gemini key, like a API keys. And so when I research a topic, just by default, it's going to use Anthropic, it's going to use OpenAI, like ChatGPT, and it's going to use Gemini, and then take that stuff and put it together for me.
Derek Banks:And it is really effective. I really like that that skill. So.
Bronwen Aker:I'm I'm seeing more and more instances where people are calling multiple LLMs as sub agents for research projects and basically having them cage match it out in terms of what the best data is to come back.
Derek Banks:But it definitely doesn't have to be. Right? Because it could be let's just say that I had a scenario where I'm I custom made my own sub agents for like web app pen testing. Well, I could give one like authorizations tasks to go look at. And I could give one, you know, you know, fuzzing task to to go do.
Derek Banks:And that way, the LLM like Brian was saying is giving a narrower task. Because I think just in general, the more specific you can be with your input into an LLM, the better results you're gonna be. You don't want it to be a generalized thing. I mean, that's really the name of the game with all these things that we've been talking about and and harnesses and, is to give more context to be very specific to get more deterministic results.
Bronwen Aker:So since this is an analogy heavy episode, I wanted to throw out two possible analogies for the sub agent. One would be a project manager. I've done a bit of project management. That role seems to kinda fit. Another one would be the, conductor of an orchestra, helping conduct the the different instruments to make everything come together.
Derek Banks:And some harnesses are kind of built that way. Like, I've been messing around with OpenCode, and its whole thing is that you make agents to do, like, specific tasks, and I I kinda like that. And so I think we need to tie this all together to security. Right? And so we did mention earlier in the prompting that this, you know, was prompt injection was kind of main main type of a task or attack kind of a risk.
Derek Banks:But with skills, I think that there are things stored, so you might have to worry more about kind of like a supply chain kind of thing. Tools, well, tools grant agents agency. Let's see. Back in the chat days, we used to say, you know, giving it a tool is granting it agency, but I think that kinda got overtaken by the whole agent term. And so, excessive privilege is really the risk that you have with tools.
Derek Banks:Memory adds persistence, and just like, you know, stored cross site scripting, stored injection, or some kind of poisoning is the risk. And then sub agents, add delegation, so, broken trust chains, which I guess is getting at, more of do I trust the input that's coming from my multiple agents? Did I miss anything on those things that's about right?
Bronwen Aker:Nope. You wanna
Brian Fehrman:Thank you, Adam.
Bronwen Aker:Wanna send us out here, Brian?
Brian Fehrman:Sure. Let's let's do it. So I hope that, this episode helped clear up some of the terminology for, everyone out there who's starting to dip their toes into the agentic side of AI now that that's becoming all the rage. And I hope you enjoyed this episode, and tune in next time, and keep on prompting.