Lori MacVittie (00:04.302) Hi everyone, welcome back to Pop Goes the Stack, where AI is everywhere except in the part where we define success metrics. I'm your host, Lori MacVittie, and I'm joined by co-host Joel Moses. Hi Joel. Joel Moses (00:18.308) Hello, Lori. Lori MacVittie Alright, let's walk through what good looks like before we deploy smart. So today we're gonna talk about agentic fingerprints Lori MacVittie (00:29.76) otherwise known as the chaotic digital exhaust of AI agents running wild in your network. Yay! Because right now, enterprise security is treating autonomous agents like very fast, very polite humans, but they aren't. They're API-guzzling black boxes that don't sleep, don't use browsers, don't care about your rate limits or policies or compliance or anything else actually. They don't, they're honey badgers. They don't care. So how do you tell what's going on? How do you know what's going on? How do you figure out how you got to the point where everything has gone haywire? And part of that answer could be agentic fingerprints. Right? We fingerprint a lot of things. We fingerprinted, you mentioned TLS, you've got TCP browsers, users, but AI fingerprints are a different beast. Lori MacVittie (01:25.071) So today we brought someone who knows about agentic fingerprints, Lee Ennis. Lee, welcome. Welcome. Lee Ennis (01:32.208) How you doing, folks? Lori MacVittie All right. Joel Moses Welcome. Lori MacVittie We want to get going. I'm gonna have Joel kick us off because he always does a great job. Joel Moses (01:39.056) Sure. Lori MacVittie So kick it off. Joel Moses Yeah, Lee, welcome. And again, your fingerprints are all over our products, and so it's Lori MacVittie (01:44.91) Ha ha ha. Joel Moses appropriate that you come talk to us about this. Joel Moses (01:48.4) We're gonna start with agentic fingerprints because I think that's a good place to start because it's one of the ones that is most often used as a potential security control for agentic traffic, which you know, if an AI agent is weaponized in anger at your application, it doesn't always identify itself nicely. Sometimes it does with the user agent, but it identifies itself in other ways. Can you explain some of those? Lee Ennis (02:13.592) Yeah, absolutely. So the idea is, as we all know, fingerprinting many, many different topics and different use cases. When it comes to AI though, the data is at large scale. So when an agent, either internally or externally in your system, is active, you need a way to identify it. We need a way to synthesize that data and kind of try and get an understanding of it. So if we look at internal what our own agents are doing, it's a very good way to backtrack and trace where things may have gone wrong, where they've gone right. And we've used it internally as a debugging tool or an understanding tool. We had that in our conversation. But externally, you could also use it to say, Hey, the patterns and the way this is being attacked look very familiar. The words it's using, the structure of the sentences can all help identify where it may be coming from from a model perspective or a family perspective. Lori MacVittie (03:03.234) Well, so one of the things I'm hearing is that fingerprinting we used to use fingerprinting as a identification of a specific device or user, right, something, an identity. And what you've been talking about is more like you're fingerprinting a workflow, right, or a process or a set of things. Yeah? Joel Moses (03:27.108) Or a behavior. Lori MacVittie Yeah. Behavior. Lee Ennis (03:29.611) Yeah, that behavior is a good way to characterize it. So we do originally, and that was the whole concept that we first started talking about is can we identify what this agent is or where it's coming by by its actions? And that is the traditional fingerprinting. If you think about it though in the more human sense, when you talk about when you go around fingerprinting, you're just looking for traces of things around the place. So you're going around with your blotter and your dust and seeing, Oh, there's fingerprints here, there's fingerprints there. That is the concept that we've kind of brought into agents because these autonomous agents move around and kind of disrupt your system in various different ways, being able to see those traces at scale in an observability tool is actually massively helpful for you understanding what's going on and what, how things might be happening. Joel Moses (04:16.761) Yeah. Lori MacVittie (04:17.08) So it's like crumbs, like I mean, so you know. Well, you know, let's assume that, you know, an agent is a cookie and there's a chocolate chip cookie and an oatmeal raisin cookie and a sugar cookie, and you're looking for the crumbs around, right, the network and the systems in order to identify which cookie it actually was Joel Moses (04:34.116) Mm-hmm. Lori MacVittie that made a mess. I love it. Lee Ennis (04:36.62) Absolutely. It's Joel Moses (04:37.711) Right. Lori MacVittie I love it. Lee Ennis very much a case of--and you can think of the cookies if there was hundreds of millions of them and then you're trying to understand do all the oatmeal raisins cause a problem or is it just a single one? Lori MacVittie/Joel Moses/Lee Ennis Ha ha ha. Joel Moses (04:47.92) In my opinion, yes. Lori MacVittie (04:50.69) Ha ha. Lee Ennis I deliberately picked oatmeal oatmeal raisin Joel Moses (04:54.136) Yes. Lee Ennis there. Apologies to anyone who's a fan. Lori MacVittie (04:54.46) I love oatmeal raisin. Okay, I'm just saying. I like that. That's, Joel Moses (05:00.89) So behaviorally Lori MacVittie they came to mind. Joel Moses though, these AI systems, especially AI agents, that are, for example, commonly either embedding their own browsers or driving a browser as if it were a user, these agents do have one property in common, and that's that they try to be in everything incredibly efficient. And so some of the things that you can see when an agent is actually using an API surface is that it's using it in an unnatural fashion. It's taking the shortest path to victory. Where a user doesn't necessarily do that because an API is hidden. It's being driven by other things, or it's filling a form, but it's ignoring certain portions of the form that are immaterial to the thing it's working on. Or it's following a cadence of accessing multiple tabs in a sequence that isn't normal for a user to do. So there are behavioral markers just because AI is AI, right? Lee Ennis (05:58.168) Yeah, absolutely. Those markers are exactly what you hone in on, right? When you're looking at large scale classification and identification of these models, those are the kind of things. The interesting thing is when you look at what it leaves out and what it puts in, you can actually distinguish between the AIs. Is this a Claude model? Is this an OpenAI model? Just based on the forms it fills out, the way it fills them out, the things it decides to skip and not skip, or generally pass through in model families and pass through even different learning systems. You can trace them all back to probably llama three point something Joel Moses (06:30.159) Mm-hmm. Lee Ennis at a certain stage. Joel Moses Well, having been involved in technologies related to anti-bot techniques, I can tell you that, you know, it's recognized that a lot of individual users have their own cadences. And it turns out that even different AI agents based on different models have unique cadences, just like humans would. So that's fascinating. Now there's some other applications of fingerprinting as well. Tell us a little bit about fingerprinting models. Why would one want to do that? Lee Ennis (07:00.654) Yeah, so fingerprinting models really comes back to understanding what it's capable of, what it's not capable of, and also being able to identify like where the source of this comes from, right? So if you're looking at the fingerprints of a model and we're able to identify an unexposed model, so a third party solution or something that we don't know the ins and outs of, understanding the actual the model behind it that sits behind it can be very powerful. Because it may be a model that we're not comfortable with. It may be a model that we don't think has the security procedures or has bias ingrained in it, or all of these other things that we need to do. Being able to read the outputs of these models, review the traces and see, actually, we see that 80% of models that show these similar traces have vulnerabilities in these type of areas is an extremely powerful tool that we can do before we implement models or before we start looking in our system to see where these models are running. Lori MacVittie (07:54.05) Or restricting Lee Ennis Yeah. Lori MacVittie what agents can use. And that's my immediate thought was Wow, every time the agent selects that model, it goes haywire, so we're not going to let it use that one anymore. So kind of fine-tuning optimizing, if you will, right, your environment to make sure that the agents are using the appropriate models and getting the right responses, or at least optimizing for best responses. Lee Ennis (08:19.03) Absolutely. Like what started as a security tool in our eyes--being able to audit and have all of this--turned into a debug and a self improving tool that we were Joel Moses (08:28.208) Hmm. Lee Ennis able to say, Hey, if we do this, then it leads to better results. Whenever we see something like this happening, it leads to something breaking or something incorrect. So how do we prevent what we see and how do we prevent patterns from going the wrong way? Joel Moses (08:41.026) Mm-hmm. Yeah, that's interesting. You mentioned debugging, and that's one of the things that's classically difficult about AI model usage. These are non-deterministic elements, and non-deterministic systems are really hard to debug because you can't connect A to B cleanly. There's a mid path that you can't observe. So talk about, for example, what you would use fingerprinting for in debugging. How would that happen? Lee Ennis (09:07.289) Yeah, it's actually one of the most powerful things that we've seen in order to get a reliable system that works continuously as expected. Generally it takes quite a bit of data, so you don't get it off the shelf. You need to actually run it and unfortunately break a few eggs. But once you have those eggs broken, it actually gets really, really powerful to be able to spot trends in the analysis. So you can cluster all of the runs or all of the workflows or all of the process traces that you see that, Hey, that was an undesirable outcome. What led to that? And they generally form into tree-like branches. And you can see them as you like visualize this into Hey, all of the paths that I didn't like or the outcomes I didn't like generally originated from a small number of nodes or decisions that the model made. How do we limit those decisions? And what you see is you end up removing 80-90% of the undesirable paths because you're able to single out individual things. It's the only real way we found to get substantial increases in our debugging through that other than blind prompting and saying, Hey, don't do this. Lori MacVittie (10:08.824) So Joel Moses Cool. Lori MacVittie we're using AI to identify to fingerprint AI and it, I... should I trust that? I mean because what you're talking about at scale, we're not doing this in real time. Nobody's doing it in real time. Like that's Lee Ennis No. Lori MacVittie just not gonna happen. You're sampling and then you're analyzing and you're finding the patterns, which is essentially this is ML, if you will, Lee Ennis (10:35.022) Yes. Lori MacVittie (10:35.586) probably. More predictive, so a little bit more trustworthy, but still you're sampling, and then we're saying, Well, we're kinda gonna AI the answer and say, That's better. So now we're, are we in an infinite loop? How do we debug the Lee Ennis/Joel Moses Ha ha ha. Lori MacVittie I'm just... Lee Ennis (10:55.504) No, it's a very fair point. I think everything becomes AI when you take a Lori MacVittie (10:59.263) Ha ha ha. Lee Ennis far enough step back. It's all a simulation. But I think you hit on what we're trying to get to there is Lee Ennis (11:08.887) the difference between traditional ML's deterministic capabilities versus non-deterministic. And non-deterministic is actually where a lot of the problems we see, right? Joel Moses Mm-hmm. Lee Ennis You can turn down your temperature as much as you want, but then the hardware comes into it. And then when the hardware matters more is how much wattage did you put through it? And doesn't matter what base points you're rounding to, it always has some level of it. So when you look and take a step back, the actual root cause is Hey, how do I get something that's going to be reliable for me? And that's turning back to traditional ML and using it to enhance the AI. Lori MacVittie Mm-hmm. Joel Moses (11:44.896) You know, that fits alongside another concept, especially in security, explainability. Explainability is very important to security analysts and unfortunately nondeterministic systems are really hard to explain. And so having a system that can fingerprint alongside a system that is non deterministic actually adds to the explainability for security people, doesn't it? Lori MacVittie Hmm. Lee Ennis (12:06.242) Yeah, massively. Both from Hey, I pen tested this, because I'm using AI now to pen test like everyone is ever since Mythos. How do I know what triggered it or what caused it to break and where that is? And then internally in my own tools, Hey, can I see where it went and use that as an audit trail? Whereas without any fingerprinting or that trace analysis, it's very hard to actually understand, Hey, I did the same thing twice. One time it broke and the next time it didn't. How do I fix this? Do I need to block all of this? And what are the downstream effects of doing something like that on a large scale project? Lori MacVittie (12:47.77) That's, it makes the compliance people very happy and Lee Ennis (12:50.617) Yeah. Lori MacVittie the governance peop-, I mean they're like, Yes, this is... But you do need that, especially you go back to that debugging, like I sent the same thing twice and I got vastly different results. Why? Like what was different? And then you find out, Oh, there was an extra word in the context somewhere that got, and you're like, Oh, okay. Lori MacVittie (13:14.134) Yeah, I mean it can be that simple, right? Joel Moses Mm-hmm. Lori MacVittie At least it starts giving you an idea of why you're getting vastly different answers all the time. Lee Ennis (13:18.829) Yeah, absolutely. Absolutely. And you can do things, so we've done things like allowing models to have variable inputs and outputs, so just natural language text. However, how much of that natural and deviation does the process need? So you can do things like similarity scoring on the entire text and actually summarizing that and only keeping Joel Moses (13:39.173) Mm-hmm. Lee Ennis that as is. So you end up eventually getting back to the single trace node of saying, Hey, don't say anything about this. This is all I really need. Cut that out. And that's how you end up with your debugging. You filter what's not needed anymore to get to a more deterministic system. Joel Moses (13:53.947) So fingerprinting wise, we've got a lot of different things in our toolkit. We've got everything from, you know, JA4 TLS fingerprinting all the way through user agent activity fingerprinting, model fingerprinting, et cetera. In AI infrastructure though, is there something within AI infrastructure that you think we aren't fingerprinting enough? Lori MacVittie Mmm. Lee Ennis (14:14.484) Ooo, that's interesting. That's a good question. Lori MacVittie (14:15.862) Ah ha ha. Lee Ennis So one thing that I find super interesting, and that is going to become more and more of a use case is the tie between the harness or the Lee Ennis (14:27.906) agentic system that's being used--the model that's being used--and then the hardware. So the hardware is becoming more and more of a question. And so usually when we're fingerprinting or when we're looking at these traces, having the tie back to all of those is really, really important because what we're seeing more and more of is the deviation caused by all of those things can be extreme in any use case. Joel Moses Yeah. Lee Ennis And so when you fingerprint, you're trying to limit and gather in as much variables as possible. But if I use an AWS or some cloud provider, I don't really know what hardware I'm necessarily getting. I'm getting, Joel Moses Right. Lee Ennis I say, "Hey, I want this instance," but is it the same instance? Does it have a different wattage? If I'm using OpenAI, I have no idea what hardware they're running on. And so even that at a zero temperature can potentially change your answers. And so those are the kind of things that you'll see, I think we will start seeing more when sovereign AI starts coming into and people have more control over everything. How do they get a complete control and view of their entire system? Joel Moses (15:30.162) That's fascinating. Lori MacVittie Well, back to hardware. Joel Moses That, from the client side anti-bot space, that's actually a really common tactic Lee Ennis (15:35.451) Yeah. Joel Moses to profile, for example, the GPU on a client system for its performance and characteristics and use that as a mechanism for determining a cluster of that Lee Ennis Yep. Joel Moses particular type of hardware. Never thought about it from the hardware perspective, but you're right, in the era of sovereign AI, it really does matter where you run and on what. Lee Ennis (15:54.672) Absolutely. Lori MacVittie (15:55.938) Yeah, we're right back to hardware. I feel like we've come full circle Lee Ennis (15:58.063) Ha ha. Joel Moses (15:58.609) Ha ha. We really have, haven't we? Lori MacVittie in like 30 years. Yeah, we're back to hardware matters again. So we have to build more hardware so we can run everything and then we can consume it all and it's a vicious, vicious cycle. But identifying hardware has been huge for the past probably 10, 15 years, right? Lori MacVittie (16:18.786) Between, you know, CPU IDs and hardware IDs and you know, mapping everything together and it does become important and can that factor into the answers? Like, Yeah, I was running without a GPU, so I gave you a lesser answer. You know, I Joel Moses (16:34.415) Yeah. Lori MacVittie This is possible these days, right? The hardware can influence, you know, how good of an answer you get. Lee Ennis (16:43.01) Yeah, absolutely. Lori MacVittie (16:45.302) See? There you go. Joel Moses (16:45.445) Yeah. You know, my takeaway in all of this is that, you know, there's a huge difference between the way these AI systems interoperate and behave. You know, legacy bots often would give themselves away with weird IP addresses, and an agentic AI Lori MacVittie Ha ha ha. Joel Moses gives itself away because it's actually the only thing that ever read your terms of service page. Lee Ennis (17:06.096) Ha ha ha. Yeah. Lori MacVittie (17:08.95) Ha ha ha, it does, yeah. Joel Moses (17:09.018) You know, it strikes me that, you know, behaviorally speaking, a lot of these AI systems, they're really bad poker players. They have tells. And that's a good thing. They have tells so that we can determine exactly what it is that they are doing, what platforms they're running on top of, and make use of that information to either design controls or to improve them, or debug them, which is an incredibly important thing to be doing. Lori? Lee Ennis (17:36.975) Yeah. Lori MacVittie (17:38.804) Well I was gonna ask Lee, you know, so what would you leave people with? What would you tell, you know, listeners about agentic fingerprints? You know, what should they do next? What should they look for? What should they be doing? Or should they just, I don't know, turn this off and leave? I don't know. Lee Ennis (17:54.958) Well, Lori MacVittie Give up. Lee Ennis that's one option. I don't Lori MacVittie Ha ha ha. Lee Ennis think I'd recommend it. No the place to start is really just trying to observe as much of the data as possible. And that can get really difficult, right? Because you have Joel Moses Yeah. Lee Ennis like how do you parse all that data? How do you understand? And then you may have different model families or different sources having different structures of their API where it gets very confusing and all of these kind of things. You can't do a lot of it without having sufficient data. And so where I would lean into is turning the toggles on and trying to save a little bit of data at a time, trying to get an understanding. And then do that within your safe sandbox. Build within your safe sandbox, store the data, apply the common clustering and various techniques, and then just go in and read the traces. Often they're very easily understandable. They're often it's a case of, Oh yeah, it makes perfect sense that the model thought it was okay to do this because it reasoned through it. It explained to me exactly what's going on. It's the only kind of use case in the world where they write down everything they're going to do before they do it. So if you can capture that, Lori MacVittie Ha ha. Joel Moses (19:01.713) True. Lee Ennis like if humans did that, the world would be a very safe place, I think, if we had to do something like that. Lori MacVittie (19:09.718) Ha ha ha ha, that is so true. My takeaway is gonna be observability. Right? I mean, we've kind of been harping on observability for years about how important it's going to be, and I think we've hit that point because Lee Ennis Yeah. Lori MacVittie so much of security, of debugging, of compliance, of performance, of just automation today is really you gotta have observability and be collecting all this data. Right? Before you can analyze it, you have to collect it. So yeah, turn on all those toggles. Make sure things are, you know, sending data somewhere and then get it together. So observability could not be more important today, especially in the AI realm. Lee Ennis (19:48.012) Yeah, your security analysts will thank you for it, I promise. Joel Moses Yeah. Lori MacVittie (19:50.742) That's I live for that is the Thank Yous from the security analyst instead of the No, you're you can't do that. Which is what I usually hear. So I'm gonna wrap that up before I get told no again. Ha ha. So we're gonna wrap up Pop Goes the Stack this week. Please hit subscribe because the next trend will arrive with a model card, a dozen caveats, and exactly zero patience for your existing tooling.