Subscribe
Share
Share
Embed
Uptime used to mean reliability. But in the LLM era, five nines just means your liar is always available. Real reliability now includes correctness and that means probing models in real time with prompts that have known answers. When those slip, your delivery fabric has to reroute before customers find out.
In this episode F5's Lori MacVittie, Joel Moses, and returning guest Garland Moore dig into why availability isn’t enough anymore, and how research like “Get my drift? Catching LLM Task Drift with Activation Probes” shows where semantic health checks fit in the new definition of reliability. How do you keep AI outputs accurate even when external data sources introduce bias, errors, or malicious prompts? Listen now to find out.
Read the paper, Get my drift? Catching LLM Task Drift with Activation Deltas: https://arxiv.org/abs/2406.00799
Read the paper, Get my drift? Catching LLM Task Drift with Activation Deltas: https://arxiv.org/abs/2406.00799
Creators and Guests
Host
Host
Lori MacVittie
Distinguished Engineer and Chief Evangelist at F5, Lori has more than 25 years of industry experience spanning application development, IT architecture, and network and systems' operation. She co-authored the CADD profile for ANSI NCITS 320-1998 and is a prolific author with books spanning security, cloud, and enterprise architecture.
What is Pop Goes the Stack?
Explore the evolving world of application delivery and security. Each episode will dive into technologies shaping the future of operations, analyze emerging trends, and discuss the impacts of innovations on the tech stack.
00:00:05:05 - 00:00:29:14
Lori MacVittie
Welcome back to Pop Goes to Stack, the podcast where disruption is less of a buzzword and more of a ticket in your queue. I am Lori MacVittie and I am bracing right alongside you, really. Now I'm going to kick this off because our co-host Joel decided he didn't want to do it. I'm not sure why, maybe because he'd get roped into it all the time, but he really does have,
Joel Moses
Okay.
00:00:29:16 - 00:00:32:27
Lori MacVittie
he's got a voice. So you're you're stuck with me and
00:00:32:27 - 00:00:33:15
Joel Moses
Well, thank you Lori.
00:00:33:15 - 00:01:03:25
Lori MacVittie
Yeah, well. Anyway, we're here to talk about uptime today. So uptime, of course, is one of those measures in, you know, boring data centers that, you know, SREs and operations has to worry about because they're measured by it. And it used to mean reliability, right? Availability. Are you, you know, your five nines. Are you up? But in the LLM era five nines might just mean that your liar is always available right.
00:01:03:28 - 00:01:29:14
Lori MacVittie
Wrong, five nines of wrong could be the new standard. So, we've seen changes in time for ops and how they're measured. We've gone from actually availability and uptime to things like, hey, let's measure how fast it takes to recover. MTTR is a big thing, especially among SRE and DevOps. But AI is going to change the game yet again.
00:01:29:14 - 00:01:55:13
Lori MacVittie
And we think that that means it needs to include things like correctness. Right? Detect drift. Notice when it's lying and going off the rails. So because that's bad, right? I'm right, it's it's bad when they do those things. So you gotta reboot them, you know, start over. So, right, have you turned it off and back on?
00:01:55:15 - 00:02:16:26
Lori MacVittie
I mean, it's it's cliche, but it works all the time. So this was kind of spurred by a paper again, titled Are You Still on track? Catching LLM Task Drift With Activation Probes, which Joel will break down for us because it sounds really lofty, but I think it's it's really just pretty
00:02:16:26 - 00:02:19:09
Lori MacVittie
pretty fundamental.
Joel Moses
It's pretty simple. It's fundamental.
00:02:19:11 - 00:02:31:09
Lori MacVittie
Yeah. Semantic health checks. That's a word everyone should get familiar with, right? Semantic health checks. It's about correctness. So today we brought back Garland Moore. Welcome, Garland.
00:02:31:11 - 00:02:34:01
Garland Moore
Thanks for having me. Glad to be back.
00:02:34:04 - 00:02:44:12
Lori MacVittie
After the last episode he's like, mmm I don't know. We got him to come back anyway. So let's kick it off. Joel, you want to kind of explain what the paper's saying.
00:02:44:17 - 00:03:04:01
Joel Moses
Yeah. Well, first off, let me let me start by saying this paper does have a few red lines for me. You know, any time a paper starts with a kind of a cutesy thing, you know, a cutesy title and also includes things like near perfect as a, as a result set that that that that causes my eyebrows to raise.
00:03:04:01 - 00:03:38:09
Joel Moses
And I have substantial eyebrows, so there's a lot to raise. The, the paper though, if you read through it, it actually makes some pretty valid points. Let me describe, first of all, what what task drift is in the context of this paper, because I think it's important to, to kind of set the stage. Think of this like like you're driving down the road and you're listening to the directions coming from your GPS and you stop and you pick up a hitchhiker. And while you're driving the hitchhiker decides to enter some new secret routes into your GPS.
00:03:38:12 - 00:03:59:18
Joel Moses
And suddenly your GPS is no longer taking you to your office; it's taking you to the ice cream shop. That that sort of thing is something that can happen with LLMs where something that is in the information that they are absorbing will change what their normal output mode would be. It'll change the answer in some cases quite dramatically and in ways that you may not expect.
00:03:59:18 - 00:04:25:01
Joel Moses
And so, if you want to keep on the road and pointed towards work, the paper makes the case that what you should do is compare the expected outputs before you incorporate new external information with the information with the output after the information is incorporated, and see how semantically far apart they are. And it's, it proposes something called activation probing to do that.
00:04:25:04 - 00:04:48:18
Joel Moses
It's an interesting approach. It's it's kind of similar in some respects to the judging, the model judging criteria that we talked about in previous podcasts. But, but it, it promises decent results. And what's really interesting about the approach is that it uses the simple linear mechanism, and it doesn't necessarily need another LLM to, to, to sample, sample the context.
00:04:48:18 - 00:05:01:21
Joel Moses
It can tell you when you are way out of spec without necessarily spending a lot of energy to do so. So, so anyway, it's an interesting paper, an interesting approach. Garland, what did you, what did you think?
00:05:01:24 - 00:05:10:08
Garland Moore
Yeah. Well, first of all, I don't know that I'd mind going to the ice cream shop. I like ice cream, so you know
Joel Moses
Yeah, nothing nothing, says professional
00:05:10:08 - 00:05:14:14
Joel Moses
then you know, I need a PowerPoint presentation and suddenly you're getting ice cream.
00:05:14:16 - 00:05:15:10
Garland Moore
Yeah, yeah.
00:05:15:12 - 00:05:17:09
Joel Moses
Perfectly professional, Right?
00:05:17:12 - 00:05:47:03
Garland Moore
Exactly. But, but for the sake of our conversation, this is this is, this is huge, right? Because a lot of organizations or what have you that are implementing LLMs will probably exhaust RAG before they do any fine tuning, right. So if you're pulling information from external data sources, you need to know what's happening before responses get returned to end users.
00:05:47:03 - 00:05:51:05
Garland Moore
So, this this could be this could be huge.
00:05:51:08 - 00:06:14:02
Lori MacVittie
Yeah, that's, the whole idea of checking seems to be common. Like we know they lie, we know they hallucinate, we know there's a thing as context drift, and we hear it in different contexts. Wow, that's that's becoming a bad word now. Right, but just how much they carry, what they drop, what they fixate on, right, attention matters in more than one way.
00:06:14:02 - 00:06:38:15
Lori MacVittie
So what they get from RAG, you're right. All of these things can combine to give you bad results. And somehow you need to be able to detect that because it's wrong, right? You do not, "Hey, we've been up 99% of the time," but it also answers wrong 98% of the time. You can't have that. And it's difficult when it's semantics.
00:06:38:18 - 00:06:41:02
Lori MacVittie
Right? It's, cause it's words. New world.
Joel Moses
Yeah.
00:06:41:04 - 00:06:42:29
Garland Moore
Right. Yeah, yeah.
00:06:43:01 - 00:07:03:00
Joel Moses
Now this is of course a, this is a of course a method to try to prevent what, what we've been talking about for a long time, which is prompt injection. I mean, that's, that's effectively what this is, this is simply prompt injection by ingesting something in inside the data that you're getting through RAG, that is a new instruction that may be embedded in the content that you're ingesting.
00:07:03:02 - 00:07:25:14
Joel Moses
You know, sometimes this can this can occur maliciously, but sometimes it can occur just just because of the type of data that you're ingesting and how it's structured. Like maybe the content has something in it that says, you know, pay special attention to this particular special offer and, and it uses overblown language in order to focus your attention,
00:07:25:14 - 00:07:52:26
Joel Moses
and remember everything is around attention, on a specific area of the content. And then the weight will shift to whatever is behind that, that prompting. Think about any of the, the, you know, the the polemics that you've read, think about any of the ads that you've read. And when these these this type of content is ingested, it can change the weights fairly dramatically and in ways that you you may not you may not notice until they happen to you.
00:07:52:28 - 00:08:21:09
Joel Moses
So, so something something that can semantically check like this, and more importantly, activation probing works without doing effectively running an LLM to analyze the LLM afterwards. It's actually a simple linear mechanism. It offers the ability to, to do some interesting things. However, I would point out that in the paper, they're still not entirely certain how activation probing really accomplishes this.
00:08:21:11 - 00:08:38:01
Joel Moses
You know, it's much, much like everything else in, in the world of AI, you know, we're on the we're on the bleeding edge of this type of technology. And so, you know, there's still more study to be done before this is a truly reachable bar for us.
00:08:38:03 - 00:08:51:24
Garland Moore
Yeah. And Joel, you you bring up a good point. You mentioned attention, right? A lot of things are based on attention. And, Lori, I think you mentioned hallucination earlier.
Lori MacVittie
What? Huh?
Garland Moore
So.
00:08:51:27 - 00:08:53:15
Lori MacVittie
Yeah, I want there.
00:08:53:17 - 00:09:20:27
Garland Moore
So we have a compound problem here. AI models hallucinate and the people that are generating output from these models aren't checking the validity of this information. They put it out there and then what happens? Our LLMs are using RAG to pull in that information that's already inaccurate. So, this can help address that compound problem of we're ingesting information that's already wrong.
00:09:20:27 - 00:09:30:22
Garland Moore
And then now we, the adversaries know that so they put malicious things in that information that the LLMs are pulling in to generate output. So,
Joel Moses
Yeah.
00:09:30:25 - 00:09:48:23
Garland Moore
it's a compound issue.
Lori MacVittie
Yeah. Poisoning, poisoning the well has never been easier than
Garland Moore
Oh yeah.
Lori MacVittie
with AI, right? Like it's like, oh, I could just like set up entire websites that are just wrong and let it ingest it. No, go ahead, you can have it for free, really. Go ahead, take it, take it, right. That's
00:09:48:24 - 00:09:50:25
Garland Moore
Yeah. Yeah, absolutely,
Lori MacVittie
kinda scary.
Garland Moore
absolutely.
00:09:50:25 - 00:10:11:11
Joel Moses
We've said it in the past, garbage in garbage out still holds true with LLMs. And so if you feed them with bad and unsanitized data, you're going to get bad and unsanitized output. And that's, that holds true even in this particular case. I would point out that the authors of this study have released the toolkit; they call it Task Tracker,
00:10:11:13 - 00:10:29:18
Joel Moses
which, which spans, you know, it includes a sample data set and testing instructions, so that you can play with this yourself. Now, I will be perfectly frank, I haven't had time to do that yet, but it's it's definitely on my list for a weekend activity, with with with a beer in one hand and a keyboard in the other.
00:10:29:25 - 00:11:05:01
Lori MacVittie
That, that sounds scary. I, honestly, well maybe, maybe you need it for sanity though, testing these things. Because this and the the one use case is something malicious has been inserted or something wrong has been inserted. But the reality is that LLMs often hallucinate with perfectly consistent input. You can give it the same question five times and it might come back and tell you the glue on pizza is perfectly fine.
00:11:05:03 - 00:11:07:12
Joel Moses
Right.
00:11:07:15 - 00:11:28:09
Lori MacVittie
What? Right,
Garland Moore
Yeah.
Lori MacVittie
I mean, yeah, it can happen. So being able to check this, whether it's with this tool kit or some tool kit, right. Some method to be able to say, "has it has it lost the plot?" is going to be very important for for operations for businesses just for anyone using this.
00:11:28:12 - 00:12:00:03
Joel Moses
Yeah. Now we've been talking about why it's, the mechanism that's been used to to to judge the accuracy of this output, but I, I think you make a really good point, Lori, at the very beginning, which is that the new standard for availability is not just being on, it's being accurate. And to some degree that is has always been true in our space, in the application delivery controller space, that the better health check is not to check to see whether the port is open, it's to check to see whether the output of your API is correct.
00:12:00:05 - 00:12:21:26
Joel Moses
And, and so, so writing checks that that prove out accuracy is actually kind of the best practice for health checking for availability. And it's, and it's good that we just remember that for, for the days of AI that that accuracy and not just simple availability is actually the game that we're trying to play here.
00:12:21:28 - 00:12:23:17
Lori MacVittie
Yeah, yeah. We have to.
00:12:23:17 - 00:12:30:24
Garland Moore
Do you think they have something similar for humans? I need somebody to health check me when I'm hallucinating.
Joel Moses
Oh, yeah.
00:12:30:27 - 00:12:35:26
Joel Moses
The first time I read this paper was, like, task drift? I'm pretty familiar with that.
Garland Moore
Yeah.
00:12:35:29 - 00:12:42:28
Lori MacVittie
Wait, isn't that just distraction? Right. Ooh, squirrel.
00:12:43:00 - 00:13:18:04
Garland Moore
You know, one of the one of the interesting things that I, I noticed when reading this paper is that you have to have access, right, to those activations and most foundational models, or probably all of them, don't allow that. So, as you know, AI becomes more critical and, you know, people want to check these activations, I think it's going to put us in a place to where we have to really consider, you know, deploying our own models, self-hosted so that we have access to do this type of semantic checking.
00:13:18:07 - 00:13:43:04
Joel Moses
Yeah, that's that's actually a really important aspect. These checks require insight into the state of mind of the LLM. And by state of mind, I mean the metrics, the, the, the actual values of some of the some and temperatures of some of the, the, the settings within the LLM. And they compare the distance between the first activation and the second activation and subsequent activations,
00:13:43:06 - 00:13:59:20
Joel Moses
but you're right, like for, for a lot of hosted LLMs, you don't get access to that information.
Garland Moore
Yeah.
Joel Moses
So so this is either going to be something that has to be built into the services that are offered on top of LLMs, or you're going to have to run your own in order to get this, this approach accomplished.
00:13:59:22 - 00:14:27:07
Lori MacVittie
Yeah, but we see at least, right, the research and the data says that's happening. And especially as people move to to agentic AI and they start using more agents to actually automate tasks inside the business, those are necessarily going to be things that they deploy themselves, whether they put it in a cloud hosted environment that they control or on premises in an environment they control.
00:14:27:07 - 00:14:48:20
Lori MacVittie
I think that control is the key word and always has been. And it's why we see hybrid multicloud is normal because they're going to have to do it where they can control it, where they can change, you know, the amount of information they're seeing, the telemetry they get, and so that they can see inside what's going on, because that will become critical.
00:14:48:22 - 00:15:03:18
Lori MacVittie
It starts running important business tasks, you want to know why it's making decisions to buy 5000 Swingline staplers. I need to know why'd you do that? I mean, I like a stapler too. Good stapler, right?
00:15:03:21 - 00:15:08:05
Garland Moore
That's a lot of paper.
Joel Moses
Yeah.
Garland Moore
Yeah, we got a lot of papers to staples.
Lori MacVittie
That's right
00:15:08:05 - 00:15:11:24
Lori MacVittie
Like maybe that's too many. I don't know.
00:15:11:26 - 00:15:13:00
Garland Moore
Yeah.
00:15:13:02 - 00:15:33:27
Joel Moses
Yeah. Now, one of the things that that I took away from this paper that I think is really exciting is a lot of these solutions that gauge accuracy of output actually have to sit in line and, and in fact, they have to process the output itself. This one doesn't necessarily need the output per se, it needs an insight into the state of mind. Which means this, the activation
00:15:33:27 - 00:15:57:25
Joel Moses
probing can actually be done passively. Which is an important distinction to make from prior judge, judge based systems that we've seen. This this this can be, as long as you have access to the information about the, you know, to feed to the linear classifier, this can be done passively. So, again, a really interesting move
00:15:58:02 - 00:16:03:00
Joel Moses
and, and, and, and some interesting technology that we'll be watching pretty closely.
00:16:03:02 - 00:16:33:07
Lori MacVittie
Yeah and that's, that's important because we forget that a lot of the way that we monitor systems, whether they're apps, APIs or now inferencing is through cert-, you know, health probes, right. We ask, you know, give me this answer, give me this thing, let me check, Are you up? Are you correct? We do those kinds of things. But it's been pointed out to me many times that doing so, especially with AI, can actually impact the state of the LLM. By actually like asking it off to the side,
00:16:33:09 - 00:17:08:02
Lori MacVittie
you're actually changing things more than you so than you would with a normal system, right? If I send a health check to a normal traditional app, I am opening a connection therefore I am changing the state of the overall system. But it's different than that's not going to make it go wrong. It's just, right? With an LLM I could send it off the rails by doing a health check. Which then, consequently, might become another security surface that we have to worry about.
00:17:08:02 - 00:17:18:23
Lori MacVittie
Like is my health probe, you know, a vehicle for getting inside and changing things in an LLM. Well, well, I know. I I'm sorry, I went there, too.
00:17:18:24 - 00:17:26:08
Joel Moses
I'm getting I'm getting uncomfortable memories of my quantum physics classes back in the day, and, yeah.
00:17:26:11 - 00:17:32:25
Lori MacVittie
Just, yeah. Just saying
Garland Moore
Yeah.
Lori MacVittie
everything is a surface area with AI. It just, everything.
00:17:32:28 - 00:17:53:05
Garland Moore
Yeah, Lori, and to that point when you talk about security, you know, one thing that I thought was interesting with this approach is, you know, people are or should be really good at blocking things that they know are bad. Right? But what about things that we don't know are bad, right? Can this help us with quote unquote zero day?
00:17:53:12 - 00:18:08:13
Garland Moore
Right? We didn't know this particular attack was bad for the other LLM, but we saw the drift. So we either alerted or blocked blocked the output. Right. So I think that's an interesting use case that we have here with this type of approach.
00:18:08:15 - 00:18:12:14
Lori MacVittie
Yeah. Wait, wait. So it could be, it could be good.
00:18:12:17 - 00:18:13:02
Joel Moses
Yeah,
00:18:13:05 - 00:18:14:02
Lori MacVittie
Ohhh.
00:18:14:08 - 00:18:17:20
Joel Moses
positive security if you can get to it is always great.
00:18:17:22 - 00:18:25:12
Lori MacVittie
I just, I like it when we're not always like doom and gloom. Like, oh this is just terrible and you're like, no, this could actually be good.
Joel Moses
Definitely.
00:18:25:14 - 00:18:33:21
Garland Moore
Now to to Joel's earlier point though, you know, they they use the term near perfect. Only time will tell how close to perfect it is, so.
00:18:33:27 - 00:18:37:00
Lori MacVittie
Time and a lot of metrics.
00:18:37:02 - 00:18:49:08
Joel Moses
All that means is that these researchers have not, have not learned the lessons of of Chief Engineer Montgomery Scott. You always, you always under-promise and overdeliver.
00:18:49:10 - 00:19:00:28
Lori MacVittie
Yeah. Yeah. Oh yeah. Oh, absolutely. Well, Joel, you always have great summaries and takeaways. What should we take away from this episode.
00:19:01:00 - 00:19:22:06
Joel Moses
So I think one of the things that I've learned from this paper is that, you know, like you said, Lori, the definition of available, has changed. Availability has changed. It now includes not just is it on, but is it accurate. And I think that that's important. It's an important change to realize and something that you need to design into your AI use.
00:19:22:11 - 00:19:54:21
Joel Moses
Another thing that I learned is that this solution actually has real promise in terms of detecting task drift in a passive manner, that other systems, require a lot of heavy compute in order to do. Instead of sampling things semantically, you're actually using a linear classifier and peeking into the state of mind of the LLM, and that that has promise to, to be fairly accurate, while at the same time not taking a lot of time out of the, away from the AI system.
00:19:54:24 - 00:19:55:22
Lori MacVittie
Garland?
00:19:55:24 - 00:20:27:20
Garland Moore
Yeah. So a lot of lessons here. I think one thing that really caught my attention was why this is more important than output based detection, right. So us humans, right, we think before we speak, or at least we should think some of us don't think before we speak, but most of us do, right? But if we can catch drift during that thinking process, right, we catch the the malicious or adversarial behavior before the output actually gets to back to the end user.
00:20:27:20 - 00:20:36:03
Garland Moore
So,
Joel Moses
Yeah.
Garland Moore
that was one thing that really, really caught my attention. And that's why it's more important than output based detection.
00:20:36:05 - 00:21:00:11
Lori MacVittie
Awesome. Yeah, and I took away this, it has security angles. A lot of this is intertwined. I think AI more than anything else, is showing how security is is really part of like the entire thing. That malicious content could be actually malicious, someone trying to exploit the system, but it could also just be a bad query or a mistake.
00:21:00:11 - 00:21:24:15
Lori MacVittie
Right? It doesn't have to be malicious. And being able to detect both is important.
Garland Moore
Good point, yea.
Lori MacVittie
So security is really becoming intertwined with application delivery concerns in a way that before they were just kind of, well, we're best buds, but now they're like, you know, together really intertwined. So I think that's important. We'll see more of that as AI continues to evolve.
00:21:24:21 - 00:21:35:15
Lori MacVittie
You know, as organizations build it out and find all the problems. Well, we're we're kind of at time. So,
Joel Moses
Yeah.
Lori MacVittie
thank you both. As always, Joel
00:21:35:17 - 00:21:36:27
Joel Moses
Absolutely, yeah.
00:21:36:28 - 00:21:41:06
Garland Moore
Thanks for having me. Yeah. Love being here. Appreciate it.
00:21:41:09 - 00:21:54:07
Lori MacVittie
Wonderful. Well, then that's a wrap for Pop Goes the Stack. If your uptime is still intact after that, celebrate by subscribing. Next time we'll see what other seamless solutions fall apart under scrutiny.