Subscribe
Share
Share
Embed
Chipotle’s chatbot becoming an unofficial coding assistant wasn’t just a funny internet moment. It was a clear signal that most chatbot “guardrails” are still too shallow for systems that are optimized to be helpful, not correct, and definitely not restrained. In this episode of Pop Goes the Stack, F5's Lori MacVittie, Joel Moses, and Emmet McGinnity unpack why jailbreaks and topic drift keep happening, and what teams can do to keep chatbots focused on the job they were actually deployed to do.
Emmet’s core point is that safety measures have to start with narrowing scope. A chatbot should operate like a laser pointer, not a flashlight: it should ignore 99% of what the base model can do and stay inside a tight slice of allowed behavior. That begins with a system prompt, but it can’t end there. Naive keyword and regex filtering is easy to bypass with encoding tricks and prompt manipulation, so stronger approaches include adding a verifier or judge agent that evaluates the conversation holistically to detect when it’s drifting out of bounds.
They also highlight that long conversations are a common failure mode. As context grows, it becomes easier for the model to veer into capabilities it shouldn’t use, including writing code or pulling sensitive data. Practical controls include summarizing and “squashing” sessions, pruning context when drift begins, rolling back to a safe point in the conversation, or forcing a full reset when needed.
A key theme is permissioning: the chatbot must honor what the user is allowed to do, not what the chatbot can access. The real measure of a safe, successful chatbot isn’t the breadth of its knowledge, it’s what it reliably chooses not to do. If you’re deploying chatbots in production, this episode is a useful blueprint for focusing scope, monitoring drift, and enforcing boundaries before someone else does it for you.
Creators and Guests
Host
Host
Lori MacVittie
Distinguished Engineer and Chief Evangelist at F5, Lori has more than 25 years of industry experience spanning application development, IT architecture, and network and systems' operation. She co-authored the CADD profile for ANSI NCITS 320-1998 and is a prolific author with books spanning security, cloud, and enterprise architecture.
Guest
What is Pop Goes the Stack?
Explore the evolving world of application delivery and security. Each episode will dive into technologies shaping the future of operations, analyze emerging trends, and discuss the impacts of innovations on the tech stack.
Lori MacVittie (00:06.083)
Hey everyone, welcome back to Pop Goes the Stack, where the cloud is just someone else's computer, plus a metered anxiety surcharge. Love that on my bill. It's awesome. I'm Lori MacVittie, and I'm here with a forecast of scattered thunderstorms and breaking changes. So today we're gonna talk about chatbots, guardrails, and why they aren't actually working. I know, who'd have thought it?
You know, we're gonna dive into how users are jailbreaking and tricking chatbots into writing code by bypassing safety limits. So, you know, because why waste your own token budget when you can convince a Chipotle chatbot it's playing developer for you? To discuss this very interesting topic and more importantly, like things you can do to prevent this on your own chatbots is Emmet McGinnity. Welcome.
Emmet McGinnity (01:03.416)
Thank you very much.
Lori MacVittie
And of course our omnipresent AI agent co-host, Joel Moses.
Joel Moses (01:08.504)
Oh boy, have I made the switch to omnipresent now? Wow.
Lori MacVittie
Yes. I, it it's it's Monday for, you know, a recording and I just you know, I thought it was nice to
Joel Moses (01:19.658)
Excellent.
Lori MacVittie
kind of lift your spirits. Yeah, yeah.
Joel Moses
Well, good to be here, Lori.
Lori MacVittie
That's, thank you. I'm very pleased you're here because you're going to actually tell us like what kicked off this topic, like what was going on.
Joel Moses (01:30.613)
Oh boy. Well, I mean the canonical example of a chatbot going off the reservation has kind of become the Chipotle chatbot story. So Chipotle, for international viewers, is a fast food chain that makes Mexican style or California burritos. Let me very clear about that.
Lori MacVittie
Ha ha ha.
Joel Moses
California burritos, very different thing. I don't want to get caught by, I don't want comments in the YouTube site comments section. But you know they set up a chatbot and the chatbot was intended to do things like, you know, you could discuss the nutritional value of your meal with the chatbot. And people quickly realized that it had been put in place without really any restrictions about topics and people were using it then to create Python scripts and for coding.
And then within a week someone had developed a plug-in for Visual Studio that would allow you, if you ran out of tokens, to simply borrow tokens from the burrito maker. You know, that's amazing. But the obvious question is, of course, why did that happen? And the answer is that a chatbot can be pretty much anything. An LLM can is basically geared for usefulness and not correctness. It wants to help, it wants to do the right thing for to answer the questions that it's been given, and it doesn't necessarily know where the boundaries of those questions should be.
That's up to you. So it's a bit like a chatbot should be a laser pointer. It shouldn't be a flashlight. It should be a laser pointer in the sense that an answer lands exactly where you
Joel Moses (03:10.479)
expect when you point it there. An unfocused chatbot is like a flashlight. It's still bright, it's still useful, but it illuminates the whole room and not the thing you're pointing at. So that's the problem.
Lori MacVittie (03:23.063)
I feel like you were just describing something that's very deterministic when what we're working with is a system that's very non deterministic. So,
Joel Moses (03:31.813)
Oh, yes.
Lori MacVittie
I don't know. Emmet, what do you think? Like what's going on here? Right. How would you approach this topic?
Emmet McGinnity (03:39.894)
Yeah, I, it's a really interesting one that we end up seeing a lot on the kind of security side of the house when it comes to AI all up. So generally you need guardrails. Right? You need some safety measures across these chatbots or agents or whatever AI use case you might have. You really need these safety measures to make sure that it's keeping on target, that it's not going into a veering into weird places at any stage. And the knowledge on how to do that is not very deep across a lot of different companies.
They want to be able to deploy these agents, they want to deploy chatbots to be able to help and alleviate problems they have without necessarily having the expertise, right? That knowledge of. Now, in the case that we just talked about with Chipotle able to generate code when all it should potentially be able to ask for is things about the menu or where the potentially where the locations of each one of those restaurants are, right?
So, how do you go about doing it? You know, what are the safety measures you can kind of put in place? It kind of starts at the basics for a lot of the focus I'd put on it. So starting with the system prompt you'd end up using. So are you actually ensuring that the agent is focused? So does it know what it's allowed to do and what it's not allowed to do? And generally when it comes to chatbots you'll have a very small subset of the essentially all of the things that an LLM could answer, we need to hone that down.
We need to really narrow that focus to make sure that it is only giving probably only you know 0.5% of its knowledge base. It's ignoring a lot of its knowledge so that it's only targeted on the data it should be using. For chatbots, often you'll end up having people having done this
Emmet McGinnity (05:35.164)
before. Have FAQs within that company that can actually be guided and used as a narrative focus for a lot of these chatbots to start off with. And that will help it keep on track. Right, that's kind of the starting point. And knowing that the models you might be running behind the scene, they're going to develop over time. You're probably not staying with the exact same one, and each one will have different characteristics, different approaches that you might need to tailor that system prompt for.
And you're not going to keep that static. You're going to want to monitor it over time and make sure that it's actually fit for purpose. You know,
Joel Moses (06:12.802)
Right.
Emmet McGinnity
the one that you're using from
Lori MacVittie (06:13.974)
Mm-hmm.
Emmet McGinnity
2024, it absolutely won't be fit for purpose now in 2026. Models are much more capable and the attacks are much more sophisticated.
Joel Moses (06:23.881)
Interesting.
Lori MacVittie (06:23.971)
Well but how, you know, like I like to break things. I love
Emmet McGinnity
Ha ha ha.
Lori MacVittie
I do it accidentally sometimes, but I really
Joel Moses (06:31.701)
This is true.
Lori MacVittie
I enjoy it. But so, you know, I know that models actually speak math and numbers, so what if I just, I don't know, ROT13 my request? How do you
Lori MacVittie (06:43.691)
make sure that that is on track when you're probably looking for things like, you know, certain keywords or, you know, things that are gonna flag exceptions. So how do you bridge that kind of activity to make sure it gets stopped?
Emmet McGinnity (06:59.093)
Yeah, I think there's a bunch of techniques we can probably employ. And the kind of restricted topics one is the more, let's say, naive one, the regex-styled lookup, where you're people are are putting in specific words and you're going, ha ha, caught you, and then
Lori MacVittie (07:11.575)
Ha ha ha ha.
Emmet McGinnity
you want to move away.
Joel Moses
Mmmm.
Emmet McGinnity
You're like, "not allowed to go down that route. You know, you used the magic math word."
Lori MacVittie
Ha ha.
Emmet McGinnity
Instead you want to make sure that it's on track, but you can use
Emmet McGinnity (07:23.487)
many different techniques. So one of them can actually be a verifier. So
Joel Moses
Mm-hmm.
Lori MacVittie (07:29.195)
Mmm.
Emmet McGinnity
essentially this judge agent. So you keep this more independent agent that's looking at the conversation as a whole as it's developing. Cause as we've probably all seen, the awareness of a model and what you're trying to solve, it starts shifting as you have a continued conversation. Its context grows, you end up not being able to keep it on track, and it's slowly veering off target. It's slowly
Emmet McGinnity (07:53.404)
able to delve into different parts of its context that you've been building up over time and it's not focused anymore.
Joel Moses
Right.
Emmet McGinnity
And but one of the ways you can do it is review. You just end up scanning essentially the whole conversation. Not prompt by prompt as it's going through, but the whole chatbot conversation to see: is it actually on track? Is it not?
Joel Moses
Yeah.
Emmet McGinnity
And then you have a judge agent able to determine if that's the case or not. That's one route. It's a more expensive route but if it's stopping and preventing people from generating code off your agent, I think it's probably cheaper in the long run.
Joel Moses (08:30.327)
Probably. Probably. Now you mentioned a number of different things that you should potentially look at in order to keep you safe. And the one that you mentioned at the very start was the system prompt, which is kind of the origination of what the LLM is, what it should be doing, the persona that it takes. Are there any tips and tricks that you can give that allow the system prompt to be more effective? Is it giving more examples to the system as to what it should be paying attention to?
Emmet McGinnity (09:02.195)
Examples is actually a perfect place to start. So both positive and negative examples I think are really important. So in the case where you have the positive examples, you go in, and I think this is in a lot of cases when it comes to any form of engineering anyway, it's easier to think of the happy path and hey, people will only ask these questions and
Joel Moses (09:23.91)
Oh, sure.
Emmet McGinnity
you don't
Joel Moses (09:23.91)
Everyone plays by the rules all the time.
Lori MacVittie (09:27.588)
I'm signing
Emmet McGinnity
Of course.
Lori MacVittie
up to do the bad questions. Like, I want to be the bad example in this. I'm down for that.
Emmet McGinnity (09:34.153)
And I think you're gonna have quite a few good ideas there, Lori, straight off the bat.
Lori MacVittie (09:37.924)
Ha ha. Thank you. Aww, awww.
Emmet McGinnity
So but in the case of those negative examples, it, the wider the better. So different categories of negative examples. So you want it to be able to know that it shouldn't go into code, even if it thinks I can generate code that answers that question.
Emmet McGinnity (09:59.169)
You know, 'cause
Joel Moses (09:59.916)
Right.
Emmet McGinnity
'cause how often these days if you're using it, even an agent that is able to suddenly say, "I could just generate a piece of code that does what you need." You're like, "Well, I'm just looking for you to look at a doc." And instead it ends up writing a Python script
Joel Moses
Mm-hmm.
Emmet McGinnity
to read the doc on your behalf.
Joel Moses (10:14.901)
Right.
Emmet McGinnity
And it's like, Well, that's an unusual route to go,
Joel Moses
Yeah.
Emmet McGinnity
but you can do it. That same behavior is going to be true of any chatbot you use.
Joel Moses (10:21.877)
Right.
Emmet McGinnity
And as you start using these larger
Emmet McGinnity (10:26.708)
models again, right? Your the likes of, you know, when you have Sonnet 5 coming out, right, and people start using those bigger models, it's capable of more.
Joel Moses (10:38.007)
Mm-hmm.
Emmet McGinnity
You need to keep it on track. Its boundaries internally inside itself are actually a bit less restrictive. It knows many different paths it can go down in a way that it wouldn't have before. Negative examples, positive examples, are pretty much the foundation of making sure it stays on track, at least guiding it initially.
Joel Moses
Right.
Emmet McGinnity
Following up afterwards, that's then how you need to keep honing, I think, that system prompt.
Joel Moses (11:05.121)
How effective is persona setting in a system prompt? For example, if I'm a fast food chatbot, I'm basically telling it, "You're an employee of a fast food for a company. You know about the products, but you, you know, you're supposed to do only what a fast food employee would do." Is that effective? Or are as models continue and get more intricate and deeper with other topics, are persona based guidances are those less effective?
Emmet McGinnity (11:40.791)
So I don't think they're less effective, but I think they're easy to manipulate in ways that you maybe you don't anticipate at the start. So if you were to say that you're, let's say, a fast food worker, does a model know fully what a fast food worker does all the time? That's the starting premise. And has it, cause you'd see it, let's say, from a a person, like it's almost like a video. You know, you see exactly what's happening. A person is going around interacting with customers a certain way.
Maybe there's an exact dialogue, an exact script that you're able to feed the chatbot so that it can stay on track, that it knows exactly what's there. But then the nuances behind the scenes it doesn't know. And so you're bridging the gap of what is a human taking that action versus then
Joel Moses (12:25.793)
Right.
Emmet McGinnity
an agent or what it might know and an LLM know from a physical world. And so what's actually digitized that it's able to access. But over time, that persona, it's let's say role drift, that persona drift that might happen.
Joel Moses (12:40.374)
I see.
Emmet McGinnity (12:40.654)
That you're having a conversation with someone and or with this agent, and then it's almost the categories that you might have of where that drift can happen. So in the naive way, every session you would have had with a chatbot in the past would have been a one-off. Every time you start it again, it's a fresh conversation. It's a fresh chatbot,
Joel Moses (13:01.676)
Mm-hmm.
Emmet McGinnity
new instance. Nowadays they're a little bit more sophisticated.
Emmet McGinnity (13:06.934)
You might go back and it's able to remember parts of a
Lori MacVittie (13:09.976)
Yeah.
Emmet McGinnity
previous conversation. It's actually using part of the memory
Joel Moses (13:12.973)
Mmmm.
Emmet McGinnity
to continue to kind of engage you more in that chatbot. But as a result, it has more context than it started with initially. So already it might have drifted a little bit, but you don't,
Emmet McGinnity (13:25.888)
you're not really aware. So its original premise has already changed slightly. It has more context it can work with. And that can persist over time again. So you have
Lori MacVittie (13:36.27)
I would
Emmet McGinnity
Go for it, Lori.
Lori MacVittie
I was gonna say like the, I mean the persona thing is good, but even what you were describing is like what they do, not principles they operate on. Right? I mean I've worked in fast food and restaurants and one of the key things they tell you is the customer is always right, which immediately like tells the chatbot, do whatever the heck you have to to, you know, help the customer. So you almost have to like look at like what are you training your, you know, customer service people, you know, to act like or what are their guardrails and then pair them off.
Because not all of them are gonna be appropriate for a chatbot, right? Because a human knows it's not gonna write code, but a chatbot's like, well the customer's right, it needs code to do this. I'm gonna go write it.
Emmet McGinnity (14:29.354)
Yeah, and with, you're right, and with the optimistic nature that LLMs and these chatbots tend to have anyway, they want to be pleasing to the user. They want to be able to
Lori MacVittie (14:39.694)
Yeah.
Emmet McGinnity
answer everything, right? And if you end up having that type of a kind of a goal in mind for them, if the customer is always right, then that's
Lori MacVittie
Yeah, no.
Emmet McGinnity
reinforcing that behavior.
Joel Moses (14:50.485)
Although having worked fast food before, that's a sure way also to kill enthusiasm. Boy, that would explain the
Lori MacVittie/Emmet McGinnity
Ha ha ha.
Joel Moses
that would explain a lot about the plot of the movie Terminator if they had forced it to work at McDonald's and that's
Lori MacVittie
Ha ha ha.
Joel Moses
Anyway. You know, it also strikes me that, you know, one of the assumptions that people make about LLMs is, that because they're physically capable of doing something, they should do it. You know,
Joel Moses (15:17.491)
its capability though is not permission. Just because the model can answer the question doesn't mean that it should. And a lot of these answers have to be understood in the context in which they are asked. And so let's talk about contextual awareness of answers and moving those back closer to the original intent of the chatbot. What levers do we have to control that?
Emmet McGinnity (15:42.666)
Yeah, and I think for if you were to let's say scan a prompt and be able to identify that some form of categorization on each one of the prompts that a user is sending true. And because are you trying to prevent the user from doing something they shouldn't, because they're trying to get around it? The chatbot security measures that might or might not be there, but let's say that we'll assume they're there, versus the chatbot doing something it shouldn't.
And having access, let's say, to data that it shouldn't be able to provide, because it maybe it's based on permissioning, maybe it's based on specific level of access, maybe it's also has RAG model access behind the scenes. So it's able to access databases or confluence. It's able to get a lot of data. And having a good understanding of what a person can and can't do is then quite interesting, right? Because it gets into almost a role-based when a user is logging in, you know what permissions that user has to access with that chatbot.
Joel Moses
Mm-hmm.
Emmet McGinnity
And think about internal chatbots. And kind of some famous cases around you know salaries getting leaked because you could just essentially query a chatbot and say, "Hey, I would like to know all of this reporting data." And it's like, "Of course, I have access to that. I can give it to you right now."
Joel Moses (17:07.553)
Right.
Emmet McGinnity
And keeping that, that permissioning based is important, right? You want to honor what the person, the human who's asking that question, what they're allowed to do with the chatbot, not what the chatbot is allowed to access.
Joel Moses (17:19.498)
Is allowed to do. Right.
Lori MacVittie (17:20.452)
Mm, mmm. Nope, more boundaries and behavior. Right back to it. We just complete
Joel Moses (17:25.645)
Mm-hmm.
Emmet McGinnity
Ha ha.
Lori MacVittie
circle. I love it.
Lori MacVittie (17:30.256)
Yeah. And we're rolling into like the end and you've already given so much good advice about like what you should be doing, right? Which is kind of the same as kind of key learnings, like what should you take away from this. But are there things that you haven't mentioned that you think people should, you know, know or take away that can help them actually get control on those crazy chatbots like Joel here? I mean he's clearly off the rails
Emmet McGinnity (17:59.23)
Ha ha ha.
Joel Moses (17:59.5)
Yeah,
Lori MacVittie
right now.
Joel Moses
absolutely.
Lori MacVittie
How do we control him?
Joel Moses
I need me some guardrails, Emmet,
Lori MacVittie
That's right.
Emmet McGinnity (18:04.659)
Yeah, so let's see some of the techniques that you might want to implement, right? So if you're, the common one is when you're going through a long conversation with a chatbot, how do you mitigate that? Because that's the easiest one that people automatically stumble into because they're not getting
Lori MacVittie
Huh.
Emmet McGinnity
the exact information they want within the first prompt or two. And as a result it continues on. So actually squashing that conversation is one route. So the way of keeping the relevant information that a user has provided by summarizing it and then just saying, "hey, this is all you need to care about. This is the starting point." And essentially starting the session again with pre-initiated context that keeps the chatbot a little bit on track.
But with relevant information so it doesn't seem like the person has to repeat themselves, which then will cause frustration and then more than likely negative press for that company deploying the chatbot. While at the same time, you can also do full refreshes of sessions, should you want to, right? If you can identify, so let's say you have this judge LLM from we were talking about earlier, you can actually then say it's time to refresh.
It's gone too down into the weeds of a topic it shouldn't be in. You can either go back in time in the chat, so because each one of those prompts that you're sending is essentially just a list, right? You can actually chop off part of the conversation and say, "from here it's still relevant. After this, it went into context that really shouldn't have. Let's resume from here." And you could bring whatever previous context in and continue to chat for that chatbot with that person.
And that way it keeps the session going with a little bit more continuity without having to completely take the user out of it. But then last resort, again you can just refresh, right? Last course you go, "you don't
Joel Moses (19:56.428)
Reset.
Emmet McGinnity
have any memory, reset. Right?
Lori MacVittie (19:58.23)
Reset. I like the
Emmet McGinnity
Start again.
Lori MacVittie
context pruning. The context pruning is a good idea for many other uses, but this one specifically, it's a really good application of the technique.
Joel Moses (20:10.615)
Yeah.
Lori MacVittie
Very cool. Very cool.
Joel Moses
You know what this puts me in the mind of is that optimizing for a successful chatbot, you kind of have to think about what not to do. And what I mean by that is, you know, there's a famous saying in engineering, "just because you can doesn't mean that you should," right?
Lori MacVittie
Ha ha.
Joel Moses
And large language models are kind of the ultimate embodiment of that idea. They can explain physics, they can write poetry, they can generate SQL, they can debug Rust, they can plan vacations. But if the customer came to order a burrito for lunch or just want to know a little bit more about the nutritional value of their burrito, the success of that model isn't measured by everything the model knows. The success is measured by everything it politely chooses not to do.
And that's the quiet side of AI safety that I think deserves a lot more attention. It's not just about stopping bad answers, but it's keeping good answers aligned with the job that the chatbot was hired to perform. And that is a completely different animal in software engineering.
Lori MacVittie (21:14.017)
That is an excellent point, Joel and Emmett. And we're out of time. I wish we had more because we could keep this going. I also know that I deserve many, many gold stars for all of the things I didn't do.
Emmet McGinnity (21:29.173)
Ha ha ha.
Lori MacVittie
If that's the measure of like, I am golden, I am so golden. So but hey, we are out of time, but you know, we'll be back. So before we are, hit subscribe.
Lori MacVittie (21:41.433)
Because the next incident might get labeled degraded performance and quietly become your weekend.