AI Security Ops

Community Q&A on AI Security | Episode 34

In this episode of BHIS Presents: AI Security Ops, our panel tackles real questions from the community about AI, hallucinations, privacy, and practical use cases. From limiting model hallucinations to understanding memory features and explaining AI to non-technical audiences, we dive into the nuances of large language models and their role in cybersecurity.

We break down:
  • Why LLMs sometimes “make stuff up” and how to reduce hallucinations
  • The role of prompts, temperature, and RAG databases in accuracy
  • Prompting best practices and reasoning modes for better results
  • Legal liability: Can you sue ChatGPT for bad advice?
  • Memory features, data retention, and privacy trade-offs
  • Security paranoia: AI apps, trust, and enterprise vs free accounts
  • Practical examples like customizing AI for writing style
  • How to explain AI to your mom (or any non-technical audience)
  • Why AI isn’t magic—just math and advanced auto-complete

Whether you’re deploying AI tools or just curious about the hype, this episode will help you understand the realities of AI in security and how to use it responsibly.

Chapters
  • (00:00) - Welcome & Sponsor Shoutouts
  • (00:50) - Episode Overview: Community Q&A
  • (01:19) - Q1: Will ChatGPT Make Stuff Up?
  • (07:50) - Q2: Can Lawyers Sue ChatGPT for False Cases?
  • (11:15) - Q3: How Can AI Improve Without Ingesting Everything?
  • (22:04) - Q4: How Do You Explain AI to Non-Technical People?
  • (28:00) - Closing Remarks & Training Plug

Brought to you by:
Black Hills Information Security
https://www.blackhillsinfosec.com

Antisyphon Training
https://www.antisyphontraining.com/

Active Countermeasures
https://www.activecountermeasures.com

Wild West Hackin Fest
https://wildwesthackinfest.com

🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits –
https://poweredbybhis.com

----------------------------------------------------------------------------------------------
Joff Thyer - https://blackhillsinfosec.com/team/joff-thyer/
Derek Banks - https://www.blackhillsinfosec.com/team/derek-banks/
Brian Fehrman - https://www.blackhillsinfosec.com/team/brian-fehrman/
Bronwen Aker - http://blackhillsinfosec.com/team/bronwen-aker/
Ben Bowman - https://www.blackhillsinfosec.com/team/ben-bowman/

Creators and Guests

Host
Brian Fehrman
Brian Fehrman is a long-time BHIS Security Researcher and Consultant with extensive academic credentials and industry certifications who specializes in AI, hardware hacking, and red teaming, and outside of work is an avid Brazilian Jiu-Jitsu practitioner, big-game hunter, and home-improvement enthusiast.
Host
Bronwen Aker
Bronwen Aker is a BHIS Technical Editor who joined full-time in 2022 after years of contract work, bringing decades of web development and technical training experience to her roles in editing pentest reports, enhancing QA/QC processes, and improving public websites, and who enjoys sci-fi/fantasy, Animal Crossing, and dogs outside of work.
Host
Derek Banks
Derek is a BHIS Security Consultant, Penetration Tester, and Red Teamer with advanced degrees, industry certifications, and broad experience across forensics, incident response, monitoring, and offensive security, who enjoys learning from colleagues, helping clients improve their security, and spending his free time with family, fitness, and playing bass guitar.
Host
Joff Thyer
Joff Thyer is a BHIS Security Consultant with advanced degrees, multiple GIAC certifications, and deep expertise in offensive security and exploit development, who enjoys crafting sophisticated malware for penetration tests and, outside of work, making music and woodworking.

What is AI Security Ops?

Join in on weekly podcasts that aim to illuminate how AI transforms cybersecurity—exploring emerging threats, tools, and trends—while equipping viewers with knowledge they can use practically (e.g., for secure coding or business risk mitigation).

Joff Thyer:

Hello. Welcome to another episode of AI security ops with our illustrious cohosts, doctor Brian Furman, doctor Derek Banks, and doctor Brian Wanakae. We're all gonna be doctors today. And I hope everybody is enjoying their holiday season as it's starting to kick off. Hope everybody's got their Christmas trees put up if they're that way inclined.

Joff Thyer:

As usual, this episode is brought to you by Black Hills Information Security. If you are interested in getting any AI architecture kinds of assessment in your organization, feel free to contact us. Visit our website, blackhillsinfosec.com, find the contact us form, and clickity clicky, proceed from there, and we will try to get back to you as we normally do. We not only try, we will get back to you and see if we have a nice match. For today's episode, we are going to be talking questions and answers from the community with regard to our prior podcasts, and let's see how we how we fare with these things.

Joff Thyer:

These are always fun because everybody has a different kind of opinion on AI tech today, and so we get some pretty diverse questions. So in our lovely panel today, who wants to take the very first question? No volunteers.

Derek Banks:

I guess I'll go. Alright. So I'm gonna remember to strike this one from the list after I say it and answer it. So one of the questions, I think this probably came from Discord a while back. If you tell ChatGPT to give you real examples of something and tell it not to make stuff up, will it make stuff up, or will it tell the truth that it has no answer?

Derek Banks:

And my answer is, well, that all depends. So I think that, you know, you're referring to hallucinations and things just getting made up. And there's different levels of that. Now I was recently using AI to help me on a pen test, and it was leading me down a road where I after I went and talked to another tester, found out that it was leading me down the wrong road. It wasn't necessarily hallucinating.

Derek Banks:

It was just steering me in the wrong direction. But I was able to figure that out kind of thing. But just straight up hallucinations. Well, I mean, I I think that those on the large frontier models for like chat GPT or, you know, Claude, for example, have really been minimized in a lot of the ways. I haven't really seen straight up hallucinations from those models in a while.

Derek Banks:

I think when you talk about like local models, a lot of those that we use are still kind of a little older. I think they might tend to hallucinate or give interesting results based on the input. A little bit more than frontier models. But also, I mean, it also depends on, you know, how things are kind of set up going into the large language model. Things like temperature will control how creative the model is.

Derek Banks:

Also, the system prompt has a a a big role in that. I was recently doing some experimentation with a a rag database, like a knowledge base and a large language model. And in my system prompt, flat out said, under no circumstances, give me any answers that weren't found in the Rag database. And I actually kind of hobbled the model. I mean, kept saying, I don't know, I don't know, I don't know.

Derek Banks:

And really, that was a little bit too extreme. So I think, you know, one of the mistakes that I see, you know, people make is that, like, it's one big AI. Right? And that's not the case at all. I think once you the more you learn how these things actually work, the more you realize it's really just a a really awesome software tool.

Derek Banks:

There is no even when we reach the AGI, you know, pinnacle, I guess, I still don't think it's gonna be like the sci fi AI where it develops consciousness and takes over the world. I just I'm still not really seeing a path to the AI taking over. People with AI, maybe that's a different story. But back to the original question, yes, you can limit hallucinations in various implementations of large language models.

Joff Thyer:

Oh, great, great answer. I love the extra explanation around that, Derek. I do relate to this idea that if you seed an LLM in your prompting with a sort of direction to begin with, it will definitely amplify that, and kind of lean you down that path, and so and and quite compliant. Right? So they

Derek Banks:

Especially like in multi turn scenarios. Right? Like I'm trying to do I was doing like trying to like write a utility, like specific for the pen test I was on and going down a road. I can't really be more specific than that. But it it, you know, it just kinda steered me in the wrong direction.

Derek Banks:

I didn't straight up hallucinate, but it wasn't correct. Yeah.

Joff Thyer:

Wow. That's a good one. Okay. Well, any other comments on that or we wanna take another question? It looks like Brumman has a comment.

Bronwen Aker:

One of the other things I've noticed about the Frontier models, particularly when you're on a premium account, is that they have the new reasoning modes, and you can do different levels of research, and that generally drops the confabulations and and, bad misses a lot because it's checking its own work. That isn't to say that if I have a question that I wanna be sure that the answer it's given me is really on target, I will play LLMs against each other where it's like I take the answer from one, I paste it into another and say, how legit is this? Or something appropriate to whatever the content is. And and that is surprisingly effective in terms of of getting a more reasonable response, correcting things that wouldn't have necessarily been caught on the initial pass. You can even do that with the same LLM where you take what it gave you, you drop it back into a chat and say, how reliable is this?

Bronwen Aker:

Is this really answering the question that I want accurately and responsibly? Are these are these links legitimate rather than having you go through and check all 15 links that it's provided to you? That sort of thing.

Joff Thyer:

So interestingly, in that response, you you allude to something that I think is actually pretty important, and and that is prompting skills. Right? If you, in your prompt, give the LLM some space to do some chain of thought reasoning, to do some multi step process, and even do some initial planning with you with the challenge that you're trying to solve, you're gonna end up with a much better result. And I think, you know, there's a lot of folks out there that are just doing really quick one shot prompts, looking for excellence out of the result, and they're just not necessarily getting it.

Derek Banks:

We're getting a lot of mileage out of this question because now I'm gonna say, oh, a lot of the times when I have, like, something serious I want to research, I have a large language model give me a prompt meant for the reasoning mode. Right? I actually just did that today. I And was really happy with the results. In fact, they got real specific on some stuff, and I I was impressed with the output.

Derek Banks:

That's for sure. So it's all in the prompting. Yeah.

Joff Thyer:

Well, seeing as I think three of us there have covered that, I'm gonna throw the baton over to doctor Furman to go with the next question and answer.

Brian Fehrman:

Alright. Next question. This I think is kind of an easy one, which is so in the case with the lawyers when ChatGPT made up cases, can those lawyers sue ChatGPT because the lies got them in trouble? So I think they're talking about the incidents that we've had in which lawyers have submitted cases with basically precedent to precedent cases as part of that that just literally just not did not exist that they got from asking ChatGPT basically to do their work for them on that. So can they, sue ChatGPT for that?

Brian Fehrman:

I don't think so. I mean, at least there's I don't think there are any laws at this point. I think that that argument is marginally better than saying that you saw something on Reddit and it wasn't true, but you, like, you know, you did it and you found out that they weren't telling the truth and misguided you. I mean, so they I mean, most models will say that basically they do their best to give you factual information, but when it comes down to it, there's no guarantee. I mean, it's just it's a tool.

Brian Fehrman:

It doesn't it doesn't know for certain, like it doesn't know what it doesn't know. I mean, this kinda goes along a little bit with the last question that that we had. And so you can get the information out of it, and oftentimes, it's probably going to be good. It's probably going to be factual in a in a lot of cases. I mean, it's gotten to the point where it's really good.

Brian Fehrman:

Confabulations aren't as bad as they used to be. But when it comes down to it, it's still up to the person to confirm that that information is correct. Don't just blindly take it and take it at face value that it is going to be truth. That's that's just that's not how it works. And if you do that, I certainly don't think it's reasonable to hold the the LLM companies accountable for it.

Derek Banks:

Yeah. I was gonna say, bet the ELUA says, basically, yeah, don't you can't hold us responsible for what this thing says. Right? Like, at least if if if if when you create an account and it doesn't say that, I'm surprised OpenAI would still be in business, to be honest. But and and I like your analogy to a tool.

Derek Banks:

Right? I mean, that would be like, Zan, can a mechanic sue craftsman because the wrench didn't fix the motor. Right?

Joff Thyer:

Like Yeah.

Derek Banks:

I it's the same kind of analogy. And the more we hammer to folks that this isn't like an entity, it's not magic, it is a mathematical tool that can that should be used in a specific way, then I think the better off we'll all be.

Joff Thyer:

Yeah. I I like to I like to think of it in in terms of always treating the LLM as kind of like this inexperienced intern, and they're going to give you some sort of response to your prompting, but you've gotta always put it in context, like, okay, you're an inexperienced intern. So whatever response you give me might be something like super creative, but not necessarily based in the truth or facts or whatever. So I need to, the more experienced person, check you on this. Right?

Joff Thyer:

So that that's kind of the position I like to take on it even though, you know, that's not necessarily true because the the LLM in their training and their ability to to augment with retrieval augmented generation is probably a lot better than an inexperienced intern. It's really your your prompting skills that kinda determine what's gonna happen. So okay. Well, that's awesome. I think it might be Brumwin's turn to pick a question from the pile, so

Bronwen Aker:

I'm just sorting through the pile, and and a lot of them are really good and could probably be used for like an hour long presentation. Let's try This one is interesting. What can AI really do to make me better without ingesting everything I do? And that's there's there's a lot of layers to that because one of the things that, I see crop up every now and again is where somebody puts a prompt into chat GPT or it's usually chat GPT because they've been around longest, but sometimes it'll be other LLMs and they say, what do you know about me? And the the prompt will vary in terms of degrees of complexity and whatnot.

Bronwen Aker:

And and one of the the early innovations that ChatGPT brought into the space was this concept of memory where it retains information about you based on your interactions with it. And, every now and again, I I double check, and it's like, who is this person that they're talking about? This person sounds like they're really together, and that doesn't feel like me at all. And then other times, it's like, yes, that really is me. Thank you very much for fluffing my ego.

Bronwen Aker:

That kind of memory retention is a double edged sword, and maybe even it has more than just two edges. And it it showcases the challenge in terms of the data retention by these frontier AI companies is that every single interaction that you have with any frontier LLM, any third party hosted chatbot, there is going to be data retention on the part of the provider in terms of what your interactions have been, what you've said, and that data is going to be mined. Without that data retention, there's no way that those third party organizations are going to be able to refine the behaviors of the tools that they're generating. So I don't it's it's a conundrum. It's it's that good news, bad news.

Bronwen Aker:

The good news is, yes, the LLMs, the other AIs, they can can help you get better and better at stuff. The cost, however, is that it has to retain data about you. It has to be able to do down the road processing and and analytics and interpretation in order to refine the behaviors and have that performance become better, it's the price of doing business.

Joff Thyer:

I think, you know, in that, it's important to distinguish some of the application architectures as well, because a simple chat application is absolutely gonna have some retention, but it's mostly gonna be in the form of the the, you know, ever increasing context window by by retaining prompts over over a session. Right? You know, in a more complex application, let's say one that's that's got lots of different agents in it, it's multi agentic, it's got orchestration in it, it's something that's much more sophisticated than a chat application, yeah, there's no doubt they're gonna be leveraging retention in that application at some level, but it's probably more going to be on the organization itself, is my thought, who's who's actually deploying such a thing. And I've got that on the mind because of Genetic AI has been a very interesting area for me lately that I've I've actually done a lot of work in that space. In terms of examples, I think it's worth putting out there.

Joff Thyer:

I think Derek has a good example here, which was when he developed a small proof of concept to write blog posts in his writing style, and it is an example of providing the LLM in the prompting process an example of him. Right? His personality, his writing style, his the the aspects that he wanted the LLM to generate in the new topical area. And that was deliberately providing something about himself as a part of that application. Derek, I don't know if you wanna speak to that at all.

Derek Banks:

Well, yeah. I mean, so there's a lot of different like level, there's a lot of nuance to providers keeping data and what they do with that data. I would say the standard rules apply if it's free, you're the product. They're definitely using your prompts to to, you know, whatever they wanna do with it. In my experience, at least with OpenAI and Anthropic, if you're paying them from an enterprise account, they claim that they're not taking the prompts and further using them, you know, to which usually hackers go, oh, and you believe them?

Derek Banks:

Oh, I mean, I don't know. Do you believe Microsoft or whatever? I mean, I I can tell you right now, if Microsoft wanted to ruin Black Hills, they could certainly do it. Right? Like so I guess at some point, you have to trust put your trust into somewhere, and then that's when the features become, you know, pretty pretty useful.

Derek Banks:

Right? It said to your point, Joff, yeah, I purposely gave, you know, blog posts in a prompt so that ChatGPT would write in my style, and it did a pretty good job, like, starting out paragraphs with so comma. Right? Like, so see, that's what I just did.

Joff Thyer:

But I blame Security Weekly for that.

Bronwen Aker:

Yeah. Yeah.

Derek Banks:

And then, you know, so I I wouldn't I don't necessarily think that allowing a large language model, especially one that you're paying for to retain some level of data on your prompts and how you're using it is necessarily a bad idea. I I just asked Cloud Desktop. I've been using Cloud Cloud more than ChatGPT recently. I just said, hey. Tell me about what do you know about me?

Derek Banks:

Right? And it gave me something that was really amazing, highlighting, like, everything that I had done through Claude all year long. It's like, dude, I should use this on my performance evaluation. It sounds great. Yeah.

Joff Thyer:

Right.

Derek Banks:

I'm not gonna read it to everybody. I was just saying that I I was really impressed. So I think that, you know, by nature, folks in the information security community are very conscious about certain types of data, especially data privacy, which is kinda ironic since every EDR takes everything in your environment and ships it off to the cloud. So, you know, it doesn't matter at that point. But I I guess, you know, what I would say is don't don't necessarily be afraid of giving providers your data.

Derek Banks:

Just make sure you understand what they say they're doing with your data.

Bronwen Aker:

One of the the things that I noticed, I was going through the the App Store just last night, and I was looking up AI apps. There are so many. And I think this is where that that paranoia that hackers and and infosec professionals professionals have really is a survival skill because every single one of those organizations, every single one of those developers with a different AI app is going to have different terms of service privacy policies, yada yada. And that's I'm in in some ways, I'm less concerned about the big companies because they have more visibility. It's the little

Joff Thyer:

Even with the big companies,

Derek Banks:

if you're concerned about it as a security practitioner, you better not have TikTok on your phone. I'm just saying. Right? Because if you read terms of service for that, like, they basically have a keystroke logger on your phone. So yeah.

Derek Banks:

But then, you know, I think I don't know that again, I I do agree with you that, like, it's we're still in the middle of this AI hype cycle, which maybe is something we'll talk about in the next episode. But I don't know that it's different than any other apps. Like, I am very cautious of the apps I put on my phone, especially now my iPhone on the latest, version. I don't know if anybody has the latest version iPhone or iOS. Or basically series integrated into everything.

Derek Banks:

It is super handy to, like, just summarize everything. But, you know, do I, you know, know that Apple is keeping my data secure? They better be because they say they are. But, yeah. I mean, I think that what is it?

Derek Banks:

Back in was it '92 or '93 when sneakers came out and, you know, they basically, data is the new oil. Yeah. Back then, data was important ever more so today.

Joff Thyer:

We we all realize that. And yeah. Okay. Well, why don't we move on to the next one unless doctor Furman has one comment to add to that that thread?

Brian Fehrman:

Nope. No. I think I think we covered it all. Maybe just as a side comment for me personally, like, I when I use the on the different LLMs that I use, Chattypetit, Cloud, whatever, I shut off the memory feature just because I don't really like it to take previous things into context when it's answering new questions for me. So I know that that's just kind of like a personal preference.

Brian Fehrman:

But for me, each time I open up a new window, I like it to be a clean slate. So that way things don't get, like, jumbled because I found before that it becomes this weird kind of it can become like a weird spaghetti mess of like you asked it something before. So it assumes that that's relevant to what you're doing now and I don't really like that. So like I shut off that memory feature personally, but about the only thing I have to

Joff Thyer:

That that that's actually a really, really good point because you can end up again, there's a prompting issue. Right? You can end up with conflicting instructions in your in your historic prompting, and then you get it you get the LLM effectively confused. So I think that's a it's a really good one. Okay.

Bronwen Aker:

Let's I must be the only person who goes through and and deletes tons of chats. I I do I do that. Lots of conversations. And also video

Brian Fehrman:

that too.

Bronwen Aker:

Brian, one of the other things that I learned is some of the LLMs will have the ability to do a temporary chat. Those give you a clean slate.

Joff Thyer:

Yeah. Let's do one final question before we close out. Okay? And the one final question is going to be kind of one of my favorites. I'm gonna paraphrase the question like this.

Joff Thyer:

Right? The actual question is how do we discuss AI with nontechnical audience? But I'm gonna I'm gonna I'm gonna phrase this question as how do you explain AI to your mother or your father? So what do you guys think about that? Non technical audience, don't know what the hype is about.

Joff Thyer:

How do we discuss that with people that just have no technical knowledge at all about what this stuff is?

Derek Banks:

I've actually done this multiple times. And I was told, oh, that makes sense, but I don't know if they were just being nice to me or not. So but I basically use the black box analogy. Like, okay. If you start out with, like, the basic machine learning, like the like the most basic machine learning, you know, that we have, it's basic, you know, data and parameters and arrow going into a black box and then expected results outside of that black box.

Derek Banks:

So if I'm predicting housing prices with linear regression, it's just a mathematical algorithm that I put in a bunch of data, and then I get out my prediction. That's no different all the way up to large language models. The same paradigm applies. So, you know, we whatever I put in, it's basically taking all of that, transforming it into numbers, and then outcomes the most probable output to my input. Not answer to my question, but output to my input.

Derek Banks:

Yeah. And that's kinda how I describe it to the the layman, all how all AI works right now.

Joff Thyer:

Right. And and we have to be careful here because implied in this question is really AI large language models, not necessarily just purely AI. We've said before on the show many times, there are multiple different models that are not large language models that are that are task specific that don't necessarily apply to this question. So from my point of view, I like to go back to the analogy of the naive child or the inexperienced intern. And I would probably start out by saying to my mother or father that, you know, what you're dealing with here is a digital simulated human reasoning entity.

Joff Thyer:

Right? And you can treat it, or you should treat it kinda like an inexperienced child and construct a question along that basis, and you would get some sort of generated text as an answer. So that's kind of where I land on it. Doctor Fuhrman, what's your thoughts?

Brian Fehrman:

Yeah. So I think it's it's important I think one of the important things for people to realize when we're talking about AI is that these things aren't actually sentient. I think that a lot of people who don't deal with the technology have this notion that these are actual thinking organisms that when you put the information in that they are definitely using that right away, they've now like, you've now given a term they're using it. Like, I mean, it's instantly integrated into, like, the collective knowledge of of that system, and that's just that's not quite what's going on. I mean, like we discussed in the previous question, sure.

Brian Fehrman:

I mean, if you're using LM for free, the company might be taking your data. They might be using it for future training, but it's almost certainly not like an instant learning experience at this thing. You know, it's not like you've told a person a secret and now they know that, you know, now they have that knowledge immediately. Right? That's just not how it works.

Brian Fehrman:

It's that, you know, kinda like what Derek was mentioning with this transformation process. Right? I mean, you're just you're giving it data. It goes through basically a transformation process based upon the data that you give to it. You get an output that is hopefully relevant to the input that that you gave it.

Brian Fehrman:

And that's really all that it's doing. So there's no magic. There's no definitely no sentience there. It's

Derek Banks:

Well, magic is the math. Right? Like, if you get down to the nuts and bolts of how a transformer actually works, I mean, it is an an engineering a software engineering marvel, right, that that we were able to come up with the ideas, and it behaves the way that it does. I I I do think that it is a a super technical like, the more you go down to nuts and bolts and understand it, super technical, but it's also still just math.

Joff Thyer:

Well, I'm I'm not I'm not about to give my 85 year old mother a linear algebra Oh, no. In my day.

Derek Banks:

A 100%. Like, I I taking those math classes, it was bad enough for me. I don't know that I'm qualified.

Joff Thyer:

And I could do that. I actually do have a degree in mathematics. So but, anyway, I would have to do a lot of refresher before I got there. I'll let Bronwyn take take the, the final statement on this one. What do you think, Bronwyn?

Bronwen Aker:

Well, the analogy that I use is that the LLMs in particular are basically autocomplete on steroids and what you put into it is going to dramatically impact what you know, the prompt that you put in is going to dramatically impact what it gives back to you. But all it's really doing is calculating, oh, here's this word and 29 times this word was an x word and 57 times this other word was an x word. And it's going to go to whichever is most likely based on the data that it already has. So as has been said, it's not intelligent. It's not sentient.

Bronwen Aker:

All it's doing is this really advanced auto complete. Generally, I find that that's something that everyone these days has enough experience with that they can wrap their heads around it.

Joff Thyer:

Probably. But the next thing your relative's gonna say is, what do you mean? What is this word autocomplete? But, anyway, we'll we'll leave it at that because I think to go any further, we're gonna get into into the weeds. If wanna go

Derek Banks:

further, you need to take training classes with us.

Joff Thyer:

That that's right. That's right. Absolutely. I'd like to thank everybody for joining us for this episode of AI Security Ops with our illustrious cohosts, and we will see you next time. Keep on prompting, be safe out there.