Subscribe
Share
Share
Embed
We explore the risks arising from the use and misuse of digital devices and electronic communication tools. We interview experts in the fields of cybersafety, cybersecurity, privacy, parenting, and technology and share the wisdom of these experts with you!
Welcome to the Cyber Traffic Podcast.
I am your host, Jethro Jones, coming to you live from the Inch 360 Conference here on the beautiful Gonzaga Campus here in Spokane, Washington.
And today we have Curtis Shelton, who is works for net SPI.
Is that right?
Net Spy.
Okay.
Looks
like nets Speed, but
Yeah, it does.
All right.
So, you're an AI researcher and you just gave a talk here and you said that you are the dark side.
Tell me what that means.
Yeah, it was a little bit of a play on words.
One of the earlier talks mentioned the good side and the bad side.
I'm sure it was a Star Wars reference they were talking about.
Mal intended individuals attacking things with AI or attacking AI specifically, and that's where all of my research has been.
So I definitely exist in the red slash purple realm of security space, and I attack models all the time, and I use models even more often to attack things.
So when I joked about being on the dark side, it was because she was talking about how there are the bad actors out there, and I emulate them all the time.
Actively putting research in everywhere I can to try and be ahead of them.
before they do find those vulnerabilities before they do put together offensive frameworks before they manage to do so.
And that's where I spend a lot of my time.
And then also a lot of what I do is trying to take existing frameworks for penetration testing and whatever medium it might find itself in, and then finding the elements of that machine learning can.
Make continuous automate, just make that better and faster to allow humans more time for that critical thinking stuff.
Let them spend less time on the mundane.
Yeah Well and that from my experience has been the best use of ai allowing me to not worry about the mundane and spend time on the things that That really matter So for example in my work I submit presentations to conferences quite often
I have a few things that I do regularly and what I used to do is just cut and paste those and have the same exact thing But what I've been doing is using AI to rewrite those descriptions in a more entertaining engaging whatever kind of
Yeah, Why wouldn't you?
Yeah So that it's not the same exact thing Even though what I'm going to be doing is largely similar regardless of what my submission is it actually makes it more exciting to apply for those things because it's a very tedious thing
to
the, it's miserable.
It is miserable.
And I, it, and I understand why they do it, but when you see the little caveats where they're like, but don't use AI to submit, submissions
they, I get why they're worried about it, and I get that they don't want people submitting things they don't understand or submitting.
Just slop of stuff that they had put together really fast.
But at the same time, like you, you would be so foolish to not adopt the machine learning wave as it is because it just makes everybody so much more ENT at what they
do
And that's a good thing.
I had a really neat friend, so he's a hardware specialist like a hardware hacker.
And one of the things he said to me early on when I first made my way into the security field was everyone is bad at their job.
Your goal is just to be a little bit less bad.
Yeah
And I mean, at the time I laughed and had a lot of, I still do have mountains of imposter syndrome, but after spending that time with him, learning from him, and then subsequently branching into
machine learning and getting to do this for a while, and now seeing how much influence AI has had on almost every industry, almost every job out there, as long as text is involved in some way, it Right?
And in a good way, it has helped people be less bad.
a good yeah.
You know, despite, obviously there, it's a double edged sword.
Right.
You know, the more they use it, the more complacent they might become.
Right.
heard a myriad of different arguments in contrast, but at the end of the day, within my small little sphere, I have seen nothing but just efficiency improvements.
As long as they're not willing to just let AI Jesus take the wheel and let it drive all on its own Right.
As long as they're there to just kind of give it that last check and make sure that it's decent and accurate with their expertise.
They've got it.
Yeah
Yeah.
And there's it's just such a cool spot to
yeah for sure So I wanna talk a little bit more about the the bad stuff the figuring out what the LLMs can do and and hacking them also I've seen a lot of stuff about prompt injection and things like that And so that's something that you're familiar with also
Yeah.
Painfully familiar
Yeah And So a lot of those things most normal people I don't think really need to worry about prompt injection because they're probably just chatting with chat GPT and there's not much that's gonna go on the that
problem comes in when you're automating many different tasks and you're not capable even of paying attention to everything that's going on And that's where those problems are going to happen Is that a fair way to look at
That is a fair way to look at it.
I would maybe encourage you to not make it as complex.
So like an example, machine learning models, right?
We'll say generative models, LLM specific do not have the ability innate to call external functionality, And so the systems like A two A or MCP or a
bunch of other very solid protocols out there that are designed to help either agents connect with one another or connect with external tooling or both.
designed with text in mind, right?
You have one model with a very strict structured prompt that gives you a structured output, which then communicates with another one, expecting that input, expecting
that output
That generally works pretty well.
However, if you simplify it and you go, okay, I have a, I have a web application and I have a chat bot here, and the web application is some SQL database where you can go and you can query the database, get information back.
It's just that no other agents, it is literally just this one model capable of querying a database.
Prompt injection is very pertinent to that situation I might be able to leverage my ability to jailbreak the target model into returning data or
querying the database in a way that it doesn't expect, and thus returning tables, deleting tables, getting data I shouldn't have access to, et cetera.
So it's not always just the sheer complexity a system that really leads to prompt injection, but it's more.
The business function in conjunction with the target function, the math underneath the hood, and all that other good stuff with what the model is intended to do.
So one, one more, one more example.
So if you are like a really fun thought that I like to pose to the people I work with now.
Is like new computers are being designed and released.
We had, we were exposed to one, again, I won't use the name, but almost the entire computer was meant to be run from a little command line, which was a chat interface with the model.
It was like a series protocol models that are all meant to call different things.
So you could literally say something like, I want you to open Excel and build meet a retirement.
Like, and it would do all of that for you.
Really, really cool.
Right.
Awesome.
Awesome, awesome idea.
But the security mind there is like, well wait a minute.
Like there, you might be able to do a lot more with that.
I can I'll leave two quick ideas.
One, I think it was Gardner just a couple days ago released a statistic suggesting that by the end of 2029 it was like 50 to 60% of injection vulnerabilities will be found through LLMs.
So one two.
At my work, my team finds those sorts of injections weekly, like command injection, SQL injection, all sorts of malformed behaviors.
Because at the end of the day, one fundamental like lesson that the industry just really hasn't grasped onto yet is math.
Is math.
If math can make it, math can break it.
All models are inevitably breakable to every single model specific vulnerability that's out there.
Now, maybe that'll change, but that inevitability of breakability.
Is just a hard concept for a, I think a lot of people to really hang on to.
So when they construe and they build these systems, they expect that the generative model is going to behave and think like a human.
When in truth it's just a stochastic hodgepodge of token tumbling that results in outputs that generally are predictable, but can be broken like any model.
/
Yeah I was reading something and I cannot remember who wrote it now but it was titled The Lesson and it was a guy saying that really what it comes down to is compute If you give it enough compute time and resources it can solve anything and it's not going to
solve it in the same way that you and I would And we designed these things thinking that they're going to react how we will So for example yeah so it he talked about in the paper he wrote he talked about how Gary Kasparov was defeated in 97 by was it Watson?
Yeah.
The Okay.
Go yeah, it was
No it was chess Chess first and then go later but He was defeated because the computer was basically looking at every move he made and then just roleplaying out every possible reaction and then choosing
the best one And the researchers would discredit the computer beating him because they would say well that's not how people play chess And what they missed was that is how computers play chess because they
have the capacity to make all those calculations And so I'm seeing this connection between what you were just saying about if it's mathematically gradable it's mathematically breakable Is that what you
said?
If math can make it, math can break
There you go that's much better So
yeah I love that.
So I. Mantra, I guess, I dunno, something I've been screaming from the hilltops forever is the inefficient way in which a lot of these frontier model providers benchmark their releases, right?
When a new version of, and hopefully I don't get in trouble for using product names.
I don't hate any of them.
They're all great, but like and a new frontier model comes out and you see these engineering benchmarks one shot, one prompt five prompt stuff.
How great is it?
How, and these.
These benchmarks are intentionally misleading as to your point.
You know, complexity is an issue, like the models need to become more capable.
But with the stochastic nature of these models, the randomness that's innate to this system, if you take well advantage of that and have something that you can
measure, given its outputs for accuracy, you could take a much smaller model, a much less proficient model, and generate thousands of prompts for the same cost as one.
And eventually get the right answer.
So, which one's more valuable in the two scenarios?
Prompting once and getting one accurate answer or prompting a thousand times and getting six accurate answers.
I would argue it's the second because despite six being accurate, you still have a thousand that are useful and they may not be perfect.
And again, this is all contingent on optimization, right?
It's all an optimization game.
How much.
Math you leverage to make sure that this output is exactly what you need it to be.
And then you can do really cool stuff on your phone, right?
Even the new GPT models that release that are on your phone level size, they're not the smartest, but given enough prompts, you can get there if you really want.
And that's what's dangerous.
That's what's cool.
So to your point, it's all about com.
It's all about compute, right?
The more you can prompt something, the more likely it is that you're going to have, you can do more damage and you can train better things.
And so it's great that these.
Pricing models require that we pay per prompt, per token, because they're running the compute.
For us, it makes a lot logistics sense, but as a user, that doesn't necessarily mean I'm interfacing with the best model, right?
Just because it's more expensive, or just because it can answer my prompt in a singular query.
For a lot of people, that's enough, right?
That's all they want.
I want to ask question.
I wanna get an accurate result and I'm done.
But for the science, half of everything the math people, the security experts the practitioners.
Often I find it a lot more valuable to get the bad answers first and then create that contrast dichotomy, train models on those outputs.
So then you can ensemble every problem that you're looking at and subsequently be able to put this system you've built against anything.
suddenly walls begin to topple models break.
You can automate jailbreaking.
You can automate SQL injection.
You can automate vulnerabilities 'cause you've just made it a classification game.
Just like old machine learning, right?
Did I or did I not succeed?
And then you optimize for that,
And how crazy that's what it comes down to
yeah at the end of the day, everything is or is not a mango.
Like
it is a silly way to look at it, but you know, objectively, that's just how the math works and you can build from that.
You can.
Objectify certain problems, optimize for those, for that objectivity, and then branch those attacks into meaningful ways in real world scenarios.
Now, is this like the perfect academic view of everything?
Absolutely.
It's a lot harder than that, But you always have co-factors there to stop you.
But at the end of the day, that general process is never done me wrong.
So
yeah How interesting so if somebody wants to like follow in your footsteps do the kinds of things that you're doing how would you suggest they go down that path
That's a hard one.
So at the time you're asking me that question, there are
a few, there are a few good resources out there.
I myself mentored under a gentleman named Will Pierce.
I don't think he would mind if I dropped his name.
He's, he is greatly accredited with building their AI red team at Microsoft and then went from there to Nvidia and built their AI red team there.
And I met him while he was at Nvidia and without him, I don't know how else I would've done it, but he's just not.
I mean, he owns his own business now that he just doesn't have energy to mentor everybody, even though I think he would want to.
And so outside of being able to find somebody credible that would be able to give you those lessons, I think there are a few resources out there that have become really good.
There's a couple books.
There is an NIST framework that they released maybe about a year and a half ago that is actually pretty spot on despite it.
Postulating that it isn't even within the paper itself, right?
Like recognizing, I don't know a lot, but like those resources are pretty on the button and then the rest is just getting in there.
Learning how to build models yourself, starting from the ground up and just getting good at it.
One place I like to point people is an open source lesson track called FAST ai.
They've also built their own Python package, which to be honest, is just like an optimized python.
It's a wrapper around PyTorch, if you've ever heard of PyTorch before.
But it'll help people learn how to build models quickly, further, how to like work with data, augment data, and just become a really neat data engineer.
Data science or machine learning engineer is probably a better way to put it.
Yeah
And then from there it's about getting the secure, like your hands dirty.
And the problem with machine learning, not the problem.
What makes it so fun is machine learning is going everywhere, right?
Oh, we've done engagements on hardware, we've done engagements in web applications.
We've done engagements on networks, phones.
You take your pick and new stuff, new tools come out all the time.
All the time that are just fantastic.
It was really difficult for somebody to sit down and say, I specialize in machine learning, right?
Because it's everywhere.
There's no way you can feasibly do that, especially in security, where it's just constantly adapting.
And so I find myself in a unique and almost dishonest position to say, like, I do what I do and I love what I do, but like, I would crumble if you pointed me at hardware again, it's been so long since I've done that.
I would flop like a fish if you pointed me at a right?
But.
So my suggestion, sorry.
That was a really long, rambly way of saying find a mentor.
If you can't find a mentor, go through fast ai, get good at machine learning, and then branch from there to finding a security specialist like specialization, excuse me, something that machine learning will sit in.
And again, web applications are probably the lowest hanging fruit there because everyone and their mom is putting LLMs in their applications.
And once you get to that point, you start getting comfortable with how the models feel and that comfort bridges into how I can break them.
And then it's just experience and suffering.
Until you there, yeah.
And suffering that.
That is a great way to end it.
Curtis, thank you so much for being part of Cyber Shops Podcast.
I really appreciate you being here and supporting Edge 360 also.
Yeah, no problem.
This was a pleasure.
Thank you.