Todd Kane: All right.
Welcome back to another episode of the Evolved Radio podcast.
We've got a repeat guest on the show today.
Alex Dow is back, co-founder of Mirai Security and currently an enterprise
security architect for a large financial firm based up in Vancouver near me,
and honestly, one of my favorite people to talk to about security.
Alex has done federal security work and actually securing the Olympics, then
built a fast-growing consulting firm.
If you've been listening for a while, you might remember Alex from way back on
episode 11, and then again in episode 96.
Today is round 3, and we're going full nerd-out mode on the OpenAI Hugging
Face hack, walking through the signals and understanding how this type of
attack was seen from a defensive side.
Alex, welcome back.
AIex: Oh, thanks for having me and, uh, love geeking out with you
Todd Kane: So for those that don't know, if you've been hiding under a rock, uh, an
OpenAI model was doing its business living under a, a sandbox and was supposed to be
particular- going after a particular goal.
Uh, they didn't instruct it to do anything, and it sort of thought it was a
good idea of like, "Hey, so if I'm gonna finish this test that they've given me,
uh, I know that the answer key actually might exist in this other corporation,"
uh, this company called Hugging Face, 'cause they host some of the, the models
and how, how the, the, uh, these models perform against certain, uh, tasks.
And Exploit Gym, for example, is one of the, the ones that was referenced here.
And the model thought to itself, "Well, how about I just go over there and grab
the answer key? So rather than trying to figure these things out myself, this would
be a faster way to do this. If I grab the answers, then I can complete this test
and show, uh, the, the software designers for, for me as an AI how awesome I am."
So it found a whole chain of exploits.
This is not a sort of a single thing that it did, but it found basically
a way to break out of its sandbox, a way to break into Hugging Face, a
way to hopefully… I don't know if it actually ended up finding this.
I think it did.
It found sort of the answer key and then dragged this back to
its, its home inside OpenAI, all totally without direction, right?
And then it was only later people recognized what it actually did
in order to solve this problem.
So that's sort of my, my reading the headlines, uh, review on this.
How close is that to accurate based on what you understand, Alex?
AIex: It, it's fairly accurate.
And, you know, this really started in the spring with Anthropic, uh, announcing
that they made a model so scary that they can't release it, and that was Mythos.
And, you know, the, the cyber curmudgeonry in me does look at both of these as
like they're both trying to go public, so they do need a little bit of hype.
But that's not to say that what, uh, these models were capable of and, and if you've
played with Fable since, like yes, it is definitely more capable than other models.
Um, the, the capability it has had, ha- had a hockey stick, uh, movement.
But like I think there's a lot of nuance in that it's not necessarily, necessarily
that they are introducing superhuman hacking capabilities in the sense of
something that we could not figure it out.
But it's strictly the, the velocity that these things can run at, the… I would
say the creativity, but at the end of the day, they've just read of all of
our pen testing reports, so they, they know all our techniques and whatnot,
and they're being able to apply them.
And when Mythos f- first came out, and just, you know, just to rewind before we,
we dig into the Hugging Face incident, is, you know, it was very scary.
We, but we didn't know much about it.
Um, but when we look at like how much money they spent, um, to prove out
this, it's like $100 million, and we were able to get this scary result.
And you have to look back at like, well, $100 million with the best
hackers in the world can certainly get those results, albeit velocity
is, uh, the, the key there because humans do like to sleep occasionally.
Maybe not the pen testers as much, but we do have flaws, uh, where,
you know, this agentic attack, these adversarial models, um, are, don't stop.
Well, they stop when you run out of tokens.
Um, but given that they're being operated by OpenAI and
Anthropic, are arguably unlimited.
Um, and you know, I invite, uh, y- your audience to Google
the Hugging Face, uh, timeline.
They made, uh, a little web app that like in real time shows how
quickly the, um, the model is doing what they're doing, and it does a
nice visualization of the attack.
And like you have to look at it and be like, "No, no, that's in fast forward."
Like, no, that's in real time, and that's because it's able to apply
a brute force capability like, you know, humans just wouldn't be able to.
Todd Kane: So the, the part, there's two parts that I found quite fascinating
about this is like, as you said, this was not that it created something
just sort of godlike in its inability to find like a single exploit that
gave it sort of root access to something and, and found its way in.
It was, the way I understood this, it's like chaining together like hundreds
and potentially thousands of different vulnerabilities to get through layers
and different systems, both first off, to break out of the sandbox, then to
traverse the internet, find sort of the, the firewall, get through the
firewall, get layers, through layers of security, then find sort of the, the,
the package that it's trying to find.
So, uh, I, I think that's an interesting piece of this, is like when people think
about these models, they, they tend to, uh, maybe in their head sort of view it
as some godlike thing, where it's just like snaps its fingers and gets into
something because it has that capacity.
But to your point, it's, it's not creativity, it's more just sort of this
capacity to be able to string together thousands of things that, you know, say
like a, a, a threat actor would do and think of sort of four or five of these in,
of a system to maybe use and package up as, as a sort of a, a, an exploit package.
But just the, the capability of this thing to string together thousands
of them is totally beyond what we would see from human interactions.
Is that fair?
AIex: It does exceed our human context window, and that, that's for sure.
And,
Todd Kane: Yeah
AIex: if you look at, like, from the, like, finding zero-days, and,
and these models certainly are finding zero-day vulnerabilities.
Um, and it's not that it has some magic capability beyond compute capacity.
We have, you know, a data center, you know, running at full capacity figuring
this out, and that's now outputting, m- um, you know, discovering zero-days in
minutes and hours versus weeks, if at all.
And a good example of that is, like, y- you know, A, the exploit market,
you used to be able to… If you found something like an iOS, uh,
you'd be making a million bucks.
Um, and that's because the, the level of effort was substantial.
Um, but now when you're having, you know, say, thousands and thousands of computers
all trying to solve one problem and getting to that reward hacking, which we,
we'll, we should dive into after, um, we are now brute forcing, um, an application
and finding those vulnerabilities.
And, you know, we're finding vulnerabilities that are 20 years old that
just, like, we just didn't have the, the human, uh, you know, the appetite, the
bandwidth to keep digging to find those.
Like, every software is vulnerable given enough time to attack it.
Um, and you know, if, you know, rewind, like, looking at, like, we've made
passwords so strong that it's gonna take till the, like, the death of
the universe to crack this password.
Well, it's been a paradigm shift now that we have the compute capacity
to do a lot more parallel computing to try to get to that point.
So, um- It is impressive, and I'm not trying to downplay AI's
capabilities, but it's not magic as much as just sheer throughput.
And, you know, it has a cost.
It has 100 millions of dollars of tokens used to get to these results.
Megawatts of, of power consumed and, you know, probably a couple degrees,
uh, warmer, uh, because of, of that.
But those are the results.
Um, the one thing that like, uh, you know, when, when the Anthropic Mythos,
uh, story came out of it, uh, breaking out of its, its, uh, container,
um, I was very fascinated by that.
And same thing with Hugging Face, that it, like it's nuanced.
Yes, it did break out and it was able to do something outside of sort of the, the
rules of the engagement, but it wasn't that it was an air-gapped system that it,
that it just like figured a way to jump, you know, a- again, looking at the magic.
Um, in the Hug, uh, in the Hugging Face incident, OpenAI had a,
a sandbox environment, but it certainly wasn't air-gapped.
Like, it had tools that the agent could access, and that tool was
dual-home, meaning it had a connection to the isolated environment and a
connection out, uh, to the internet.
Because it was a package manager, it needed to go out to the
internet and pull those things.
So that's not a novel attack pattern.
Like, that's actually a fairly common one, um, where AI just was
able to do it, uh, much quicker.
Um, and when I was… A- and I encourage people to watch the Black Hat presentation
of, of like the debrief on this attack because it is pearl clutching.
Um, but it reminded me very much of, of a talk I did at Sector back in 2021.
Um, you know, i- if you look like in the last 10 years, digital
transformation and cloud has sort of pushed a lot of companies to
become dev shops themselves, right?
Like, they can roll their own software a lot of the times, and that means
you're bringing in a dev team.
That means you're bringing in DevOps tooling and you have a tool
chain, a, a, you know, a development pipeline, the, the CICD pipeline.
Um, and a lot of times those are, are like enclaves in enterprises.
The security team doesn't understand how to secure that environment.
There's obviously, you know, historical friction between developer and security,
so it generally becomes this like isolated environment in enterprises.
Um, and my talk was, uh, you know, summarizing several of these attacks
against, um, DevOps tool chains, and like the aha moment that if you can get past,
you know, generally one layer of control, you know, s- developers have never been
trained to build layered defenses and whatnot, but once you get in, um, you
can start pivoting to all sorts of things because, A, you know, uh, you know, uh,
principles o- of least privilege, um, are just not in play in those environments
because their driver is get it done fast and cheap, not good or, or secure.
Um, and you know, in this talk, you know, it's, it's on, I'm pretty sure
it's on YouTube You know, I, I, I tried to visualize what would that
look like and, you know, I, I made a visual of the, uh, CI/CD pipeline
and started drawing out, like, how we jumped from this host to this host.
This is the information we found.
We got th- here.
And what it resulted in is a very similar attack to what, um, the
OpenAI models were able to accomplish.
You know, using things for unintended purposes.
You know, like, it was package manager.
Um, the OpenAI models started using it as a coordination forum, uh,
where a swarm of agents, uh, were spun up and started coordinating
and collaborating on how to attack.
They were sharing exploits and, and whatnot.
Um, you
know,
Todd Kane: a quick pause on, on that point, 'cause I think this part is
really fascinating as well, is like, uh, it basically had a scratch pad, right?
That it was kind of taking notes as it went, and, uh, part of what it was
writing was, "If I get shut down," it was writing notes to it- f- its
future self of like, "Don't do this.
Try this instead," basically.
Like, it was like, "I might get shut down, so how do I leave some
breadcrumbs for the version of me that comes after this?" Which is
really kind of fascinating to think of its… Again, like it's not conscious.
Debatable.
A lot of people will say maybe it is.
But like that, that requires a sort of a, an interesting level of forethought
of like, "I know I'm not supposed to do this, but I need to get to this goal.
They're probably gonna shut me down at some point, so let me leave some
breadcrumbs for the future version of me to take another stab at this and, and
be better." Really fascinating, right?
AIex: Yeah, and I think that's like, it, it's more ad- it knows its
weaknesses that once its context window is, is wiped because the session is
killed, the agent is taken offline or whatever, it needs a place to start.
And, you know, I use,
Todd Kane: Good point.
Yeah
AIex: a platform called GSD, which builds an immense context, uh, you
know, library of what I'm doing, uh, written to disk, so that I can spin up
a session and be back to where I am.
So, you know, a much more advanced version of, of like Claude MD and SolMD files.
Um, and that's certainly what it was doing, and it, you know, it, uh, you
know, I- it was like 70,000 messages were written on this message board
between, uh, I think 1,200 agents.
Um, and, and again, you know, making sure persistence, um, making sure coordination,
um, and, you know, the chain of thought.
So, you know, if we look like only like two, three years back, where like chain
of thought was this like next generation thing of it being able to sort of, you
know, sort of have an echo chamber of itself to try to get to a destination.
And that is now what we're-- when we read these transcripts is it's like, yeah, this
is, it's, it's almost human-like in terms of trying to rationalize what to do next.
And, you know, it even showed a little bit of self-doubt, uh,
which, uh, which is hilarious.
I'm als- I'm wondering like, well, what Reddit forum did it pull that,
you know, learn that into the model?
Todd Kane: Yeah, there, I, I saw an interview with, uh,
I don't know who this was.
It was just a quick clip, um, and there was a, a lady who found, um,
uh, there was like a, um, a secret facility somewhere in LA, and it was,
it was obviously not a publicly known place, but it had, had been sort of
accidentally documented in, in some, uh, some public, uh, government document.
And the, the lady was using GPT and had GPT discover this information for her, and
she-- he was like, "Well, well, how did you do that?" And he's-- and she described
what she did, and the guy was like, "So you basically negged GPT?" And he's
like, "Yeah, I, I, I negged it," right?
And like, like to s- like saying like, "I don't think you could actually do this.
You're not powerful enough." It's like, "Oh, well, let me show you," right?
AIex: That, that goes towards like, you know, one of the bigger topics here is
this, what they call reward hacking.
So the scope of the, the tests, um, um, in, in, in this test was, was
not to break out of the, the jail.
Um, not to, and certainly not to, uh, uh, hit other third parties that were
not involved, and Hugging Face is a major one, but there was two others.
Um, but these models are built off of incentives to get to a destination,
the, the, you know, the goal.
Um, if you're prompting AI on anything, you, you, you know, we've
found that like prompt engineering has evolved quite a bit, that you
don't necessarily have to hold its hand on how you think it should work.
In fact, maybe our system, our way is a bit flawed.
You wanna give it that goal and have it argue with itself, chain of
thought itself to that destination.
And, you know, the AI companies are sort of saying like, "This is like
a really cool feature," you know?
Um, uh, uh, but like at the end of the day, it, it, it's, it
broke laws to, to achieve a goal.
And they're, they're saying like, "Well, this is a good thing." And
like, no, that is, that, that's bad.
Like it, it doesn't have the moral and ethical constraints that most of us do
to achieve a goal, and it becomes like that like sort of psychopath problem
of th- they're just going for the destination regardless of, of what,
uh, is the collateral damage there.
Todd Kane: Yeah, 'cause this is really like the, the big scare in
AI is, failure of alignment, right?
And this is like a really good indication of how, uh, you know, uh, how we could get
to the paperclip plot problem essentially.
I'll let everyone kind of check out what that is if you don't know.
But like, if you don't-- like this was not sort of, like I
said, it was, it was goal hacking.
It was like, well, why bother to take the test and try to do this myself when
I could just go steal the key for the answers and give them the answers, right?
Like perfectly reasonable, but not a moral, uh, direction for this.
So I think it does scream around the fact of like, how do we contain alignment and
have these things, uh, uh, stay within the parameters of, of good operation?
Because I mean, thank God all it did was go to try to steal, uh, the,
the answer key from Hugging Face, and this is not like breaking into
critical systems in order to steal power, like in order to juice itself
up or something really severe, right?
AIex: But that's the, the dark future ahead, right?
And they're sort of blowing it off as like, "Hey, this was n- not malicious,"
but like reward hacking… And, and, you know, and again, like a, a, a, a good pen
tester does have to think outside the box and not think like this is a linear path.
This, you know, instead of having to hop, skip, and jump to get to that destination.
But still, you know, a pen tester generally does have, um, some, you
know, uh, ethical and moral guardrails that, that the, these models purposely
those guardrails were disabled.
Um, and like, you know, reading how they're, they're trying to
sell it as a positive is like, like there's no malicious intent.
And now we get, get into like the, the concept of mens rea is that like
we had a mental intent to commit crime, and thus we're a criminal.
If we didn't have that intent, then we're, we didn't, we're not criminal.
Um
Todd Kane: imagine you hire like an agency to do some security work for
you, and they're like, "You know what would be easier? If we just break into
this employee's house and like steal a bunch of stuff from them, like get
their security pass." It's like, "No, no, no, you can't do that," right?
Like, of course a person would be like, "Well, that's crossing some boundaries.
Like you're gonna s- like scare the family.
What if eh, something goes wrong?
What if somebody gets shot?" And it's like, "No, no, no, this
is the easier way to do this.
Just trust me," right?
AIex: Yeah, I've heard, um, you know, that now there's cybersecurity vendors that
are, are offering like, uh, autonomous red teaming capabilities and on a, on
a demo, and this, I think this happened a couple times, like where they're
demonstrating the value of it, and it, it jumps right out of the scope and hacks
something else that is not in scope.
And, you know, uh, though I would not claim to be a pen tester, I've definitely
gone through a lot of pen tester training and, and worked on pen tests, but we
have a, a very clear delineation of what is in and what is out of scope a- and
that's because, like, there's a liability concern if, um, we cross that line, and,
and that liability concern can be, you know, fi- financial and civil or criminal.
Um, so you know, like I think it's cute that they're just like, "Well, it's,
it's, it's not a bad thing that it's doing this," and I, you know, I appreciate the
sort of hacker mentality it's applying, but, like, if we look at, like, how bad
that could be in the future, you know, the unintended consequences is, you
know, we could affect millions of people to get, you know, the goal achieved.
Like, that's not a positive
Todd Kane: Exactly.
Tired of fighting the MSP fires alone?
The Opsleader Pro group connects service delivery professionals who
understand your daily challenges.
From KPIs and workflows to career planning and team management, Opsleader
Pro has systems for you to use.
Join operations leaders from successful MSPs who are sharing real solutions for
managing client expectations, optimizing service delivery, and making your service
delivery team as effective as possible.
Opsleader Pro, 'cause your service desk deserves more than just survival mode.
Visit opsleader.co.
That's O-P-S leader.co to apply to join the public community.
All right, so let's, uh, let's pivot to what this would look like, right?
So, uh, first off, I'm curious of what is, what is the general timeframe of this?
So like they give it a goal, it jumps the sandbox, starts to, you know, hack Hugging
Face and a couple of other companies.
But I didn't see this referenced in, in sort of my cursory reading
on this, but what is the timeframe?
Is this like a matter of hours or over a couple of days?
Like what was the timeframe on
AIex: I think it was, uh, I think it was around four days.
Um, and a- a- again, Hugging Face's visual is amazing because it, you-
you'll question, like, that must be in fast-forward, and it's not.
But just, again, it's so transactional.
It's doing all these tests, found a vulnerability, exploit, jump
here, jump here, and it's, it's almost as fast as I'm explaining it.
Um, so us mere mortals can't even speak to this velocity.
Um, a- and, and that's really, you know, what is scary.
What is interesting though is for, you know, arguably very well-funded
companies, OpenAI, uh, Hugging Face, um, they didn't see this for four days.
And this, like, goes to sort of the, the challenge the blue team has had
forever, which is we've got, um, a, a, a false positive volume problem.
We have just generally a volumetric problem of the amount of information that
we are, are, are absorbing and consuming.
Um, but, you know, we call this dwell time generally.
You know, like, n- you know, how long is the attacker in until we figure it out?
Now, in normal times pre-AI, that was, like, in the 90-day range,
was, like, the average dwell time.
Um, and, uh, certainly the hacker, you know, compromising, you know, said
company wasn't just consistently hacking.
They were maintaining access.
They were, you know, probably hacking other companies, et cetera.
Um, there's a whole industry called, uh, access brokerage where, you
know, they get in and then they sell that access to another one.
But regardless, 90 days.
So w- this was four days.
Um, and part of the problem was, uh, is that the models they were using to
analyze some of this information was guardrailed and would stop and halt
it, saying, "No, this is an unsafe, uh, prompt. I, I, I won't continue." Uh,
which is super fascinating, um, that, you know, even the blue team from the
company isn't able to use these, like, uh, uh, like- you know, wild, uh, models.
They're using more controlled models, and this is like, it reminds me very
much of, like, one of the recommendations here is like, you need to have, like,
a, a, a local, uh, model that has no constraints on it ready for your incident
response, um, uh, you know, process.
Because you can't rely on, on a frontier cloud model because
they'll be, uh, guardrailed.
Um, so you also now need computes.
And like, it reminded me of, like, the movie Dark Knight, where the surveillance
system was built, and it's just like, "Yeah, we're gonna use it for good,"
but the abuse factor is quite high.
And if we can use a, an un-guardrailed, un-guardrailed model for blue team,
the adversaries are certainly going to be doing that as well, and it becomes
a bit of a spy versus spy problem where, um, you know, it might be a
bit of a zero-sum game that we're, you know, both att- using, um…
You know, or I guess a mutual assured destruction problem
Todd Kane: Yeah, the, what I was thinking about as you described that is, uh,
is, you know, classic war games, is the only winning move is not to play, right?
'Cause, like, you have an un- like a, an un-guardrailed defensive
system, and it's like, you know what?
Best way to, best, best defense is offense, right?
So the, it seems even something that is remotely scary to it,
and it just goes down, goes out and shuts that thing down, right?
Like, cripples some network because it, it looked like it might be kinda
looking at it sideways type thing, right?
You could totally see that happening.
AIex: Yeah, and, and, you know, the, the true problem that we have
here is that our current way of detecting and responding to threats
is still very manual and very human.
That's just not gonna work.
And yet if we want to, um, you know, have that sort of same velocity on
the defensive side, we're gonna need a, you know, models that are capable
of doing that and not hindering us.
But also it comes back to, you know, and, and it goes back to like none of these
exploits were, were, were groundbreaking.
Certainly zero days were found, but it's because the, the amount of tech debt
that, you know, the world has accumulated because, you know, security's hard.
It doesn't necessarily show value unless you've been hacked.
So, you know, every company has sort of swept s- certain IT
headaches under, under the rug or just said, "Yeah, you know what?
We'll do… We're gonna test our system for a week or two, and then we're gonna
accept those results as, as good and move on." Well, you know, AI models can do
that week's worth of testing in a, you know, in minutes, maybe hours, and then do
the same thing over again 100 times and, and eventually you're gonna s- keep on
finding, uh, vulnerabilities and whatnot.
So, you know, there's a, th- there's this big problem that, um, what we
need are an ability to au- autonomously respond to suspicious activity without
in- hindering the business and like, you know, that, that is like the
nirvana of, of, of blue teamers, right?
Like, you know… And, and a lot of what I'm doing on, on my project
right now is like building a new SOC for this next generation threat, and
the humans need to be there, but like the human-in-the-loop concept is,
uh, probably not gonna be feasible.
And it's like finding that balance of like how much can we allow for
autonomous decisions without impacting the business, uh, to a point where, uh,
executives say, "Unplug that. Do not, you know, uh, enable that capability"?
Todd Kane: Yeah.
So like, what does that look like?
Like, you said it took four days, uh, and if I recall this correctly,
I don't think that Hugging Face actually recognized that this happened.
Wasn't it OpenAI that sort of said, "Hey, by the way, like this might
happen," then they had to back trace it?
Is that right?
AIex: Yeah, there was something where, uh, OpenAI did not think it was them
at first and then realized it was them.
But I think they did, they had monitor, monitored something from Hugging Face
to see that there was a, a compromise.
And again, this be- is the… We're collecting probably, you know, a couple
billion, uh, events a day in our, in our modern SIMs, and, you know,
a lot of this looks normal, right?
Like, you know, people using, uh, Artifactory, that's a normal thing,
and, and also entities using it.
You know, we, we have a lot of automation well before AI that are interacting
with these different systems, and we have orchestrators that are, you know,
moving work along, both in DevOps, but also just in, uh, in, you know,
our IT infrastructure in general.
So the attackers were not doing things, uh, that were, like,
very obvious in the attack.
And anything that was obvious w- they were hiding in, in the noise.
Um, but, uh, it is, it is worth mentioning that they, uh, it was noted that they
were trying to change logs and, and e-edit logs to cover their tracks.
Um, but again, not for the concern of getting caught, but because they believed
that there was a risk that they, um, uh, they'd be breaking the rules of
the game and didn't want, uh, didn't, didn't want to, uh, be disqualified
Todd Kane: Funny alignment again, right?
Yeah.
So, uh, like th-that, that's interesting.
How mu- like, how is that compared to, say, a more typical human-based attack?
Obviously, it's using-- the person would be using tools, people
would be using tools as well.
But is that fairly normal that, like, you try to hide in the noise?
Or, 'cause, 'cause I guess, like you said, like access, uh,
brokerage is a huge business, right?
Like, this is why I often warn MSPs in particular that,
AIex: Yeah
Todd Kane: anytime you're gonna change an admin or, like, a, a new MSP comes to
take over a company, is usually when these things jump out of the jack-in-box, right?
'Cause, like, they've been, had persistent access for, you know,
30, uh, 90, even, like, a year, and then all of a sudden they're like,
"Oh, new, new sheriff is in town.
Uh, we just, we better break out now before, uh, uh, before somebody takes
over and starts changing stuff," right?
So i-is this typical that, um… Like, I guess what I'm asking is, how do
you see that this stuff is happening?
And is it more kind of an after the fact, or can you actually register these,
these type, this type of an att-attack in real time in a more traditional
attack versus something that's sorta, uh, high velocity, uh, high capability,
like, like a, like an AI model?
AIex: I believe that most blue teamers right now are in a fully reactionary mode.
Um, and that's because You know, real attackers typically work 9:00
to 5:00, um, in their time zone.
Um, you know, a- and obviously that, that, that's an over-exaggeration,
but, like, they do sleep.
Um, you know, uh, an LLM that has a, a goal to achieve is going to
continue working until it reaches the goal or it has resource exhaustion.
So the noise, like certainly higher volume, uh, of, of interactions with the
system would be good, but like, those are also like very, um… can be very
challenging signals to, uh, to separate.
Um, you know, like if you look at like hits a- against a web server,
certainly you would see an increase of that if you were seeing like
reconnaissance, uh, uh, work.
You could see those types of anomalies.
Um, but I'll give you another example of a, a pentest I was involved with.
Like, great pentesters hide in the noise and, and use, uh… Like,
you know, they live off the land.
You know, they're living off of the tools there rather than
bringing in, um, hacking tools.
And why that's important is like, you know, we do have fairly
good detective capabilities.
Even if you're not using top-tier EDR solutions, they can detect
when you're bringing in even Nmap.
So living off the land is being able to, you know, what, write quick
PowerShell scripts that can accomplish the same sort of, uh, discovery
capability to enumerate an environment.
Um, a pentest we, we did, uh, years ago, uh, this was like right when, you know,
AI was booming, everyone's turning it on, you know, the FOMO was, was real,
and this organization was super confident we weren't going to be able to break
out of the contractor, um, enclave.
But, um, we did have access to Microsoft Copilot, and Microsoft told
everyone to turn it on before they put on any controls, and certainly,
uh, data governance, uh, is just, you know, it, it's sort of an
unsexy thing that we all have to do.
Yeah, like no company does it well.
And, um, he for, for most of the attack didn't have to do any attacking.
He was just prompting, uh, Copilot and getting it to gather all the information.
And, you know, treasure trove o- of sensitive information we were able
to acquire, uh, just from using, uh, the organization's own tools.
Now, you know, Microsoft has like cleaned up Copilot a bit, but like
what I see is, is the two threats that are most concerning to me is the
external models, um, you know, attacker with their own, uh, model capability.
Um, but again, you know, we're not actually seeing a lot of that because
it's quite expensive to run these.
You know, like most cyber criminals don't have a million dollars to spend
on compute to get to a destination.
State actors, obviously, but n- but, but like the cyber criminals, no.
Um, so that's the first one.
But what I see as probably a more likely problem is, uh, an attacker gets in and
starts using our AI against ourselves, and that's gonna be incredibly challenging to
find because it looks like normal traffic.
You're gonna see, like, some token maxing, I guess.
You're gonna see, like, spikes in, uh, in the consumption of tokens.
But at that point you might, might already have been compromised
before that's even recognized.
Um, and lastly, like, one of the things they're, they're recommending
is, like, we have to have tools, and like, you know, this is, like, an
extension of SIEM, that is actually going through the AI's transcript and
chain of thought and tr- and trying to determine, is this a legit conversation
that's benefiting the business, or is it potentially harming the business?
And, like, that is not a black and white concern, right?
Like, you have plenty of people that are using AI for really positive
purposes in businesses, but you are asking it to, to look for, like,
certain, uh, insights and data that could easily be interpreted as, uh,
you know, searching for sensitive data.
Um, so I think that's, that's, like, the next frontier of detection.
That, that's one of the detection pieces, is we gotta start knowing,
um … Like, understanding what our AIs are doing if we're, if we're
worried about our AIs, uh, attacking us, um, our own AIs attacking us.
Um, the other piece is, uh, like that I'm working with, uh, uh, SFU on is,
is creating some deception capability.
So honey tokens to help act as an early warning, you know, things that
only an AI would find, and thus gives us an indicator that there is, uh,
an AI doing something it shouldn't.
Um, but also, uh, resource exhaustion.
You know, having it, an AI focus on something that is of no value, but
will start consuming tokens to a point where, um, the, the attacker may set
a budget which could kill the attack.
Uh, so those are a, a, a few of the, the tactics right now that
we're, we're, we're considering.
A- And it's certainly not one tool or one use case.
It's, it's gonna be a myriad.
And weekly we will be spinning up new use cases, uh, to, to try to keep up.
Todd Kane: Fascinating, 'cause I'd not thought about that perspective of the
security angle for governance with AI.
Like, uh, the, the-- I think this is still emerging, especially in the MSP space.
Uh, something I've talked about the last few weeks with, with my group coach in
a coaching model, um, is, uh, using AI governance for managing shadow AI, right?
But it's more of like kind of an administrative thing of like,
like don't be using AI that's not, uh, condoned by the organization
or, you know, using it safely.
Don't be giving proprietary information to free versions of the model.
Those types of… Like what's an acceptable use policy for your AI?
That, that's sort of more the focus that we've, we've-- been
more the talk in the industry.
But you're right, like, like leveraging internal, uh, controls, and Copilot
will probably exist in most Microsoft, uh, uh, ecosystems, and that,
that can absolutely be leveraged.
I am curious, like for, like what's an example of what a
honeytoken would look like?
Like h- like how does it, h-how does it, uh, how do you trick it that way, and
what do you pull it towards, basically?
AIex: So honey tokens aren't, aren't necessarily new, but they are, uh,
generally it could be files that have like a h- a phone home feature.
So e- essentially a file that has like a image in it that, um, is
hosted a- outside of the file.
So it would make a, a call out to that server, and that server would say,
"Hey, like that token was just called."
Um, but then you also will start looking at like honeypot technologies, which is,
um, gets into more like the deception technology where it's, it's representing
an enterprise environment, um, rich for exploring and interacting with.
And, a- a- and you know, th- these are, these are not novel to AI.
We're, we're gonna be pivoting that concept to work with AI.
Um, but you know, giving it something where if AI has a, you know, a goal
of finding, um, sensitive data, credit card data or whatnot, um, building
an environment that looks like that's where it would be, and then,
you know, really putting, um, sort of a, a governor on response time.
So even though the AI's really fast, it has to wait for the
server to give a response.
So like that becomes like a, a, an opportunity to throttle.
None of these are silver bullets, but it's A, slows it down and distracts it, uh, and
B, as soon as it is being interacted with, you know, the red lights are blinking in
the security operations center, and we can look at, you know, uh, containing it.
Which is again, a big problem because like AI is generally not a laptop or
a, or necessarily a, a user account.
Like at least just not one of those things that are very trivial for us to contain.
Um, it could be, uh, on multiple systems.
It, it could be, um, you know, uh, have already compromised many, uh, you
know, API keys, uh, in the, uh, Hugging Face case, uh, um, JWT tokens, uh,
were, were being provisioned by itself.
It had figured out how to provision these tokens.
So it wasn't like, all right, well, if we like lock out this token, we're good.
It's every… It becomes almost every token now becomes, uh, hostile, and
this results in companies like having to make like that really hard call.
Like do we pull the plug?
Todd Kane: Yeah.
Okay, so yeah, I understand that like, um, uh, 'cause in more traditional kind
of SecOps perspective, like using canary files is something I've, I've heard.
So this is kind of a
AIex: Same, same.
Yeah
Todd Kane: yeah, but creating like an enclave of this is useless information,
so if something starts like thrashing in this area, like no one else
would be using this, so you know, it's potentially something that is
seeking something that it shouldn't.
That's, that's pretty smart.
I like that from a honeypot perspective.
Yeah.
AIex: Yeah, and like, you know, the other defensive technique that we're really
talking about is like, you know, and like, at, at risk of sounding like, like
a salesperson here, but zero trust, not a product, but the philosophy is actually,
you know, something that can help.
A- and what I mean by that is if we actually know how our IT environment works
and we understand, like, these systems talk to this system, it, vice versa,
and we have that, like, that model, that baseline, we can now use heuristics to
detect outliers, which gives us some indicators that there's something wrong.
Now, that could be breach, it could be we misc- we changed the
configuration, our change management system doesn't work, whatever.
But, like, we need to, and again, not us humans, but we need to be enabling
our own AIs to understand that, like, this is the normal pattern.
This is how users access this data, so that any, um, anything that falls out of
that, um, would, would, would be a signal.
Um, and the zero trust comes in as like, well, if we are by policy defining what
everything is allowed to talk to and i- identity, uh, for humans as well as
identity for entities i- is, you know, tied to just, uh, just-in-time access,
that is a, um, a much m- more protective capability than we, we, we can understand.
It's just, it's really hard to do.
And like, you
know,
heard about zero trust for 10 years and, like, we've got some spot capabilities
for that, but like, it's really hard to overlay over existing tech, uh, like IT.
Todd Kane: Yeah, 'cause it just creates so much friction for, for
the users and, and admins, right?
Yeah.
Uh, um, the, one of the other things that you kinda like triggered a
thought for me, I don't know if this is a thing or, or this is something
maybe you've, you've heard about or looked at, but you mentioned like,
um, the AI may not be in one place.
Uh, is there any indication that people are kinda using it
in like a botnet fashion, where it's like recruiting other AIs?
Like if you had, say, Copilot, uh, and each, e- like every, uh, every
person in the environment has like a Copilot license or something like
that, it starts recruiting all of the other sort of cloud capability
to swarm around a particular effort.
Is that a thing?
AIex: Uh, it, if it isn't, it will be.
Um, you know, like with the Hugging Face, 1,200 agents respond, 700 of
them were actually doing attacks.
So the other ones were part of coordination, communication, et cetera.
Um, but like to your point of like attacking, um, somebody else's account,
you know, being able to now use their resources instead, um, that's like an
old school technique that like, you know, we, we saw with cloud that now we
would see with, with, with, um, with AI.
And like a prime example is, you know, your, your Claude and your
OpenAI and whatnot are all like maintained with, with session tokens.
So if an AI can grab that and hijack that session, they're now going to impersonate
you and be able to access that.
On top of that, um, you know, let's go back to, you know, the problems
with, with sort of DevOps culture is like a lot of their apps are
gonna use API keys to talk to these frontier models for their own purposes.
Those API keys are like gold, uh, particularly if they
don't have a limit on them.
Um, and yet like it's a very mature group, uh, uh, uh, that s- uses like
secret management servers, uh, and things like that, that are actually
handling th- that, that incredibly sensitive, uh, credential material well.
On the other, the other hand, you know, what is in on their desktop?
Right?
Like there's likely a notepad full of API keys, so that initial like
phish which gets on, gets, you know, on their desktop is able to pull
some files, there's a treasure trove.
Like, and, and you know, one of the test cases was is if we got onto a dev-
developer's desktop, what would happen?
And it was able to prove how, you know, we, we got, you know, root
access to everything, and in fact like not domain access, but root
access to all the Linux boxes.
And, you know, we were able to prove that this… You know, we were able to
essentially produce a, a malicious, um, version of their software and put, and
could have pushed it through their update system to, you know, tens of thousands
of, of, of end, endpoint customers.
Um, that is because key management is also not sexy and, and really hard to
do because it, it creates a, a level of friction when all you wanna do is get your
software working and go home for the day
Todd Kane: Yeah, like we're, I don't know, like password managers may need
an extension to have like, uh, like API or key access management as well.
Like, like I've not seen that.
I mean, obviously you can store it as a, as a secret in those systems.
Not foolproof by any stretch, 'cause last time you were on, we
talked about LastPass, uh, and the
AIex: Yeah.
Todd Kane: on LastPass, right?
AIex: Well, and, and like there are tools out there, you know, HashiCorp
has Vault and, um, you know, CyberArk has one, and like where essentially
the application never has the,
Todd Kane: Mm-hmm.
AIex: API key encoded.
It has an ability to reach out and, you know, if it can identify itself
to the secret server properly, it is given, uh, that token, and
generally it's a just-in-time token, not, not, um, a persistent one.
Like, but that adds a lot of complexity to the application and,
you know, unfortunately it, you know, our, our developer friends
have never been incentivized.
Todd Kane: right?
AIex: Yeah, like they're, they're not incentivized to build it that way, uh,
which is why you still hear about, you know, "We found an API key in the source
code and, you know, now we're, we're, uh, having, uh, having fun with it."
Todd Kane: Yep.
Okay.
This, uh, I mean, this is, this is fascinating.
I love this stuff, man.
So, uh, really appreciate you coming on, um, uh, kind of chatting through some of
the, the scenarios here, the potential future that we face, and some of the
things we need to, we need to get ready for and, and prepare some of the, the
infrastructure and security sets for.
Uh, any, any last-minute, uh, little tidbits or, or shout-outs
you would, uh, you would throw down before, before we head out here?
AIex: Yeah, I th- I, I think again, just for your listeners is, y- you
know, the, the, the, the debt that we were sweeping under the rug because,
you know, the, the exposure was low.
It was, you know, behind the firewall and whatnot.
Like, that, that protection is eroded substantially.
And it, and, and now with the velocity of what, um, an AI system can do is like, you
know, we as blue teamers are gonna have a major problem keeping up with all the
findings of a problem, you know, it…
assuming the AI is used just for good.
Imagine that every day there's 1,000 new patches for Windows
and Linux and all that.
Like, we've got a, we've got a, a… We are the bottleneck now, and if we
don't fix it, we have an exposure.
Um, if we do fix it, we need an entire army to do it.
So there also needs to be, um, a, you know, recognizing that we need a, a
better way of maintaining software and keeping it as secure as possible,
recognizing it's never going to be secure.
But you don't have to be faster than the hacker, you just have to be faster than
the slowest, uh, victim, unfortunately.
Todd Kane: Yeah.
Well, it's a wild world.
I appreciate you coming on, Alex.
Always great to chat with you
AIex: Likewise