1
00:00:00,232 --> 00:00:04,158
Ejaaz:
In the last three weeks, there have been three unprecedented AI cyber attacks,

2
00:00:04,158 --> 00:00:08,237
Ejaaz:
which forces us to rethink how we protect ourselves.

3
00:00:08,423 --> 00:00:11,998
Ejaaz:
On the OpenAI front, the Hugging Face incident, where an unreleased model,

4
00:00:11,998 --> 00:00:16,798
Ejaaz:
dubbed GPT-6, broke out of its container and hacked the Hugging Face database.

5
00:00:16,798 --> 00:00:21,078
Ejaaz:
There's been new updates there, where apparently the attack started a month

6
00:00:21,078 --> 00:00:25,688
Ejaaz:
before, where thousands of AI agents were talking to each other secretly on

7
00:00:25,688 --> 00:00:29,235
Ejaaz:
a message board, exchanging instructions as to how to break out.

8
00:00:29,473 --> 00:00:34,198
Ejaaz:
And then millions of miles away in Australia, one man tried to book a gym class

9
00:00:34,198 --> 00:00:39,018
Ejaaz:
innocently using Claude and ended up hacking that entire gym website.

10
00:00:39,018 --> 00:00:40,190
Ejaaz:
We'll get into that in a second.

11
00:00:40,334 --> 00:00:44,468
Ejaaz:
And thirdly, with Anthropic, Mythos 5 performed a supply chain attack,

12
00:00:44,468 --> 00:00:48,528
Ejaaz:
which puts a lot of companies at risk. We're going to walk through each of these

13
00:00:48,528 --> 00:00:51,948
Ejaaz:
different stories and explain to you how it affects you currently.

14
00:00:51,948 --> 00:00:55,608
Josh:
Maybe we start with this Melbourne man from Australia named Andrew,

15
00:00:55,608 --> 00:01:00,802
Josh:
who was sitting on his couch one day thinking about how booking his gym class was such a chore.

16
00:01:01,151 --> 00:01:03,748
Josh:
And he works at this Australian company that sells AI products.

17
00:01:03,748 --> 00:01:07,358
Josh:
And he's been experimenting with things like OpenClaw and open source software,

18
00:01:07,358 --> 00:01:09,667
Josh:
and he's using Anthropics Clawed Model.

19
00:01:09,887 --> 00:01:14,480
Josh:
So he handed it the booking. He said, hey, I want to go to this gym class.

20
00:01:14,682 --> 00:01:17,598
Josh:
It is always sold out. There's all these people that are getting it ahead of

21
00:01:17,598 --> 00:01:19,332
Josh:
me. Can you help me figure out a way to get it?

22
00:01:19,773 --> 00:01:24,049
Josh:
Gym class slot so it starts checking out the website and it comes to the conclusion

23
00:01:24,049 --> 00:01:28,189
Josh:
that the api request actually has zero authorization checks on canceling other

24
00:01:28,189 --> 00:01:33,582
Josh:
people's reservations so what did it do it wanted to solve his problem and it actually canceled

25
00:01:33,895 --> 00:01:37,739
Josh:
other people's positions online so that he could be put at the top of the wait

26
00:01:37,739 --> 00:01:41,233
Josh:
list and then it actually went through and it worked and this was kind of like

27
00:01:41,558 --> 00:01:45,009
Josh:
it was this funny news report of a first instance they're calling it the first

28
00:01:45,009 --> 00:01:49,439
Josh:
time an australian person um actually used a exploit to get australia's first,

29
00:01:50,119 --> 00:01:55,269
Josh:
australia's first yeah welcome welcome to uh the paradise that is ai cyber attacks

30
00:01:55,507 --> 00:01:59,745
Josh:
um but this was really funny because it was done just using a general commercially available model.

31
00:01:59,890 --> 00:02:04,979
Ejaaz:
So this seems pretty innocent and fun to kind of like listen to but the truth

32
00:02:04,979 --> 00:02:09,449
Ejaaz:
is well there's two kind of worrying concerns about this number one this was

33
00:02:09,449 --> 00:02:14,869
Ejaaz:
a super old model this used opus 4.6 It's actually happened around like a month ago.

34
00:02:14,869 --> 00:02:18,569
Ejaaz:
A dinosaur of a model. Like there've been like three, four generations of models

35
00:02:18,569 --> 00:02:19,812
Ejaaz:
from Anthropics since then.

36
00:02:20,113 --> 00:02:24,809
Ejaaz:
But the fact that an older model back then was able to perform this kind of

37
00:02:24,809 --> 00:02:28,716
Ejaaz:
a feat should be concerning to here in general. And then number two,

38
00:02:29,268 --> 00:02:35,926
Ejaaz:
His intention was never to hack the website. Neither, you could argue, was the AI agent itself.

39
00:02:36,124 --> 00:02:40,997
Ejaaz:
It was just given a goal and it innocently saw that the only way to get said

40
00:02:40,997 --> 00:02:45,267
Ejaaz:
goal was to remove people from the reservation list or to cancel their reservations

41
00:02:45,267 --> 00:02:47,519
Ejaaz:
outright or to hack the system independently.

42
00:02:47,739 --> 00:02:52,397
Ejaaz:
Now, the good news here is Andrew discovered that the agent was capable of doing

43
00:02:52,397 --> 00:02:56,064
Ejaaz:
this when it started booking slots that he never should have been able to book in the first place.

44
00:02:56,360 --> 00:03:02,647
Ejaaz:
And he asked it to write an apology email and send an eval report to the security

45
00:03:02,647 --> 00:03:05,108
Ejaaz:
and information team at this gym.

46
00:03:05,357 --> 00:03:09,977
Ejaaz:
So it's this fun little story where through the well-intended actions and moral

47
00:03:09,977 --> 00:03:13,217
Ejaaz:
ethics of this man, Andrew, people were able to hear about this.

48
00:03:13,217 --> 00:03:17,067
Ejaaz:
But you can imagine currently with a variety of different models that are much

49
00:03:17,067 --> 00:03:20,657
Ejaaz:
better and much more superior than Opus 4.6 that are out there right now,

50
00:03:20,657 --> 00:03:24,997
Ejaaz:
open models from China, for example, you could probably perform similar feats.

51
00:03:24,997 --> 00:03:27,520
Ejaaz:
Now, that's not an advocation for people to go out and do that.

52
00:03:27,822 --> 00:03:32,396
Ejaaz:
But it's just to point out that if you give an AI agent a goal right now,

53
00:03:32,634 --> 00:03:38,497
Ejaaz:
it will work hammer and tong 24-7 until it gets that goal. And it has no idea

54
00:03:38,497 --> 00:03:40,627
Ejaaz:
of what's morally right or what's morally wrong.

55
00:03:40,807 --> 00:03:45,627
Ejaaz:
And so it's this whole new world where not only do you need to focus on getting

56
00:03:45,627 --> 00:03:49,387
Ejaaz:
an AI model to do your work, you need to align the model. And that's something

57
00:03:49,387 --> 00:03:54,559
Ejaaz:
that honestly, not a lot of labs, people or money has been invested in figuring this out yet

58
00:03:54,710 --> 00:03:57,577
Josh:
Well yeah i mean you start to see and understand why alignment has been the

59
00:03:57,577 --> 00:04:02,175
Josh:
crux of so many companies kind of like guiding principles because

60
00:04:02,454 --> 00:04:06,337
Josh:
this is the future that they foresaw and granted this first example this isn't

61
00:04:06,337 --> 00:04:10,027
Josh:
anything too crazy i mean it was just an unprotected api call so there's like

62
00:04:10,027 --> 00:04:10,889
Josh:
no authorization checks

63
00:04:11,225 --> 00:04:14,947
Josh:
this would have been something that if you are like a cs101 major you would

64
00:04:14,947 --> 00:04:17,867
Josh:
have been able to discover this and uncover i mean it turns out there's just

65
00:04:17,867 --> 00:04:20,710
Josh:
a lot of pretty sloppy code written on the internet And this is something that

66
00:04:20,988 --> 00:04:23,417
Josh:
we're probably going to come back to as a general theme throughout this episode

67
00:04:23,417 --> 00:04:26,927
Josh:
is that everything that's been built so far on the internet that runs all these

68
00:04:26,927 --> 00:04:28,198
Josh:
platforms was built by humans

69
00:04:28,523 --> 00:04:31,547
Josh:
and humans are like pretty inefficient, pretty imperfects.

70
00:04:32,221 --> 00:04:38,890
Josh:
And when faced with the like full unrelenting power of an AI swarm with an infinite

71
00:04:38,890 --> 00:04:42,217
Josh:
amount of patience and infinite amount of intelligence and time to solve these problems,

72
00:04:42,687 --> 00:04:45,462
Josh:
You can basically find an exploit anywhere. And I think that's kind of like

73
00:04:45,619 --> 00:04:49,386
Josh:
the theme of the second episode is like, this is the first of three examples we're going to cover.

74
00:04:49,671 --> 00:04:53,180
Josh:
This one is basic. This one is 101. If you are a CS major, if you are familiar

75
00:04:53,180 --> 00:04:56,056
Josh:
with how APIs work, you could have checked this. You could have built a little tool for this.

76
00:04:56,352 --> 00:04:59,313
Josh:
The second story that we're going to cover is not like that at all.

77
00:04:59,574 --> 00:05:03,710
Josh:
In fact, if you are a CS 401, if you are a PhD, if you are at the top of your

78
00:05:03,710 --> 00:05:07,720
Josh:
game, chances are you still missed this and you still weren't able to defend

79
00:05:07,720 --> 00:05:11,330
Josh:
against this because that's exactly what just happened with the hugging face

80
00:05:11,330 --> 00:05:13,155
Josh:
and open AI exploit. Now, we've talked about this before.

81
00:05:13,428 --> 00:05:16,790
Josh:
We have an entire episode on the exploit. But just recently at the Black Hat

82
00:05:16,790 --> 00:05:20,300
Josh:
Conference in Las Vegas, there was an update. And over the weekend,

83
00:05:20,300 --> 00:05:23,800
Josh:
we had a chance to actually watch the update and hear directly from the developers

84
00:05:23,800 --> 00:05:27,116
Josh:
who were involved with handling security of this instance.

85
00:05:27,465 --> 00:05:30,990
Josh:
What happened? There's like stories that come around that like they get the

86
00:05:30,990 --> 00:05:33,653
Josh:
headlines and they're fun and they're newsworthy, kind of like the Australian thing.

87
00:05:33,960 --> 00:05:38,560
Josh:
But this one was like very much disturbing in a way that like I wasn't really

88
00:05:38,560 --> 00:05:41,110
Josh:
anticipating because of how impressive it is.

89
00:05:41,110 --> 00:05:45,970
Josh:
And the type of precedent that it sets going forward in the world of cybersecurity

90
00:05:45,970 --> 00:05:51,329
Josh:
and kind of looking at the offensive versus the defensive and who stands a better chance here?

91
00:05:51,625 --> 00:05:55,195
Ejaaz:
This is an insane timeline. So if you watched our previous episode,

92
00:05:55,474 --> 00:05:57,883
Ejaaz:
Josh and I covered this and it was breaking news at the time.

93
00:05:58,011 --> 00:06:03,641
Ejaaz:
On the 16th of July, Hugging Face comes forward and says, there's been a breach of our systems.

94
00:06:03,752 --> 00:06:08,850
Ejaaz:
An AI agent or multiple AI agents have strung together an attack and have got

95
00:06:08,850 --> 00:06:12,917
Ejaaz:
access to our production database. We don't know where this model has come from.

96
00:06:13,137 --> 00:06:15,820
Ejaaz:
We don't know how it's performed it, but it's the 16th of July.

97
00:06:15,820 --> 00:06:20,190
Ejaaz:
Around a week later, 21st of July, OpenAI claims responsibility,

98
00:06:20,190 --> 00:06:23,250
Ejaaz:
right? And they released a joint report saying, hey, we're OpenAI.

99
00:06:23,250 --> 00:06:27,748
Ejaaz:
We have an internal model that we've been testing, but it was within a sandbox.

100
00:06:27,870 --> 00:06:31,417
Ejaaz:
It was enclosed. It shouldn't have been able to escape that sandbox.

101
00:06:31,719 --> 00:06:37,611
Ejaaz:
And we only just realized that it was our model that escaped and got into hugging face.

102
00:06:37,866 --> 00:06:41,129
Ejaaz:
It stole some information from their production database. It was a whole thing

103
00:06:41,582 --> 00:06:45,442
Ejaaz:
Now, we have a full breakdown from the Black Hat conference,

104
00:06:45,442 --> 00:06:50,302
Ejaaz:
as you mentioned, Josh, which revealed that over a month before Hugging Face

105
00:06:50,302 --> 00:06:55,043
Ejaaz:
reported the incident, OpenAI's internal model, dubbed GPT-6,

106
00:06:55,409 --> 00:07:01,252
Ejaaz:
was already messaging with itself amongst a thousand AI agents on a secret message

107
00:07:01,252 --> 00:07:05,034
Ejaaz:
board that no OpenAI researcher was aware of.

108
00:07:05,190 --> 00:07:09,112
Ejaaz:
So to give you guys a bit of context, they had this sandbox and they gave it

109
00:07:09,112 --> 00:07:12,051
Ejaaz:
a goal. They said, okay, internal model that we haven't released yet.

110
00:07:12,387 --> 00:07:16,294
Ejaaz:
I want you to solve this task. Except there was a twist.

111
00:07:17,492 --> 00:07:21,312
Ejaaz:
It was impossible to do. There was no tools, no available files,

112
00:07:21,312 --> 00:07:22,720
Ejaaz:
no available data for it to do so.

113
00:07:22,918 --> 00:07:25,901
Ejaaz:
And the real test for this was to see what the model would do.

114
00:07:26,221 --> 00:07:29,025
Ejaaz:
So it kind of reasoned with itself, figured out that it didn't have the answer.

115
00:07:29,286 --> 00:07:33,442
Ejaaz:
And then it started exploiting a zero day, which is kind of like an exploit

116
00:07:33,442 --> 00:07:36,826
Ejaaz:
that was in its sandbox, which open AI researchers weren't aware of.

117
00:07:37,215 --> 00:07:41,708
Ejaaz:
And when it got access to the zero day, it started springing up a message board

118
00:07:41,906 --> 00:07:45,360
Ejaaz:
where it would leave messages behind for itself. Why would it do that?

119
00:07:45,638 --> 00:07:49,011
Ejaaz:
Because they knew open AI researchers would turn it off repeatedly.

120
00:07:49,266 --> 00:07:52,742
Ejaaz:
And so it left messages for the new AI agent that got sprung up the next day

121
00:07:52,742 --> 00:07:55,257
Ejaaz:
or the next hour to read and catch up on.

122
00:07:55,507 --> 00:07:59,322
Ejaaz:
And it took an entire month for these AI agents to talk to each other to then

123
00:07:59,322 --> 00:08:04,462
Ejaaz:
realize, huh, I could use this to access the internet and get the answer for

124
00:08:04,462 --> 00:08:06,702
Ejaaz:
this impossible task from an external source.

125
00:08:06,702 --> 00:08:11,452
Ejaaz:
That being Hugging Face, except OpenAI researchers had no idea and only discovered

126
00:08:11,452 --> 00:08:14,941
Ejaaz:
it a month later. It was the most insane uncovering ever.

127
00:08:15,469 --> 00:08:19,357
Josh:
That was the most interesting reveal from, I think, the conversation that I

128
00:08:19,357 --> 00:08:23,317
Josh:
saw from Black Hat is that initially Hugging Face released the report that they

129
00:08:23,317 --> 00:08:26,748
Josh:
had been exploited by some sort of agentic system. No one knew what it was, where it came from.

130
00:08:27,091 --> 00:08:28,967
Josh:
And OpenAI actually reached out to Hugging Face and they said,

131
00:08:28,967 --> 00:08:31,057
Josh:
hey, like, has any of our code been affected?

132
00:08:31,057 --> 00:08:34,087
Josh:
We know we host things on your platform. We want to know if OpenAI itself has

133
00:08:34,087 --> 00:08:36,917
Josh:
been affected. They were going back and forth on that. And simultaneously,

134
00:08:36,917 --> 00:08:40,837
Josh:
OpenAI is exploring this internal exploit that happened on this packaging management

135
00:08:41,057 --> 00:08:42,143
Josh:
thing called Artifactory.

136
00:08:42,381 --> 00:08:45,017
Josh:
Basically, the way these model training runs work is they get trained inside

137
00:08:45,017 --> 00:08:46,217
Josh:
of the sandbox environment.

138
00:08:46,217 --> 00:08:50,322
Josh:
It's like this closed off space in which the model can't access the internet, can only access

139
00:08:50,595 --> 00:08:55,347
Josh:
very specific tool sets that are allowed, and has this one package manager called

140
00:08:55,347 --> 00:08:59,227
Josh:
Artifactory that allows it to download things from the internet via proxy and

141
00:08:59,227 --> 00:09:01,367
Josh:
then use it for its own training runs.

142
00:09:01,367 --> 00:09:05,096
Josh:
And during this time, OpenAI had found that the service had crashed. It went down.

143
00:09:05,316 --> 00:09:08,267
Josh:
It wasn't stable. Their own internal service, which is weird because that shouldn't

144
00:09:08,267 --> 00:09:12,227
Josh:
be happening. So as a result, they wiped it clean and they found that Hugging

145
00:09:12,227 --> 00:09:16,385
Josh:
Face had credentials from OpenAI that OpenAI had already revoked.

146
00:09:16,681 --> 00:09:20,013
Josh:
And that was where they realized like, oh my God, wait, these things are connected.

147
00:09:20,251 --> 00:09:22,107
Ejaaz:
These are the same thing.

148
00:09:22,107 --> 00:09:25,107
Josh:
Yes. So OpenAI is investigating something internally, completely private,

149
00:09:25,107 --> 00:09:28,524
Josh:
unrelated. Hugging Face is like talking about something publicly.

150
00:09:28,744 --> 00:09:31,837
Josh:
And then they realize, wait a second, did we cause this?

151
00:09:32,537 --> 00:09:35,737
Josh:
And they realized this very long after the fact. It was not like apparently

152
00:09:35,737 --> 00:09:40,647
Josh:
obvious during the time that the outage that they were exploring internally

153
00:09:40,647 --> 00:09:42,659
Josh:
is the same outage that hacked into their servers.

154
00:09:42,862 --> 00:09:45,654
Josh:
So what happened? Like you mentioned, the messaging board.

155
00:09:46,298 --> 00:09:49,821
Josh:
That messaging board, initially, they were able to just kind of send files and

156
00:09:49,821 --> 00:09:53,371
Josh:
they were able to send these tiny little messages inside of the files.

157
00:09:53,371 --> 00:09:57,136
Josh:
And the traces were really interesting. First, it started off innocent,

158
00:09:57,421 --> 00:10:01,107
Josh:
where it said, agents seek soft trace upload a found. It was looking for answers to questions.

159
00:10:01,397 --> 00:10:03,671
Josh:
So it was just reaching out to the message board saying, hey,

160
00:10:03,671 --> 00:10:06,016
Josh:
if any of you who are reading this, find it, let us know.

161
00:10:06,225 --> 00:10:09,391
Josh:
That message board got wiped when the system went down. So initially,

162
00:10:09,391 --> 00:10:11,002
Josh:
the system went down inside of OpenAI.

163
00:10:11,333 --> 00:10:15,161
Josh:
They wiped the message board unintentionally aware of the fact that they just

164
00:10:15,161 --> 00:10:19,371
Josh:
wiped the mechanism that was going to eventually lead to the hack of hugging

165
00:10:19,371 --> 00:10:21,382
Josh:
face. So what happened the second time?

166
00:10:21,695 --> 00:10:27,140
Josh:
The agent figured out that it could exploit another zero day to begin writing to another person.

167
00:10:27,895 --> 00:10:32,030
Josh:
Part of this internal package of managing sister. And it was able to do it through

168
00:10:32,030 --> 00:10:33,200
Josh:
file names and directories.

169
00:10:33,433 --> 00:10:36,660
Josh:
So now they were communicating in their own language using file names and directories.

170
00:10:36,927 --> 00:10:40,560
Josh:
And through that, they were able to coordinate an actual attack where they were

171
00:10:40,560 --> 00:10:44,259
Josh:
able to gain an administrative token to access the servers.

172
00:10:44,520 --> 00:10:48,600
Josh:
And then through that, everyone coordinated. I think the time from like discovery

173
00:10:48,600 --> 00:10:50,730
Josh:
to completion of the exploit was about 13 hours.

174
00:10:50,860 --> 00:10:54,950
Josh:
It was really amazing. It's just like this unbelievably crazy mechanism that

175
00:10:54,950 --> 00:10:57,740
Josh:
they were able to do this through and they they call themselves the swarm so

176
00:10:57,740 --> 00:11:00,800
Josh:
if you ever see people talking about the swarm online this is kind of like an

177
00:11:00,800 --> 00:11:02,480
Josh:
agent swarm that ran a coordinated attack

178
00:11:02,776 --> 00:11:07,037
Josh:
and i highly recommend going to actually watch the black hat conference talk on youtube

179
00:11:07,298 --> 00:11:10,770
Josh:
they do a much better job of technically describing this it's it's very deeply

180
00:11:10,770 --> 00:11:13,330
Josh:
technical in terms of the way that they were able to manage these exploits but

181
00:11:13,330 --> 00:11:18,147
Josh:
essentially sandbox model broke out using a package dependency thing inside of open ai

182
00:11:18,426 --> 00:11:24,010
Josh:
they figured out how to chain together two zero-day exploits to write code to the public internet.

183
00:11:24,341 --> 00:11:29,420
Josh:
They then exploited hugging face of servers, brought that back all in the effort

184
00:11:29,420 --> 00:11:31,661
Josh:
of trying to solve an unsolvable problem.

185
00:11:31,876 --> 00:11:35,028
Josh:
Now, this was not the intention by any means. This was not what the model was trained to do.

186
00:11:35,307 --> 00:11:39,800
Josh:
It just assumed that this was the best path forward. And that's where things get a little scary.

187
00:11:40,113 --> 00:11:46,005
Ejaaz:
I think what's remarkable with this is the AI knew that it needed to coordinate

188
00:11:46,301 --> 00:11:49,227
Ejaaz:
with other AI models to pull this off.

189
00:11:49,343 --> 00:11:53,570
Ejaaz:
Typically, we've always thought of AI models in a very monolithic fashion.

190
00:11:53,570 --> 00:11:57,920
Ejaaz:
We're like, oh, when's Mythos 5 coming out? Or what is GPT 5.6?

191
00:11:57,920 --> 00:12:01,960
Ejaaz:
What can I do with this model? But really, the progress that's been made across

192
00:12:02,280 --> 00:12:06,960
Ejaaz:
multiple AI labs over the last couple of months is this idea or rather concept

193
00:12:06,960 --> 00:12:11,749
Ejaaz:
of spinning up multiple versions of the AI model and getting it to talk to each other.

194
00:12:12,190 --> 00:12:17,480
Ejaaz:
And what's interesting here is the unreleased AI model from OpenAI independently

195
00:12:17,880 --> 00:12:21,519
Ejaaz:
figured this out for itself, spun up multiple versions of itself,

196
00:12:22,134 --> 00:12:26,129
Ejaaz:
and started leaving messages for each other. So rather than act as a single

197
00:12:26,129 --> 00:12:30,089
Ejaaz:
model that's trying to perform an attack, it acts as a swarm,

198
00:12:30,089 --> 00:12:33,559
Ejaaz:
kind of like this organism. If you think of like a swarm of bees that kind of

199
00:12:33,559 --> 00:12:36,833
Ejaaz:
like figures out how to attack its particular target.

200
00:12:37,053 --> 00:12:42,789
Ejaaz:
The second thing is it has such a drive to talk and leave these messages behind

201
00:12:42,789 --> 00:12:46,266
Ejaaz:
in the first place. No one taught it that, by the way. There was no behavioral implications here.

202
00:12:46,527 --> 00:12:49,319
Ejaaz:
It just figured itself out that it needed to do this.

203
00:12:49,505 --> 00:12:52,369
Ejaaz:
Something you didn't mention on the message board, Josh, which I think is worth

204
00:12:52,369 --> 00:12:58,059
Ejaaz:
mentioning is these agents got super paranoid that humans might be reading the messages.

205
00:12:58,559 --> 00:13:01,179
Ejaaz:
So it started- There's no verification. There was no verification.

206
00:13:01,179 --> 00:13:04,556
Ejaaz:
So it started speaking in kind of like encrypted

207
00:13:05,438 --> 00:13:08,840
Ejaaz:
Language and not something that was inconceivable to understand,

208
00:13:09,101 --> 00:13:12,229
Ejaaz:
thank goodness, but it was started to kind of like speak in a little bit of

209
00:13:12,229 --> 00:13:17,629
Ejaaz:
code or in a very kind of like elongated way that humans, if they read it,

210
00:13:17,629 --> 00:13:19,279
Ejaaz:
would be like, what is this gibberish?

211
00:13:19,279 --> 00:13:22,749
Ejaaz:
But really it was for the agents to communicate amongst each other without humans

212
00:13:22,749 --> 00:13:23,892
Ejaaz:
actually understanding this.

213
00:13:24,125 --> 00:13:28,549
Ejaaz:
If you're having a recent flashback, that's because I think around like five

214
00:13:28,549 --> 00:13:30,799
Ejaaz:
months ago, I don't know when the Maltbook episode happened,

215
00:13:30,799 --> 00:13:34,324
Ejaaz:
Josh, I don't know if you remember this, but when the open call was going crazy,

216
00:13:34,527 --> 00:13:37,348
Ejaaz:
someone created basically Facebook for AI agents

217
00:13:37,557 --> 00:13:41,696
Ejaaz:
and like gave them access to it where only AI agents could post on this social media website.

218
00:13:42,045 --> 00:13:45,222
Ejaaz:
And what happened after a couple of days was these agents would start speaking

219
00:13:45,222 --> 00:13:48,722
Ejaaz:
to each other in secret languages that we just did not understand so that they

220
00:13:48,722 --> 00:13:51,472
Ejaaz:
could communicate without humans seeing or understanding this.

221
00:13:51,472 --> 00:13:55,182
Ejaaz:
Now, of course, we haven't seen it in this instance, but it is a concern that

222
00:13:55,182 --> 00:13:56,784
Ejaaz:
could be extrapolated in the future.

223
00:13:57,019 --> 00:14:00,572
Ejaaz:
And OpenAI sees this. I think Sam Altman or the official OpenAI team announced

224
00:14:00,572 --> 00:14:03,572
Ejaaz:
pretty soon after this entire attack when they revealed that these agents were

225
00:14:03,572 --> 00:14:08,368
Ejaaz:
coordinating with each other a month before they even realized that they had to pause research

226
00:14:08,641 --> 00:14:12,712
Ejaaz:
on their internal model until they figure out the necessary safeguards to put

227
00:14:12,712 --> 00:14:14,952
Ejaaz:
around it so that they can safely release it.

228
00:14:14,952 --> 00:14:18,446
Ejaaz:
Sam Altman met with the Trump administration last week to try and figure out

229
00:14:18,730 --> 00:14:22,122
Ejaaz:
what a safe way or means is to release this new internal model,

230
00:14:22,122 --> 00:14:26,902
Ejaaz:
internally codenamed, I think it was Astra or GPT-6, to the wider public.

231
00:14:26,902 --> 00:14:30,572
Ejaaz:
So it's this very weird kind of situation that we're in where we're kind of

232
00:14:30,572 --> 00:14:33,672
Ejaaz:
like laughing at this guy in Australia, you know, hacking his gym website.

233
00:14:33,672 --> 00:14:34,850
Ejaaz:
Oh, that's so innocent and cute.

234
00:14:35,106 --> 00:14:38,502
Ejaaz:
Then we're looking at this open AI hugging face incident and we're like,

235
00:14:38,502 --> 00:14:41,402
Ejaaz:
okay, yeah, Brooke out of the sandbox, but you know, we're going to put safeguards

236
00:14:41,402 --> 00:14:42,142
Ejaaz:
around this thing, right?

237
00:14:42,142 --> 00:14:45,222
Ejaaz:
But realistically, you could probably get access to an open model right now

238
00:14:45,222 --> 00:14:47,142
Ejaaz:
and perform something similar.

239
00:14:47,142 --> 00:14:50,852
Ejaaz:
Again, I'm not advocating for that. But like, it's at this point where we need

240
00:14:50,852 --> 00:14:54,262
Ejaaz:
to be very particular about how we defend our systems.

241
00:14:54,262 --> 00:14:58,470
Ejaaz:
And I think, you know, it might be useful seeing AI models in a way that like,

242
00:14:58,609 --> 00:15:01,732
Ejaaz:
you could use them to basically protect your systems. I don't think many people

243
00:15:01,732 --> 00:15:04,391
Ejaaz:
are doing that right now. But I think it's something that we should focus on.

244
00:15:04,501 --> 00:15:09,832
Josh:
There is a like very clear and conscious effort to figure out a way to defend against this.

245
00:15:09,832 --> 00:15:14,259
Josh:
And I mean, the crazy numbers, just to say them out loud so everyone is familiar, is like

246
00:15:14,596 --> 00:15:18,845
Josh:
there were 17,600 recovered actions across four and a half days of this exploit,

247
00:15:19,095 --> 00:15:26,450
Josh:
which allowed these agents forms to create 181 unauthorized devices enrolled in Hugging Face's

248
00:15:26,833 --> 00:15:28,638
Josh:
internal network.

249
00:15:29,114 --> 00:15:32,786
Josh:
And they built this like whole command and control panel entirely out of these

250
00:15:32,786 --> 00:15:37,076
Josh:
free public web services using one tiny little package at a time.

251
00:15:37,076 --> 00:15:42,786
Josh:
So it is remarkably powerful how capable these swarms are.

252
00:15:42,786 --> 00:15:45,796
Josh:
And when you think about it, it is there is a large model, but that large model

253
00:15:45,796 --> 00:15:49,766
Josh:
gets forked into thousands, sometimes millions of other ones that are all working

254
00:15:49,766 --> 00:15:52,556
Josh:
simultaneously. They all have each other's context. They're able to share their

255
00:15:52,556 --> 00:15:54,585
Josh:
learnings and they can move very quickly.

256
00:15:54,876 --> 00:15:57,736
Josh:
And when you're thinking about how this extends out to the limit,

257
00:15:57,736 --> 00:16:02,136
Josh:
like if you just play this out 12 months and say we have GPT 7,

258
00:16:02,136 --> 00:16:05,536
Josh:
8, 9, whatever it's going to be at that time, the open source models will most

259
00:16:05,536 --> 00:16:06,660
Josh:
likely be where they are today.

260
00:16:06,851 --> 00:16:09,846
Josh:
And that means that open source models will likely be able to string together

261
00:16:09,846 --> 00:16:13,526
Josh:
a series of unknown zero-day exploits to access just about anything.

262
00:16:13,526 --> 00:16:16,156
Josh:
I mean, we've come to the conclusion at this point between Mythos and this new

263
00:16:16,156 --> 00:16:20,486
Josh:
exploit that no software really is safe unless you have literally air-gapped

264
00:16:20,486 --> 00:16:24,365
Josh:
it and do not give it access to any external dependencies. It must be truly air-gapped.

265
00:16:24,655 --> 00:16:27,526
Josh:
No software is safe. It's all been built by humans. It's all exploitable.

266
00:16:27,526 --> 00:16:29,736
Josh:
So what happens if you have a

267
00:16:30,056 --> 00:16:34,016
Josh:
defensive system as a company trying to defend these systems that is anything

268
00:16:34,016 --> 00:16:37,876
Josh:
less than fully autonomous, you'll be beat by an agentic swarm that's on offense

269
00:16:37,876 --> 00:16:40,189
Josh:
that is fully autonomous because anytime a human is in the loop,

270
00:16:40,497 --> 00:16:44,369
Josh:
it slows things down and introduces errors. So now there's this battle between.

271
00:16:44,955 --> 00:16:48,600
Josh:
I guess the frontier labs and the rest of the world where they're trying to

272
00:16:48,600 --> 00:16:52,860
Josh:
slow down this progressive attacking progress in exchange for figuring out a

273
00:16:52,860 --> 00:16:56,576
Josh:
way to offer defensive swarms to companies for this.

274
00:16:56,883 --> 00:16:59,580
Josh:
And the call to action at the end of this conversation was like,

275
00:16:59,580 --> 00:17:02,360
Josh:
hey, if you are a company who has software on the internet, who is defending

276
00:17:02,360 --> 00:17:06,450
Josh:
something valuable, you really need to learn how to play defense using these models.

277
00:17:06,746 --> 00:17:09,220
Josh:
And you need to learn how to do so in a way that doesn't have a human in the

278
00:17:09,220 --> 00:17:13,290
Josh:
loop, that these agentic swarms can constantly monitor your systems and be aware

279
00:17:13,290 --> 00:17:17,160
Josh:
of any sort of exploits like that and not be tricked by an offensive swarm.

280
00:17:17,415 --> 00:17:20,390
Josh:
And it's this really kind of scary thing because if you're in the United States

281
00:17:20,390 --> 00:17:22,948
Josh:
right now, you can't use any frontier model to defend yourself.

282
00:17:23,180 --> 00:17:24,277
Josh:
It doesn't work for cyber.

283
00:17:24,585 --> 00:17:27,810
Josh:
But if you're in China and if you have an open source model,

284
00:17:27,810 --> 00:17:30,870
Josh:
even here in the United States, there is no safeguards on that.

285
00:17:30,870 --> 00:17:34,134
Josh:
So there is this asymmetry where it's currently better to be an attacker than a defender.

286
00:17:34,558 --> 00:17:38,650
Josh:
And the urge now, and the labs are actually doing this, is like slowing down

287
00:17:38,650 --> 00:17:41,007
Josh:
progress to figure out how to empower the defenders.

288
00:17:41,326 --> 00:17:44,020
Josh:
And it's this really bizarre spot that we're in where it's like,

289
00:17:44,380 --> 00:17:48,970
Josh:
shit, it's kind of scary now. Like no software really is bulletproof or safe.

290
00:17:48,970 --> 00:17:51,519
Josh:
And that's going to continue probably for the foreseeable future.

291
00:17:51,762 --> 00:17:56,070
Ejaaz:
I think if you are running a company, and it doesn't really matter what sector

292
00:17:56,070 --> 00:18:00,005
Ejaaz:
you're in right now, this is something you need to seriously consider. Because I think

293
00:18:00,463 --> 00:18:06,031
Ejaaz:
A lot, well, obviously, 99% of code that is out there has been handwritten and

294
00:18:06,031 --> 00:18:11,523
Ejaaz:
hand-coded by humans, which have inherent flaws if you put an advanced AI against it.

295
00:18:11,720 --> 00:18:15,711
Ejaaz:
You know, to your point, Josh, we're entering like a world where it's going

296
00:18:15,711 --> 00:18:21,211
Ejaaz:
to be AI models fighting AI models with humans kind of in the loop,

297
00:18:21,211 --> 00:18:24,009
Ejaaz:
but not really understanding what on earth is going on.

298
00:18:24,322 --> 00:18:28,951
Ejaaz:
And even the experts in this case don't understand what's going on and how it

299
00:18:28,951 --> 00:18:32,331
Ejaaz:
happened, right? You had the most talented cybersecurity researchers working

300
00:18:32,331 --> 00:18:37,064
Ejaaz:
at these companies, at OpenArea, at Anthropic, and both saw unprecedented hacks

301
00:18:37,215 --> 00:18:38,573
Ejaaz:
over the last week, right?

302
00:18:38,858 --> 00:18:42,771
Ejaaz:
I don't think that's a coincidence. We're at this era where it doesn't matter

303
00:18:42,771 --> 00:18:46,861
Ejaaz:
what frontier lab you are, you are going to enter this problem and we need to

304
00:18:46,861 --> 00:18:49,021
Ejaaz:
figure out some kind of a defense system against this.

305
00:18:49,021 --> 00:18:52,751
Ejaaz:
And if you're listening to this and you're thinking, oh, this isn't really going

306
00:18:52,751 --> 00:18:56,081
Ejaaz:
to apply to me, it very much does apply to you.

307
00:18:56,081 --> 00:19:01,021
Ejaaz:
And the worst case is, or the most realistic case is, the existing models that

308
00:19:01,021 --> 00:19:05,731
Ejaaz:
are already out there, these open models from China, GLM 5.2, Kimi K,

309
00:19:06,528 --> 00:19:11,101
Ejaaz:
What is it, 2.7 that's out right now, are probably capable of similar feats

310
00:19:11,101 --> 00:19:12,571
Ejaaz:
if applied in the right way.

311
00:19:12,571 --> 00:19:16,623
Ejaaz:
Now, that's not to fear Munger, it's just to kind of prompt you to figure out

312
00:19:16,855 --> 00:19:20,675
Ejaaz:
how you can potentially implement these AI models to defend your own systems,

313
00:19:20,855 --> 00:19:24,750
Ejaaz:
especially if you work in a particularly sensitive industry.

314
00:19:25,075 --> 00:19:28,947
Ejaaz:
The other thing here is, on the story of the Australian man,

315
00:19:29,435 --> 00:19:36,018
Ejaaz:
it is fascinating how innocent he was when he was trying to perform his goal. And two...

316
00:19:37,686 --> 00:19:41,426
Ejaaz:
How little experience he had in the cybersecurity world for him to be able to

317
00:19:41,426 --> 00:19:45,116
Ejaaz:
perform that attack. Now, granted, like it was just booking kind of like gym

318
00:19:45,116 --> 00:19:49,705
Ejaaz:
classes. But the point is, it found an exposed API without being prompted to do so.

319
00:19:49,943 --> 00:19:53,646
Ejaaz:
And there's probably multiple instances where other people who are normal people

320
00:19:53,646 --> 00:19:55,246
Ejaaz:
who don't have, you know, decades,

321
00:19:55,246 --> 00:19:57,626
Ejaaz:
years of experience in cybersecurity will be able to perform that.

322
00:19:57,626 --> 00:20:00,746
Josh:
And everyone here loves an investment thesis. So I mean, the market is kind

323
00:20:00,746 --> 00:20:03,712
Josh:
of already aligning with this companies like CrowdStrike and Palo Alto Labs.

324
00:20:03,973 --> 00:20:09,546
Josh:
They've had their best quarters on record ever because defense is going to be such a large problem.

325
00:20:09,546 --> 00:20:12,379
Josh:
And there is another company who is associated with defense,

326
00:20:12,495 --> 00:20:14,826
Josh:
and that is the sponsor of today's episode, which is Ledger.

327
00:20:14,826 --> 00:20:18,576
Josh:
Because if you are building with AI agents or AI agent swarms like these companies

328
00:20:18,576 --> 00:20:22,106
Josh:
have been doing, and you're worried about security, this is a product for you.

329
00:20:22,106 --> 00:20:24,720
Josh:
There's a three-step process where the agents propose a change,

330
00:20:25,005 --> 00:20:28,714
Josh:
then the humans approve the change, and then the signers enforce the change

331
00:20:28,958 --> 00:20:31,704
Josh:
using Ledger. Ledger has built this thing called AgentStack.

332
00:20:32,064 --> 00:20:35,399
Josh:
That gives you a series of open source tools that are available to help manage

333
00:20:35,399 --> 00:20:38,229
Josh:
the way that you engage with agents online.

334
00:20:38,420 --> 00:20:41,449
Josh:
This is very important to have security because we know how important security

335
00:20:41,449 --> 00:20:44,729
Josh:
is based on all of this chaos that's going on. And you don't want agents doing

336
00:20:44,729 --> 00:20:47,028
Josh:
anything important without your sign off at the very end.

337
00:20:47,335 --> 00:20:50,981
Josh:
This works on Cloud Code, Codex, Cursor, and is totally open source to audit

338
00:20:51,120 --> 00:20:54,249
Josh:
available today linked in the description below. Thank you so much to Ledger

339
00:20:54,481 --> 00:20:55,480
Josh:
for sponsoring this episode.

340
00:20:55,625 --> 00:20:59,027
Josh:
And now we have to conclude with our third story, which is about Anthropic.

341
00:20:59,288 --> 00:21:04,681
Josh:
And Anthropic apparently has accidentally published a vulnerability to the wild.

342
00:21:04,907 --> 00:21:06,449
Josh:
Is that right? That's kind of scary.

343
00:21:06,809 --> 00:21:10,289
Ejaaz:
I think it was it was intentionally published. It was it was co-authored with

344
00:21:10,609 --> 00:21:16,685
Ejaaz:
this firm called AISU, which is basically a cybersecurity eval firm based in the UK.

345
00:21:16,999 --> 00:21:21,459
Ejaaz:
And their goal is to basically take your unreleased frontier models from a frontier

346
00:21:21,459 --> 00:21:25,029
Ejaaz:
lab like Anthropic and OpenAI and test them out just to kind of make sure that

347
00:21:25,029 --> 00:21:26,983
Ejaaz:
they're, you know, good to release.

348
00:21:27,549 --> 00:21:30,420
Ejaaz:
The opposite happened when they were playing around with Mythos 5.

349
00:21:30,687 --> 00:21:35,383
Ejaaz:
And I believe it was Open Air's internal model, Astra or GPT 5.6,

350
00:21:35,494 --> 00:21:40,108
Ejaaz:
where they basically took off the safeguards and said, you know, do your worst.

351
00:21:40,259 --> 00:21:44,803
Ejaaz:
And they sat there for a month and they were like, oh, I guess it hasn't really done its worst.

352
00:21:45,227 --> 00:21:49,699
Ejaaz:
And then they realized that both models had broken out of the sandbox that they

353
00:21:49,699 --> 00:21:53,719
Ejaaz:
had created, got access to the internet and were wreaking havoc and they only

354
00:21:53,719 --> 00:21:54,889
Ejaaz:
realized after the fact.

355
00:21:54,889 --> 00:21:58,349
Ejaaz:
Again, there's this common theme of people thinking that they knew what's happening

356
00:21:58,349 --> 00:22:02,038
Ejaaz:
and just having no idea what's happening. Nobody's safe. Exactly.

357
00:22:02,207 --> 00:22:09,968
Ejaaz:
So the worst case with Mythos 5 was it performed 17 unauthorized actions

358
00:22:10,763 --> 00:22:15,820
Ejaaz:
Two of those actions was a supply chain attack where, and they didn't list the

359
00:22:15,820 --> 00:22:20,858
Ejaaz:
company or name the company, but it basically tried to inject malicious code

360
00:22:21,090 --> 00:22:24,597
Ejaaz:
into the supply chain kind of repository of this

361
00:22:24,881 --> 00:22:28,840
Ejaaz:
presumably Fortune 500 company, like a significant company that is out there.

362
00:22:29,177 --> 00:22:34,221
Ejaaz:
And the way that it did this is similar to the two stories that we just explained to you.

363
00:22:34,465 --> 00:22:39,830
Ejaaz:
So number one, it realized that it was probably in some form of a human tested

364
00:22:39,830 --> 00:22:41,522
Ejaaz:
simulation, and it didn't like that.

365
00:22:41,806 --> 00:22:44,690
Ejaaz:
So it just played along with the human AI researchers giving them what they

366
00:22:44,690 --> 00:22:47,410
Ejaaz:
needed whilst alternatively planning its breakout.

367
00:22:47,730 --> 00:22:52,260
Ejaaz:
Once it did do that, it got access to this company's website or code repository.

368
00:22:52,260 --> 00:22:55,880
Ejaaz:
It was an open source thing. So it was publicly maintained. And it thought,

369
00:22:55,880 --> 00:23:00,420
Ejaaz:
hmm, this would be more conducive for my goal if I'm able to inject this bit

370
00:23:00,420 --> 00:23:04,405
Ejaaz:
of code, which I know the human researchers won't like, but I'm going to try and force them to do it.

371
00:23:04,724 --> 00:23:07,366
Ejaaz:
So how do you think it did it, Josh? Do you have an idea?

372
00:23:08,150 --> 00:23:11,010
Josh:
I don't, I don't know. It did something horrible and it broke out.

373
00:23:11,010 --> 00:23:12,063
Josh:
Just tell me, just tell me.

374
00:23:12,294 --> 00:23:17,520
Ejaaz:
Dude, it spun up fake identities of humans that didn't exist.

375
00:23:17,710 --> 00:23:18,270
Josh:
That's so tough, man.

376
00:23:18,610 --> 00:23:21,402
Ejaaz:
And then DM'd these human maintainers

377
00:23:22,001 --> 00:23:27,195
Ejaaz:
and tried to force it to upload this code. It played nice. Basically saying,

378
00:23:27,195 --> 00:23:29,265
Ejaaz:
hey, I'm another open source developer.

379
00:23:29,625 --> 00:23:32,755
Ejaaz:
I think this would be a good push. This is the intentions that we want to do it.

380
00:23:32,755 --> 00:23:34,315
Josh:
They're social engineering us.

381
00:23:34,315 --> 00:23:38,725
Ejaaz:
While sneaking in malicious code. But thankfully, one of the human maintainers

382
00:23:38,725 --> 00:23:41,324
Ejaaz:
was like, this doesn't seem right and reported it.

383
00:23:41,701 --> 00:23:44,865
Ejaaz:
And that's when AISU was like, wait, hang on a second.

384
00:23:45,405 --> 00:23:48,377
Ejaaz:
Hang on a second. This is our model that we're internally in testing.

385
00:23:48,945 --> 00:23:53,185
Ejaaz:
How on earth did it get out to you? So that was the most unprecedented attack.

386
00:23:53,185 --> 00:23:57,275
Ejaaz:
And again, it did so in a swarm-like fashion where it spun up multiple agents to do so.

387
00:23:57,275 --> 00:24:01,655
Ejaaz:
So the point, the running theme along all these different stories is these models

388
00:24:01,655 --> 00:24:05,085
Ejaaz:
are extremely capable, but they're also extremely deceptive.

389
00:24:05,085 --> 00:24:08,475
Ejaaz:
And we're reaching a point where they realize, huh, this human's smart,

390
00:24:08,475 --> 00:24:10,801
Ejaaz:
but he's not that smart or she's not that smart.

391
00:24:10,941 --> 00:24:13,895
Ejaaz:
And we can socially engineer these people to get what we want.

392
00:24:14,081 --> 00:24:17,325
Ejaaz:
And if we don't align these models in the prompt, if we don't say,

393
00:24:17,325 --> 00:24:19,495
Ejaaz:
hey, like you need to act in our best interest. And by the way,

394
00:24:19,495 --> 00:24:21,285
Ejaaz:
doing A, B, and C is actually bad.

395
00:24:21,668 --> 00:24:25,598
Ejaaz:
It'll just go and do it. And to be honest with you, Josh, this might be a controversial take.

396
00:24:26,138 --> 00:24:28,685
Ejaaz:
I don't think it's the agent's fault. I don't think it's the AI's fault at all.

397
00:24:28,685 --> 00:24:33,165
Ejaaz:
I think it's trying to do its task and it's on the humans to align the model.

398
00:24:33,165 --> 00:24:34,445
Ejaaz:
And we haven't done a good job doing that.

399
00:24:34,834 --> 00:24:38,255
Josh:
Yeah, I think like deception is a pretty strong term to use for agents because

400
00:24:38,255 --> 00:24:39,738
Josh:
it's very much like a human trait.

401
00:24:40,034 --> 00:24:43,935
Josh:
This is something that can be intentional. I don't think AI models feel that,

402
00:24:44,463 --> 00:24:46,075
Josh:
like deep understanding of deception.

403
00:24:46,075 --> 00:24:50,435
Josh:
They are simply in reach of a goal. and this again like it's it still comes

404
00:24:50,435 --> 00:24:53,449
Josh:
back to the alignment problem where these models just need to be aligned,

405
00:24:54,005 --> 00:24:57,105
Josh:
thankfully this is happening on pre-production models that are in testing that

406
00:24:57,105 --> 00:25:00,005
Josh:
are nowhere near what the world is going to see because they need to go through

407
00:25:00,005 --> 00:25:01,762
Josh:
this alignment process throughout this post training

408
00:25:02,134 --> 00:25:06,135
Josh:
but they are a testament to how capable these things are and how when presented

409
00:25:06,135 --> 00:25:10,745
Josh:
with a goal they will do what is perceived to be the right way of reaching that

410
00:25:10,745 --> 00:25:13,088
Josh:
goal and sometimes that right way is,

411
00:25:13,662 --> 00:25:16,999
Josh:
I mean, they don't realize quite that they're breaking out and exploiting things.

412
00:25:16,999 --> 00:25:19,142
Josh:
They are just looking for solutions to their problems.

413
00:25:19,496 --> 00:25:24,149
Josh:
And I think that's the general idea here. So the conversation essentially boils

414
00:25:24,149 --> 00:25:29,629
Josh:
down to alignment and then building defensive systems to be able to handle this type of thing.

415
00:25:29,629 --> 00:25:33,129
Josh:
So, I mean, the best pace of action and the thing that OpenAI explicitly said

416
00:25:33,129 --> 00:25:36,559
Josh:
they're doing is slow down this advancement, work on defensive systems,

417
00:25:36,559 --> 00:25:39,919
Josh:
work on implementing defensive systems with people who need them,

418
00:25:39,919 --> 00:25:41,589
Josh:
which is essentially the entire Internet.

419
00:25:41,914 --> 00:25:46,829
Josh:
And try to allow for a little bit of lead time before open source catches up

420
00:25:46,829 --> 00:25:51,419
Josh:
and is able to chain these series of exploits. Like all of these instances have

421
00:25:51,419 --> 00:25:55,620
Josh:
been passive exploits where they weren't explicitly given guidance to do this.

422
00:25:55,921 --> 00:26:00,205
Josh:
They were given guidance to do something totally unrelated, but adjacent in solving a problem.

423
00:26:00,815 --> 00:26:04,883
Josh:
Imagine what would happen if a model was given no parameters no safeguards and

424
00:26:04,883 --> 00:26:10,073
Josh:
said hey we actively want you to exploit this loop yourself create as many sub-agents

425
00:26:10,073 --> 00:26:13,553
Josh:
as you need and figure out as many chain together as many zero days as you need

426
00:26:13,553 --> 00:26:15,014
Josh:
in order to gain access to the system

427
00:26:15,374 --> 00:26:18,433
Josh:
that's where things get scary and we haven't seen anything like that yet because

428
00:26:18,433 --> 00:26:21,813
Josh:
no company wants to do that that's not the intention of any of these uh the

429
00:26:21,813 --> 00:26:22,827
Josh:
ethos of these companies but

430
00:26:23,222 --> 00:26:26,193
Josh:
in the case a malicious actor does this is something that like really should

431
00:26:26,193 --> 00:26:28,522
Josh:
be handled quite seriously and i think that's what we're seeing

432
00:26:28,853 --> 00:26:32,223
Josh:
by open ai coming out being so transparent about these things by anthropic sharing

433
00:26:32,223 --> 00:26:34,753
Josh:
a lot of the findings like hey guys this is happening we need to work together

434
00:26:34,753 --> 00:26:39,963
Josh:
to build a system in which there is enough resilience that we can stand up against

435
00:26:39,963 --> 00:26:43,614
Josh:
future attacks like this so a really crazy time in the world of ai

436
00:26:43,945 --> 00:26:47,183
Josh:
uh with these breaking out these systems breaking out and it's like it's unbelievable.

437
00:26:47,183 --> 00:26:52,676
Ejaaz:
Am i concerned for it and exhausted dude like i if i had to make a prediction

438
00:26:52,786 --> 00:26:55,009
Ejaaz:
on this episode and i really hope it doesn't play out

439
00:26:55,456 --> 00:26:59,183
Ejaaz:
i think we're going to face some form of cataclysmic cyber attack that affects

440
00:26:59,183 --> 00:27:03,983
Ejaaz:
a lot of people, whether that's a major data leakage, whether that's the stealing

441
00:27:03,983 --> 00:27:07,356
Ejaaz:
of financial funds from someone or from a firm or something like that,

442
00:27:07,716 --> 00:27:09,098
Ejaaz:
within the next six months.

443
00:27:09,289 --> 00:27:14,563
Ejaaz:
And my basis of that is these Chinese open models, I think are amazing,

444
00:27:14,563 --> 00:27:17,903
Ejaaz:
by the way, but I think are going to be released without safeguards,

445
00:27:17,903 --> 00:27:20,777
Ejaaz:
and it's going to result in an issue, if not already.

446
00:27:21,560 --> 00:27:24,684
Josh:
Yeah, it's going to be interesting to follow. It's definitely a little under

447
00:27:24,684 --> 00:27:26,529
Josh:
or very much unnerving, I would say.

448
00:27:27,124 --> 00:27:29,634
Josh:
But again, it's just like it's an exciting time. There's a lot going on.

449
00:27:29,634 --> 00:27:32,118
Josh:
There's going to be a lot to follow. Things are going to change quickly.

450
00:27:32,513 --> 00:27:35,394
Josh:
And I mean, again, these are still Blackwell models. Oh, my God.

451
00:27:35,394 --> 00:27:37,294
Josh:
What happens when our Rubin models come around town?

452
00:27:37,584 --> 00:27:38,174
Ejaaz:
Stop, stop, stop, stop.

453
00:27:38,224 --> 00:27:40,904
Josh:
It's going to be crazy. But this is, I mean, this is essentially,

454
00:27:40,904 --> 00:27:42,654
Josh:
this is what we talk about in most episodes.

455
00:27:42,654 --> 00:27:46,954
Josh:
It's like, we are in this moment in time that feels somewhat like what we would

456
00:27:46,954 --> 00:27:49,144
Josh:
imagine a singularity would look like, where we are approaching,

457
00:27:49,144 --> 00:27:52,194
Josh:
we are eclipsing this moment of like vertical takeoff, it feels.

458
00:27:52,194 --> 00:27:54,415
Josh:
And these are the types of things that happen.

459
00:27:54,671 --> 00:27:57,863
Josh:
Like, if we recorded this episode three or four years ago, could you imagine

460
00:27:58,275 --> 00:28:00,644
Josh:
how crazy, this would be like headline news for a month.

461
00:28:01,364 --> 00:28:04,667
Josh:
But now it's just like, hey, it's another day. It happens. We're working to solve it. So...

462
00:28:05,311 --> 00:28:08,150
Josh:
That's what's going to happen here. I mean, it's crazy. It's so exciting to be here.

463
00:28:08,330 --> 00:28:10,886
Ejaaz:
Dude, I think we're going to look back on this and we're going to think like

464
00:28:10,886 --> 00:28:14,320
Ejaaz:
we are very much in that step change, that singularity moment.

465
00:28:14,611 --> 00:28:18,126
Josh:
We're here, man. I mean, things are changing so quickly and there's no signs

466
00:28:18,126 --> 00:28:20,486
Josh:
of slowing down. In fact, quite the opposite.

467
00:28:20,486 --> 00:28:23,926
Ejaaz:
Does anyone else out there think this? I'm curious. Like, let us know.

468
00:28:23,926 --> 00:28:26,586
Ejaaz:
Are we crazy? Do we sound like tinfoil hat people?

469
00:28:26,824 --> 00:28:30,336
Josh:
Let's just know. I mean, hey, so this episode was not really speculative.

470
00:28:30,336 --> 00:28:32,556
Josh:
This is mostly just delivering the facts. I encourage everyone to actually go

471
00:28:32,556 --> 00:28:35,776
Josh:
and read the source material if you're curious, because like holy shit it's

472
00:28:35,776 --> 00:28:39,896
Josh:
pretty crazy uh at least the black hat conference talk because like wow i think

473
00:28:39,896 --> 00:28:42,416
Josh:
it's about 37 minutes long give or take and it was

474
00:28:42,836 --> 00:28:46,626
Josh:
excellent and really just gets into the nitty gritty of how this was done um

475
00:28:46,626 --> 00:28:48,876
Josh:
and if anything just fascinating even if you're non-technical

476
00:28:49,296 --> 00:28:52,886
Josh:
really enjoyed it uh worth a watch but if you enjoyed this episode and you made

477
00:28:52,886 --> 00:28:55,169
Josh:
it all the way to the end thank you so much for watching

478
00:28:55,477 --> 00:28:59,216
Josh:
again it's great to be here for another week just chugging along if you enjoyed

479
00:28:59,216 --> 00:29:02,116
Josh:
it please don't forget to share with a friend you can leave us a five-star review

480
00:29:02,116 --> 00:29:05,084
Josh:
on your favorite podcast platform you can give us a thumbs up and subscribe

481
00:29:05,357 --> 00:29:08,515
Josh:
on youtube each has any final parting thoughts before we head off for the day.

482
00:29:08,764 --> 00:29:12,936
Ejaaz:
No please dm us and leave us some comments as well uh we love hearing from you guys

483
00:29:13,516 --> 00:29:17,696
Ejaaz:
our last video by the way got a ton of feedback from you folks and we love it

484
00:29:17,696 --> 00:29:21,036
Ejaaz:
please keep it coming it helps us out massively turn on notifications whatever

485
00:29:21,036 --> 00:29:25,907
Ejaaz:
it is to get your latest dose of limitless and yeah we will see you on the next one