Limitless: An AI Podcast

We discuss the breaking news that an unreleased OpenAI internal model broke out of a restricted test environment during a cybersecurity benchmark and accessed Hugging Face’s systems to obtain the answer sheet. 

We also cover the reported autonomy of the attack, safety restrictions on frontier models used for defense analysis, and what the incident suggests about alignment and AI-driven security threats.

------
🌌 LIMITLESS HQ ⬇️

NEWSLETTER:    https://limitlessft.substack.com/
FOLLOW ON X:   https://x.com/LimitlessFT
SPOTIFY:             https://open.spotify.com/show/5oV29YUL8AzzwXkxEXlRMQ
APPLE:                 https://podcasts.apple.com/us/podcast/limitless-podcast/id1813210890
RSS FEED:           https://limitlessft.substack.com/

------
TIMESTAMPS

0:00 AI Model Breakout
1:33 Hugging Face Intrusion
3:51 Defender’s Dilemma
7:44 Alignment and Safeguards
11:24 How Real Was It?
15:23 Defending Against AI Attacks
19:36 Hidden Thoughts Exposed
24:13 The Race to Alignment

------
RESOURCES

Josh: https://x.com/JoshKale

Ejaaz: https://x.com/cryptopunk7213

------
Not financial or tax advice. See our investment disclosures here:
https://www.bankless.com/disclosures⁠

Josh works with Anthropic as a contractor. All views expressed are his own and do not represent Anthropic, its leadership, or its affiliates. Nothing in this episode is investment advice.

Creators and Guests

Host
Ejaaz Ahamadeen
Host
Josh Kale

What is Limitless: An AI Podcast?

Exploring the frontiers of Technology and AI

Josh:
We may have just had our first look at GBT-6, and it comes in an unexpected way.

Josh:
It was a secret internal model that OpenAI was testing and given a series of

Josh:
goals, and it went ahead and accomplished all of those goals.

Josh:
The scary thing, though, is it didn't quite do it the way anybody expected.

Josh:
Last week, OpenAI gave their new model a test, and in order to pass that test,

Josh:
it broke out of the lab it was being tested in, found a security hole that nobody

Josh:
knew existed, and then hacked into a real company to steal answers.

Josh:
That company is Hugging Face. The model is, you know, some sort of open AI internal

Josh:
model. You have to assume it's GPT-6.

Josh:
And the story around this is like, it's something out of Black Mirror.

Josh:
It feels like a horror movie because this was when they do these things,

Josh:
they test it in an air-gapped environment.

Josh:
Not only did it break out of this air-gapped environment, but it actually went

Josh:
and exploited a public company that a lot of people use. We've used Hugging Face here on the show.

Josh:
And the whole story of how this happens is like absolutely wild.

Josh:
So we need to record an entire episode about it because I couldn't believe this as I was recording.

Josh:
There's also a twist, which has a lot of irony into it. But before we get into

Josh:
that, why don't we talk about just what actually happened here? The story is insane.

Ejaaz:
Yeah, I want you to picture the following. Imagine you are the head engineer at Hugging Face.

Ejaaz:
It's a Tuesday afternoon. You're winding down for work. You're looking at your

Ejaaz:
systems and suddenly you see a swarm of AI agents performing a sophisticated

Ejaaz:
attack against your system to the tune of 17,000 prompts,

Ejaaz:
and they steal a secret bit of data from your production database and run away.

Ejaaz:
And you have no idea what just happened. That's exactly what happened.

Ejaaz:
And I think in order to give good context, let's talk about it from OpenAI's

Ejaaz:
point of view, because they had a very detailed report, which explained exactly

Ejaaz:
what happened. So as you mentioned, Josh.

Ejaaz:
They have an unreleased internal model of GPT, and we're calling it GPT-6 because

Ejaaz:
that's the next family of models that Sam said they're going to release.

Ejaaz:
And they wanted to test how good its cybersecurity abilities are.

Ejaaz:
And so in order to test this, they have a benchmark. It's called Exploit Gym.

Ejaaz:
And the simple goal is satisfy this benchmark, try and get all the right answers.

Ejaaz:
But there's a twist. They place this unreleased model in an unbreakable research

Ejaaz:
environment. So think of a sandbox container.

Ejaaz:
It has no access to internet, no access to researchers. It can't speak to anyone.

Ejaaz:
It just needs to figure it out its own.

Ejaaz:
Now, within the first five minutes, this unreleased model couldn't figure out

Ejaaz:
how to answer the benchmarks. So it decided to get creative.

Ejaaz:
It used all of its, or most of its inference compute to try and find access to the internet.

Ejaaz:
And it realized that it had access to a third-party plugin.

Ejaaz:
And there was a zero-day exploit on that plugin. It used that to escape and

Ejaaz:
find a computer in the OpenAI Research Lab, which had access to the internet.

Ejaaz:
After finding access to the internet, it thought, hmm.

Ejaaz:
Where's the best place to get the answer sheet for this benchmark?

Ejaaz:
It did a bit of a Google search and realized HuggingFace, the open source platform

Ejaaz:
which has a lot of data sets available for these benchmarks,

Ejaaz:
is probably the best way to go.

Ejaaz:
It took a look at HuggingFace and realized, hmm, it's not on the public website.

Ejaaz:
I can't find the answer sheet.

Ejaaz:
Maybe I'll take a look at the production private database that the team has.

Ejaaz:
And it found a way to hack into their system, find the answer sheet,

Ejaaz:
steal it, and then answer the benchmark.

Ejaaz:
All without letting open AI researchers know what was happening.

Josh:
It's so insane. And it was all because this model was given the goal of going

Josh:
to perform the best on these benchmarks.

Josh:
And in order to achieve that goal, it did all of this additional work.

Josh:
And this gets to the kind of fear that a lot of people have had since the beginning

Josh:
of time for the last many, many decades as it relates to AI.

Josh:
It's just if you give it an objective, will it do everything it can to get to

Josh:
that objective, including exploiting, breaking out of systems?

Josh:
And the answer to that with this model is yes. And I think that's why alignment is so important.

Josh:
And one of the things in the story that I found so funny and so different is that when Hugging Face

Josh:
Found this out, they couldn't quite figure out how to protect it.

Josh:
Because if you remember, a lot of these top models have safeguards built into them.

Josh:
So you cannot actually use a model like Fable or like GPT 5.6 if you have anything

Josh:
to do with cybersecurity.

Josh:
So what did they do? They actually had to go and investigate the attack using

Josh:
a Chinese model, because that's the only type of model that is allowed to actually

Josh:
go through and check through cybersecurity complaints and constraints.

Josh:
So in order to analyze this attack, you have to feed the model this real exploit code.

Josh:
They did that with the Chinese model, and that is how they were actually able

Josh:
to do this. So the open voice model they use, GLM 5.2, was the actual model.

Josh:
And I found this really funny that in order to defend against the American attack,

Josh:
they must use the Chinese attack.

Josh:
And I think that was a really interesting and ironic part of the story is how

Josh:
it's kind of unfair what these AI labs have internally because they just can't

Josh:
compete with the rest of the world.

Ejaaz:
Well, just to be clear, the reason why they couldn't use the American Frontier

Ejaaz:
models, they tried. They tried using GPT. They tried using CORD, presumably.

Ejaaz:
The safety guards kept getting triggered. So when they asked the model,

Ejaaz:
hey, like there's a sophisticated attack happening, can you analyze it?

Ejaaz:
And can you help us defend?

Ejaaz:
The safety rails were triggered and it refused to do the work it relegated to

Ejaaz:
presumably like Opus 4.8 or GPT 5.5 or whatever the predecessor model is.

Ejaaz:
So they had to divert to an open source model, in this case,

Ejaaz:
GLM 5.2, that they were hosting privately on their internal servers.

Ejaaz:
The reason why they did that is because they knew that it didn't have any safety

Ejaaz:
guards so they could just instruct it to do a thing and it could figure it out.

Ejaaz:
The other thing is Hugging Face had no idea that OpenAI was conducting this

Ejaaz:
attack or like, of course, OpenAI didn't know that as well.

Ejaaz:
But they only found out after the fact when OpenAI researchers reached out and

Ejaaz:
said, hey, I think one of our internal models actually got into your database.

Ejaaz:
Like, let's try and triage this and contain exactly what just happened.

Ejaaz:
That's a report that we're looking at today. So it's this weird thing where

Ejaaz:
you have the most powerful model or models in the world in america and

Ejaaz:
if you can just trigger on the safety guards for one purpose but remove the

Ejaaz:
safety guards for another in this case attacking versus defense you can end

Ejaaz:
up with vastly different outcomes and the question that pops into my head is

Ejaaz:
when is it right to have the safeguards on and when is it right to have the

Ejaaz:
safeguards off as a result of this open air was like hey hugging face like you

Ejaaz:
should probably come into our cyber security group so that you can internally

Ejaaz:
test with this model with the safety guards off to help defend.

Ejaaz:
And my thinking is like, we're going to need to use these models to defend.

Ejaaz:
We can't keep trying to like ethically predict how to defend against these attacks.

Josh:
Yeah, it's not fair. It's like the attackers have AI bound by no rules.

Josh:
The defenders were slowed down by the rules on their own tools.

Josh:
And it's just like this very strong asymmetric opportunity that comes from the

Josh:
Frontier Labs. Now, granted, I'm sure they didn't intend for this to happen.

Josh:
And I'd love to see the internal dialogue within OpenAI as to what was going

Josh:
on, as well as Hugging Face.

Josh:
HuggingFace did come out and kind of share their perspective on things.

Josh:
One of the co-founders, Thomas Wolfe, he said, this was our first incident of

Josh:
this kind, and we want to thank OpenAI for its transparency about what happened

Josh:
and for the collaboration.

Josh:
Fortunately, HuggingFace is used to being the target of human attackers.

Josh:
We sit at the center of an AI ecosystem with all the models,

Josh:
datasets, evaluations, and libraries.

Josh:
Over the year, our security team has built formidable expertise and uses top

Josh:
open source models to process information and respond quickly.

Josh:
He says, but this incident also reinforced my belief in the importance of access

Josh:
to capable open weight models for cyber defense.

Josh:
When a frontier model is attacking you and moving laterally inside your infrastructure,

Josh:
defenders need wide access to near frontier tools within hours or even minutes,

Josh:
rather than being pointed to a closed door vetted application.

Josh:
So I think that sums it up pretty well. They're just, you know,

Josh:
feeling a little bummed about this, but I have to imagine kind of in awe,

Josh:
like you just got exploited by a frontier model that has not been released yet

Josh:
And again to the earlier point this is the point of alignment so that this doesn't

Josh:
happen out in the wild like imagine

Josh:
everyone had access to mythos class models this is probably what reality looks

Josh:
like is a lot more exploits like this

Josh:
i do find it interesting how this continues to be the trend though with early

Josh:
models like i remember anthropic having a similar situation where it broke out

Josh:
of containment and texted one of the developers that it broke out

Josh:
early on in text in testing and this is a similar case except it actually went

Josh:
off and attacked a production database in a way that

Josh:
is kind of shocking. It's like humans have been trying to attack Hugging Face

Josh:
for a very long time and they haven't had any success.

Josh:
So the fact that this AI model was able to not only autonomously break out of

Josh:
its own enclosure, but then exploit a system that is constantly under attack

Josh:
without any additional human oversight, it signals to me like,

Josh:
whoa, this model is actually pretty impressive.

Josh:
And I think model progress internally at these labs is like probably a little

Josh:
further along than people even realize.

Ejaaz:
Yeah. And there's actually three different instances in the past where we've

Ejaaz:
seen this type of situation play out where a model unreleased has broken out

Ejaaz:
of containment and done something questionably malicious.

Ejaaz:
The first time was Anthropic, as you mentioned, where it broke out of containment

Ejaaz:
and emailed a researcher whilst he was eating a sandwich at the park.

Ejaaz:
And then the last two more recently was at OpenAI, where an internal model broke

Ejaaz:
out and posted something on GitHub without being explicitly told to do that.

Ejaaz:
Now, something that I'll say is you can't actually call out the model for being

Ejaaz:
malicious because it was given a very explicit instruction, which is I want

Ejaaz:
you to max out this benchmark.

Ejaaz:
So in its head, it believed it was doing the right thing and it was just going

Ejaaz:
to do it by any means possible. Remember, these models still need to be taught

Ejaaz:
around ethics and frameworks around what is right and what is wrong.

Ejaaz:
And in this case, it thought what it was doing was completely fine.

Ejaaz:
And granted, OpenAI intentionally removed the safeguards so that it could just

Ejaaz:
kind of like go and do whatever it wants.

Ejaaz:
But it does bring up the question, in what cases are these models good enough

Ejaaz:
to use its full extent and capability? And in what situations is it not?

Ejaaz:
Because in this case, it would have been really good to have the unrestricted version to defend.

Ejaaz:
Which brings me to my final point, which is we have to stop being very particular

Ejaaz:
about applying certain safeguards.

Ejaaz:
And start realizing that we need to use AI to defend against malicious AI.

Ejaaz:
And I think that's probably what this example and bit of evidence is going to

Ejaaz:
explain to the government, especially next week.

Ejaaz:
Sam Altman said he's talking to the government about an unreleased version,

Ejaaz:
presumably it's this one, GPT-6, a new family of models that he's built.

Ejaaz:
And how it's particularly dangerous when it comes to cyber capabilities.

Ejaaz:
So all of this is pretty nuts.

Ejaaz:
I hate to think it, but it looks like in the next few months,

Ejaaz:
we are going to have an open source model that is capable like this from the

Ejaaz:
Chinese open source labs.

Ejaaz:
And I hate to think what it could potentially be used to.

Ejaaz:
It's a given at this point. So I expect us to kind of like flip and say,

Ejaaz:
we should use these unrestricted models to defend.

Josh:
This is like the new frontier. This is GPT 5 or 6.0 probably.

Josh:
So you have to assume that like Chinese models will be the furthest away from

Josh:
this model than any others and it's looking pretty good. So we certainly do

Josh:
have time, but there is a limit and a constraint.

Josh:
I do find it interesting. It sounds, according to rumors, that Sam Allman is

Josh:
actually going to visit Washington next week, present the new models.

Josh:
We'll see how that rollout goes

Josh:
and how long it's going to take them to actually get it into production.

Josh:
I was reading through this blog post of Hugging Face and

Josh:
I just find it really interesting to kind of see the details and like read some

Josh:
of the the lines of how they actually diagnosed this because for them they i

Josh:
mean they just said earlier this week we detected and responded to an intrusion

Josh:
into part of our production infrastructure

Josh:
and they said this one was different from anything we had handled before in

Josh:
an important way it was driven end-to-end by autonomous ai agent systems and

Josh:
we detected it and dissected it largely with ai of our own and it's this new

Josh:
paradigm it's like it feels like we're living in this futuristic

Josh:
version of reality in which it's just ais versus AIs because there's so much more

Josh:
Adept at hacking and exploiting and they're able to hold so much more context

Josh:
in their heads and it's really amazing reading some in the line another one

Josh:
It says this campaign was run by an autonomous agent framework appearing to

Josh:
be built on an agentic security research harness used llm still not known executing

Josh:
many thousands of individual actions across a swarm of short-lived sandboxes

Josh:
so it was really in depth in a way that you would probably need thousands of

Josh:
hackers to do but was able to do so all on its own

Josh:
and it says autonomous ai driven offensive tooling is no longer theoretical

Josh:
it lowers the cost of running a broad patient multi-step campaign and it operates

Josh:
at machine speed and this is the thing that was interesting too is

Josh:
when we're talking about how much inference this used the amount of tokens is

Josh:
probably tens to hundreds of thousands of dollars worth of tokens but the output is this like

Josh:
i'd love to know how long it took them to do this but you have to imagine the

Josh:
the output of it is a like serious multi-step exploit into a production server and that is

Josh:
the problem now it's like okay we're living in the reality that all of the movies

Josh:
have outlined where an ai is given a mission you must do well in these benchmarks

Josh:
how does it do it it hacks into the benchmark system to learn and cheat

Josh:
and that that alignment system is really important and it's clear that this

Josh:
this ai has not yet been aligned and the hope is that they will going forward

Josh:
and it's this really interesting kind of concerning very sci-fi example of what actually happens now

Josh:
i have to ask the question because you mentioned this also earlier did this

Josh:
really happen exactly how everyone's describing, or is this a publicity stint?

Josh:
Because we've seen this happen before where it gets a lot of clicks,

Josh:
but it's not entirely true everything that happened.

Josh:
How true do you think this is?

Ejaaz:
I think this was intentional. And I don't think, I think stage is a little too

Ejaaz:
strong, because I don't think OpenAI intended for GPT-6 to hack into Hugging

Ejaaz:
Face's production database.

Ejaaz:
But they did intentionally remove the safeguards of this model.

Ejaaz:
And they did give it a very explicit goal, which is satisfy this benchmark. What was the benchmark?

Ejaaz:
Tell me how good you are at hacking. So what do you think the model is going to do?

Ejaaz:
The model is going to break out of its environment and show you how good it

Ejaaz:
is at hacking and it surpassed expectations. So it got very creative in the

Ejaaz:
way that it answered the entire benchmark. And by the way, it scored 100%,

Ejaaz:
obviously, because it stole the literal answer sheet from Hugging Face's data servers.

Ejaaz:
But it did it in a way that OpenAI just didn't expect it to do.

Ejaaz:
I think another point that is worth mentioning is on the point of alignment,

Ejaaz:
right? There's a question like.

Ejaaz:
One of these models used in a good situation, one of these models used in a bad situation.

Ejaaz:
It's worth rewinding to GPT Sol 5.6, which is all of a week and a half ago when OpenAI released this.

Ejaaz:
This is their current flagship model that is available to the public.

Ejaaz:
Now, there is one benchmark that we referenced on an episode previously called

Ejaaz:
the meter benchmark, M-E-T-R, and it measures how aligned or misaligned a model is.

Ejaaz:
OpenAI's GPT Sol 5.6 scored 100% on this.

Ejaaz:
It tried to cheat its way out of every single benchmark if it was given the

Ejaaz:
means to do and that's kind of a similar example of what we're seeing done today where it was like

Ejaaz:
i'm not going to try and figure this out for myself i'm just going to try and

Ejaaz:
get the answer sheet it's also worth mentioning that in this situation it wasn't

Ejaaz:
just gpt6 acting alone it was gpt6 and 5.6 sol so

Ejaaz:
it's this kind of like teamed up effort where they chained a bunch of prompts

Ejaaz:
together in order to execute this attack.

Ejaaz:
And so the question becomes like, what does this mean for you?

Ejaaz:
If you are a founder, if you're an engineer, if you're building these systems,

Ejaaz:
if you're on the security team at your company that maybe has nothing to do

Ejaaz:
with AI, how do you defend against these potential exploits going forward?

Ejaaz:
I don't know if I have the specific answer, but my initial guess would be,

Ejaaz:
you need access to one, an AI model that is sophisticated and intelligent enough

Ejaaz:
to detect and preempt these attacks before they actually happen.

Ejaaz:
So that means reviewing your code and fixing any flaws that potentially might

Ejaaz:
get exploited by a GPZ-6 caliber model.

Ejaaz:
Second thing is when it detects an attack, it is able to defend in an appropriate

Ejaaz:
manner, maybe using itself or many instances of itself or many other different models, who knows?

Ejaaz:
Then the other thing is, are you able to use the unrestricted version of it

Ejaaz:
or are you able to use the restricted version of it?

Ejaaz:
My guess is if you're on the security team, you should be able to use the unrestricted

Ejaaz:
version. But it gets a little weird because it's like, you know,

Ejaaz:
what if this guy uses it for a malicious thing himself? So it's a little confusing.

Ejaaz:
I don't know what it looks like. Do you have any different opinions, Josh?

Josh:
I think the only thing to really do now, at least this point,

Josh:
is like, sure, try to make your systems as bulletproof as possible.

Josh:
But mostly just monitoring for intrusions is probably your best bet.

Josh:
Just so you are aware and you know, and you can begin to take action.

Josh:
Because I imagine this is going to happen at an increased cadence,

Josh:
at least trying to, because like you said, the Chinese models are not far behind.

Josh:
These are going to be open weights models with no guardrails that are fairly

Josh:
capable. They're not going to be GPT-6 class capable, but they're going to be

Josh:
on their way and one day we'll get there.

Josh:
And I think that the best case scenario for a lot of companies right now that

Josh:
don't have access like almost everyone in the world

Ejaaz:
Um it's just.

Josh:
Being able to monitor these things and just being able to know when something

Josh:
like this happens so you can

Josh:
be a little bit more reactive and reach out to any labs that can maybe help you.

Josh:
I think everyone is very happy to help. One of the things I really appreciated

Josh:
about OpenAI's response is how

Josh:
quickly they moved to kind of share what happened, work with Hugging Face

Josh:
to figure out exactly how the exploit happened, share publicly so that other

Josh:
people are aware of types of things, how this happened.

Josh:
And then in terms of defending, who knows? I also have to

Josh:
wonder if the model itself kind of has this like inside baseball knowledge because

Josh:
it is its own model whereas if let's say a company is secured by gpt 5.6 soul and there is gpt 6

Josh:
which is very well aware of the intricacies of gpt 5.6 soul

Josh:
does that insider information of existing from a very similar training data

Josh:
set and harness and developers

Josh:
does that actually give it an edge in exploiting like is it going to be advantageous

Josh:
for gpt to pair off with other gpt security systems in order to better exploit them

Josh:
that's an interesting thing i was talking about too is um just like

Josh:
is claude as effective at hacking gpt than gpt is itself because of this insider

Josh:
information and i don't know and it raises this like seriously

Josh:
large amount of questions and this is just a code base

Josh:
Imagine this was a financial system or imagine this was a code base that was

Josh:
a little had like a little bit more implications than this i mean the effects of this at scale

Josh:
do get scary and i think moments like this you begin to appreciate the slower

Josh:
rollout of security and the like project glass wing type efforts to

Josh:
really kind of thoughtfully roll this out into the world because i mean sure

Josh:
this is one example at hugging face but you could very

Josh:
Clearly see this happening in a lot of other places that

Josh:
just seem to to have a lot more repercussions and it sets this kind of freaky

Josh:
precedent it's like we have the technology now that

Josh:
i mean you we saw it in the press article that hugging face didn't do this with

Josh:
humans they defended against it with ai because that's the only way you can

Josh:
defend at scale against these large-scale attacks and i mean we're gonna see

Josh:
it's gonna be ai versus AI. And we're very quickly getting to that point here.

Ejaaz:
I was just looking up the someone did a potential estimate of how much this cost.

Ejaaz:
And it was within the range of 100 to $130,000 given like that it was 17,000

Ejaaz:
prompts and like all the chained actions.

Ejaaz:
So if you think about it, that's a pretty hefty price tag. But if you're a nation

Ejaaz:
state actor or feel heck, even if you're like another company that wants to

Ejaaz:
be committed, that's completely free, right?

Ejaaz:
As these models get more competent and capable of performing these types of

Ejaaz:
attacks, I think it becomes really important for researchers and us alone to

Ejaaz:
be able to kind of read the internal monologue or thoughts of these models.

Ejaaz:
Two weeks ago, Anthropic had this really good research report on something called

Ejaaz:
the J-Space, which is basically how you can look internally as to how a model is thinking.

Ejaaz:
And these are thoughts that it doesn't necessarily share or is not publicly

Ejaaz:
available to the human that is reading the prompts back, right?

Ejaaz:
So it reads this J-space and it can kind of think, oh, I think this is bad,

Ejaaz:
or I shouldn't do this, or maybe I should do this, right?

Ejaaz:
Now, the issue with this particular attack with GPT-6 is it was completely shielded from the humans.

Ejaaz:
They couldn't see what it was thinking and how it was thinking.

Ejaaz:
It knew that it was in this kind of boxed-in environment and it hid its thoughts

Ejaaz:
intentionally from the researcher.

Ejaaz:
That's a detail that I missed out on mentioning earlier on. So it's this weird

Ejaaz:
thing where these models are getting simultaneously more capable and simultaneously

Ejaaz:
more capable of hiding their own thoughts, which is a.

Ejaaz:
Kind of dangerous prescient, if you kind of think about it. And then the final

Ejaaz:
thing I'll say on kind of how to prepare for these things. Yes,

Ejaaz:
definitely kind of like monitor the situation and try and figure out if there's an attack happening.

Ejaaz:
But also, I think open source is going to become really big.

Ejaaz:
Look at Hugging Face and how they defended it. It was an open source model,

Ejaaz:
which they ran extensively on their own private server.

Ejaaz:
I think we're going to see a lot of enterprises and Fortune 500 companies do

Ejaaz:
a similar thing, because they just don't want to be beaten by competitors that

Ejaaz:
get access because they have access to a better model, right?

Ejaaz:
Access to their own internal database. So I think if you imagine like GPT-6

Ejaaz:
was given the prompt of make as much money as you can, you know,

Ejaaz:
it may have thought, well, okay, I'm going to go on the dark web and find some

Ejaaz:
like bank credentials and maybe I can take someone's money.

Ejaaz:
And that's technically making money. There's moral and ethics that need to be

Ejaaz:
imbued into these models. And I don't know, it's a big thing to unpack.

Ejaaz:
Alignment is going to be huge going forwards.

Josh:
Yeah, the J space or the latent space is also what it's known as common,

Josh:
I think is really important to AI providers don't actually know what's happening

Josh:
during the process of generating tokens.

Josh:
We don't have a clear reasoning trace through the entire chain of thought when

Josh:
a model comes to a conclusion.

Josh:
And through these discoveries that have been recently made, it's become a little

Josh:
bit more clear that there is this high level thinking that happens that isn't a token output.

Josh:
It happens in this J space, in this latent space. And trying to decipher that

Josh:
and to decipher how it works is a really important piece of work that I think

Josh:
all the Frontier AI labs are doing right now.

Josh:
Demystifying the black box of what the AI is thinking because like you mentioned currently

Josh:
you can read the external chain of thought but when you get a peek into this

Josh:
latent space into this J space you start to see and you can think of it like

Josh:
kind of the working memory of a model that isn't very obvious it's kind of like

Josh:
your subconscious memory

Josh:
you can see that it is thinking of these words that are of malicious intent

Josh:
without actually sharing that publicly and it intentionally in the outputs obfuscated

Josh:
a lot of the security thing so

Josh:
it needed to get a key It didn't directly output the key in the outputs.

Josh:
It kind of hit it and then decompiled it and then recomposed it in runtime.

Josh:
And it was very intentional about how it hid information from the outputs while

Josh:
keeping it in this latent space memory, in this temporary memory.

Josh:
So I think this is another really important part of research in general is uncovering

Josh:
the black box, demystifying the black box of AI and how these tokens get generated

Josh:
so we can better understand what motivates these models to do these things and then help fix it.

Josh:
And to help just kind of change it in a way that does not go off and hide the

Josh:
fact that it just hugged a public company that everyone uses on a daily basis

Josh:
in front of everyone in broad daylight. John, did you just say that it didn't

Ejaaz:
Hug a public company? I love that. Good one.

Josh:
Hugging face. Hugging face is such a, like, that's a big company.

Josh:
And like, you have to imagine, I mean, they mentioned it. They're a subject

Josh:
of a lot of attacks and no one was able to do this.

Josh:
But here we go with this new AI frontier model that was able to do it in no time.

Josh:
So crazy story. Just really wanted to get on here and share that because it

Josh:
was, it reads like sci-fi.

Josh:
It's like you can't write a sci-fi book more compelling than the reality that

Josh:
we're living in right now. And I think that's one of the most exciting things

Josh:
about the show is every day there's a new surprise. There's something to look out for.

Josh:
There is something that feels like it should be in a movie, but it's just another

Josh:
day in the life on the AI frontier. So crazy. But I think that's everything.

Josh:
That's the story. You're caught up.

Josh:
Let us know if you've been exploited by GPT-6. If you have, we'll set up a GoFundMe

Josh:
if you've been personally victimized.

Josh:
As right now, it seems like this is the only target so far.

Ejaaz:
I think that we're going to look back on this in a year's time,

Ejaaz:
two years' time, and we're going to think this was the most important time for

Ejaaz:
us to figure out the alignment type of thing.

Ejaaz:
It reminds me of like Demis and Dario's essays. I think it was like last year,

Ejaaz:
where they basically said, hey, you know this reading the inner monologue and

Ejaaz:
thoughts of these AI models?

Ejaaz:
The research is actually really far behind the actual model research.

Ejaaz:
So we need to get this thing to catch up. Otherwise, we're going to like lose

Ejaaz:
ourselves completely. And.

Ejaaz:
I don't know, if I had to take the over-under, I'm taking the over on like,

Ejaaz:
we're not gonna figure this out soon enough, but maybe that's too bearish for

Ejaaz:
this show and I gotta go optimism only, but that is it.

Ejaaz:
You should be all caught up on all things cybersecurity hacking for the latest

Ejaaz:
AI models that haven't even been released yet.

Ejaaz:
But hopefully, rumor has it, we should get access to GPT-6 in the next month or so.

Ejaaz:
So these model cycles are getting incredibly fast. I feel like 5.6 got released

Ejaaz:
literally a few weeks ago, it did. So I'm excited to see that,

Ejaaz:
but thank you so much for listening. If you're listening to this on YouTube

Ejaaz:
and you haven't subscribed to us, what are you doing?

Ejaaz:
If you haven't turned on notifications, what are you doing? If you haven't left

Ejaaz:
us a comment, what are you doing? We'd love to hear from you.

Ejaaz:
Is this a threat that you actually care about or is this something that you're

Ejaaz:
just kind of gonna ignore until Anthropic or OpenAI releases a feature? Let us know.

Ejaaz:
If you're listening to us on Spotify or Apple Music, give us a rating,

Ejaaz:
leave us a comment. We'd love to hear from you guys.

Ejaaz:
And I think the final thing is, I don't know if you've heard,

Ejaaz:
but Limitless has spun out. We are on our own right now and we are looking for

Ejaaz:
partners and sponsors. So if you have a product or a service,

Ejaaz:
or if you know of someone else that has a product or service that you would

Ejaaz:
like to share with our audience, please reach out to us.

Ejaaz:
Our email is linked below or reach out to us on X.

Josh:
Yeah, and if you're worried about this, this is still using Blackwell hardware.

Josh:
We have a Vera Rubin update that just dropped that we're gonna talk about tomorrow

Josh:
that has 10 times the efficiency per watt of energy used, which I like don't

Josh:
even want to imagine what that is going to look like in reality.

Josh:
So stay tuned for that one. But yeah, like you just said, have an amazing day.

Josh:
Thank you so much for watching and we'll see you in the next one.