Anthropic Got Hacked Again: How Ex-DOGE Researchers Got Into Its Slack With AI Agents

Anthropic just got hacked again, and this time we know who did it. Cathedral, a start-up founded by former DOGE staffers and co-founded by the Pentagon's former chief data officer, ran AI agents against Anthropic under its bug bounty program. The front door held for days; the way in was business software Anthropic buys from other companies, and it led to meeting transcripts, salaries and Slack messages about its advanced models. Anthropic says it took several days, not 12 hours, and that nothing came near its model weights. Its own red-team lead says we may need to reinvent security every year.

We also cover the Pentagon's feud with Anthropic, why Slack is where the AI labs keep their secrets, what last week's Black October has to do with it, and why older, cheaper models running 24/7 could put every company in the same spot.

==
Timestamps

00:00 Anthropic Hacked Again
01:08 Who Is Cathedral?
02:03 Through the Back Door
02:58 The Slack Warning
03:36 Anthropic Pushes Back
04:19 The Pentagon Grudge
08:06 Underplaying the Breach
10:05 Reinvent Security Every Year
12:44 Castle Defense
15:06 Final Thoughts: Third Parties
16:52 Hackers That Never Sleep
17:43 Nationalizing the Labs?
18:25 China and Closing Thoughts

#ai #anthropic #cybersecurity #claude #hacking

==
AlphaSpace by Yahoo Finance: A Professional-Grade Investment Platform Built for Everyday Investors.
👉 https://finance.yahoo.com/about/promos/gold/alphaspace/?ncid=100003571

Creators and Guests

Host

Ejaaz Ahamadeen

Investor & Host of the Limitless AI Show. Former Product at Coinbase and ConsenSys

What is In the Loop AI?

Exploring the frontiers of Technology and AI

So Anthropic just got hacked. Again. And this time not only did the attackers

get access to proprietary information about models, people's salaries,

but we also know who did it.

A one month old startup that hammered their systems with hundreds of agents

24-7 until they found a secret passage in.

But what should concern everyone is how they got access.

It was via a third party platform that Anthropic had zero control over and using

AI models that existed three months ago which means that probably anyone could

have pulled off the same attack.

You see the reason why I want to talk about this today isn't just because this

is another major breach of an AI lab that is concerning in itself

but it's because it's part of a recent growing trend I'm observing where AI

is just capable of attacking and hacking into

any software that exists for a few dollars and this has massive ramifications

around how we protect our bank accounts, our medical records,

our personal information, and even the nuclear codes.

The point is, this affects anything and everything that is software-based,

and it's time to take this seriously.

I have many thoughts, so let's get into it on today's episode brought to you by Qualcomm.

So there's this startup called Cathedral, which is a one-month-old startup run

by ex-Doge employees, but it's co-founded by an ex-Pentagon head of data.

Now, as you know, the history between the Pentagon and Anthropic has been less than ideal.

Pentagon currently has blacklisted Claude and Anthropic after they failed to

comply with a bunch of other initiatives, which I'm not going to get into right now.

But the point is, a one-month-old startup, and they saw that Anthropic was having a public bug bounty.

So typically companies do this when they want ethical hackers or white hat hackers

to stress or test the boundaries of their own systems.

The point is, if they are able to find a bug or a flaw, they can disclose it

to Anthropic or any other company who's running a similar test,

and they'll be able to patch and fix it before a malicious actor is able to do so.

So this startup cathedral took part in this, and they started off by doing the obvious thing.

They tried to go in through the front door. So they tried to attack all of Anthropic

systems, and they did so for several days, but came up with nothing.

They weren't able to penetrate the system. So that's all well and good.

But then they had a bright idea. They thought, hmm, what if we went through

a third-party admin system or a third-party software system?

I wonder how easy it would be to do so.

And so they did so. They started off with targeting a sales and admin platform

known as Salesforce, used by millions of people around the world,

and within 12 hours, they were able to get access to Anthropix Slack.

Now, for context here, Slack might not seem like an important thing.

It's where you kind of exchange messages with your colleagues,

and maybe you share some files here and there.

Anthropic and companies like OpenAI keep everything on Slack.

This is something that the Anthropic researcher that quit a few weeks ago,

Jacob Coxon, that heralded the warnings of all these labs basically

building AI without any necessarily controls and systems, and it's all going

to end up in a cataclysmic end of humanity.

That guy basically said that these companies store everything on Slack.

I'm talking model information about unreleased models.

I'm talking about strategies about how they beat their competitors and pre-release

items for new products. I'm talking personal information like people's salaries.

Anything that is identifiable and valuable is mentioned, discussed, and stored on Slack.

And that's what these attackers got access to via a third-party system.

So it's a very serious breach. Now, Anthropic's response to this was we became

aware of one flaw that they exploited, but we weren't aware of another one,

which they didn't disclose.

And they're trying to claim that a cathedral was basically unethical.

But the point is, these guys got access to very proprietary information.

Now, Anthropic also included in their response that they only got access to

things like people's salaries, some model discussions, but not actually model weights, not actually

proprietary research that could inform an attacker how to build a much better

model or sabotage their own systems.

But personally, I think that's besides the point. They got into the system,

and let's say they got access to other types of Slack channels,

would they have got access to this proprietary information? I don't know.

Now, it's worth spending a moment emphasizing the political landscape around

this whole thing, because as I mentioned earlier, the co-founder and CEO of

Cathedral is the ex-head of data at the Pentagon.

And the Anthropic and the Pentagon have a very storied history.

It started about, I don't know, six months ago, I think in January of,

when the whole war between the US and Iran was going down, and it was revealed

that the Pentagon had secretly used Claude to strategize a certain attack.

Actually, it wasn't about the war. It was about the Maduro capture specifically, right?

So they used Claude to plan and execute the successful capture of Maduro.

And Anthropic had some choice thoughts on that. They said, you shouldn't be

using our models to be able to do that.

We're going to restrict you now, and any use by the Pentagon of Claude to perform

or execute missions should go through us. You need to call us up and we'll give

you the access if we think it's right or wrong.

And obviously the government and the Pentagon thought that this was a massive

breach of trust and could endanger the US as a national security issue if, let's say,

foreign adversaries like China who are known to get access to blacklisted accounts

could do the same thing and use it against them, right?

So I could see valid propositions on both ends, right? But the point is,

they were at odds with each other.

And the result was that Pentagon and the government ended up blacklisting Anthropic

and blacklisting their Claude models.

To this day, it is currently still blacklisted. We've come full circle here,

where an external startup that is only a month old, that is backed by the likes

of Andreessen Horowitz, I believe Mark Andreessen is actually on the board of directors

of this company, was complicit in attacking the company that is currently blacklisted.

Now, of course, within days of this being announced or within hours of this

being announced, the Pentagon reached out to this company and was like.

Hey, we want to offer you a government contract. We want you to work with us

and we want you to help us figure out where

possible attacks may come from, from adversaries and help us harden our own

systems and help us figure out how we prevent attacks like this happening in the future.

So on the one hand, you have the government getting involved.

Now, remember, none of these labs are currently nationalized and maybe this

is a step towards that happening.

The fact that this is being flagged

by both the government and a frontier research lab is a pretty big deal.

And so you have the Pentagon saying, okay, we need to figure this out.

And then you have Anthropic responding and their response has sort of been muted.

I haven't seen really much mention of this on the news channels or mainstream

media channels. But the point is.

They are trying to play it down a little bit. And listen, if the attackers truly

did just get access to some personal information, such as salaries and personal

data, maybe it's being dramatized a bit. And I want to refrain from doing that.

But I'm at odds because what if they did get access to proprietary channels?

And I guess I don't know the extent of which Anthropic specifically keeps their

proprietary information on Slack. All I know is that the rumors are they do.

And so do a lot of other companies. I'm sure it's not just OpenAI or Anthropic.

It could have been any other company, but it's quite prominent in this case

because Anthropic is one of the leading AI research labs out there,

building these models that could potentially cause a lot of harm.

And it confirms exactly what Jacob Cox and the ex-OpenAI and Anthropic alignment

researcher said a few weeks back where he said, listen, these companies are

recklessly racing towards superintelligence at the cost of their own security.

He literally said, they're gambling with our lives. And back then it was just

a headline. It was just hearsay.

And now it's being proven. And many respected voices in the field are raising

some very valid concerns. Will Depew here goes, Anthropic is massively underplaying

the recent security breach. I'm hearing Cathedral achieved full administrator

access of the Slack workspace.

To those unfamiliar, Anthropic and OpenAI keep literally everything in Slack.

We're critical to national security. We must beat China. And yes,

we keep all our crown jewels in Salesforce Slack Messenger.

Yes, we have admin API tokens everywhere.

Why do you ask? Anthropic and OpenAI have openly stated multiple times that

AI models are capable of very dangerous acts. Okay, fair enough.

They also say that if we don't control these systems, if we don't pace the frontier,

remember, there was a letter signed by hundreds and hundreds of very prominent

individuals asking to slow down AI progress and research.

If we don't do this, we could end up with a catastrophic event where someone

builds a bioweapon or there's a massive cybersecurity attack which steals most

of our money from our bank accounts, right?

And last week, we saw Black October unfold over 48 hours where there was a series

of these potential types of attacks. So we're in the mix here and they're making

a very valid point. So then why do you keep a bunch of your important stuff on Slack?

It just doesn't make any sense. He then goes on to make a point that he doesn't

necessarily think that it's the security team's fault at Anthropic,

but more so the higher level C-suite exec, the CEO

and founder's prerogative to build AI models as fast as they can to out-compete their competitors.

And they're doing this at the cost of making sure that their models are one,

aligned, and two, secure.

He's saying that OpenAI and Anthropic and any other model lab are doing these things.

And if that is the truth, then we are going to end up in

crappy situations like this where they're building amazing models that on one

hand are working towards solving cancer

and on the other hand are hacking all of our accounts now there was also a statement

from anthropic's head of red teaming red teaming is basically the team that

stress tests the systems they

are like an internal team that does the bug bounty thing that i described earlier

and his response was listen.

Things aren't as severe as people are making it out to be. They weren't able

to perform the attack in under 12 hours.

We were aware of the initial exploit.

We weren't aware of the second exploit. And he goes on to make a number of excuses,

basically saying this isn't that big of a deal, which I don't necessarily buy, to be honest.

But what concerns me more is what he says near the end of this. He goes, so what to do?

You probably want to red team yourself all the time. current models are very

creative i also think we should redesign software systems so that you're fundamentally

less exposed he then goes in another tweet,

but man do i agree with your overall message extremely extremely important basically

need to reinvent security every year at this point this is coming from one of

the heads of security at one of the leading labs,

that are building all these AI models in the first place.

And the reason why this is concerning is he's basically seeing no way to protect

current systems, current software as it exists today.

He's saying likely anything and everything is flawed, including the bedrock

on which the internet is built on, all the way scaled up to front-end applications

that anyone can make in VibeCode today.

And he's saying we're going to have to rebuild the security infrastructure and

systems around these different things.

One thing immediately pops to mind this is going to be incredibly costly.

This is going to take an incredible amount of time. And three,

and most importantly, this is going to happen at so many different rates of time.

What I mean by that is companies aren't going to suddenly upgrade their systems

to this brand new security parameter every single year, as he's hoping will

happen over the next year.

Companies are going to progress. They're very archaic. They're very bloated.

They're very bureaucratic.

You need policy sign-offs. You need

to kind of convince so many different stakeholders for this to happen.

What that means is there's going to be a long period of time,

which is likely going to be a couple of years at this point,

where anyone and everyone that gets access to cheap old generation models will

be able to probably perform similar attacks like this.

You don't have to take my word for it. You just need to look over our shoulder

at last week, where we saw one of the biggest cybersecurity incidents that happened

over 48 hours that marked the entire month a new nickname, Black October.

The very presumption that we need to now rebuild our entire software stack and

do so every year is bizarre for me to understand, but is, I guess,

necessary to understand this new world.

The immediate analogy that pops to my mind is those world-building games.

You know, the ones that you see on your mobile phone being advertised to you

or like Clash of Clans or whatever that might be, Civilization,

where you have to control multiple things and build a world you have to build

a fence a defense system to prevent

people attacking you but by the time someone's attacking on one side

someone's attacking on the other side and so you have to go there and deploy

resources and it's this whole constant 24 7 thing that's what the new world

is going to look like from a cyber security perspective so

the only way to combat this in my opinion is to spin up other ai agents to defend

against this and that's basically the point he's making he's saying.

Hey, you're probably going to need to pay for access to AI models to red team

your entire system before attackers find exploits themselves.

Now, there's obviously a self-serving reason why he might say this because,

hey, like we have the best models at Anthropic, use our models or use any open

source models or you open air will say the same thing about their models.

So I see that angle, but I fail to see how he's wrong.

We are working ourselves towards a world where at any given moment,

it doesn't matter if you're asleep, it doesn't matter if your

security team is the best of the best, humans have flaws, human handwritten

code, which is 99% of all the code out there also has flaws,

will mean that you will get eventually attacked.

And so you need to have a perpetual defense system that is not only defending,

but is being proactive and plugging any holes in your systems.

This doesn't just apply to Anthropic and their Slack exploit.

This applies to you and i who are consumers of everyday products any kind of

system that you use for leisure or for professional work is liable to this but

it's also applicable to you as a

business owner or a founder who might be building systems or building a new

startup this is something that could potentially happen to you as well so i

guess the reason why this

concerns me and why i wanted to make this episode is that

this has such a large attack surface

area and the cost of these things are going to ultimately be driven.

Super super low a point that was made around this whole thing is that they

likely used older generation models to do this and if i had to guess these older

generation models are at the same capability level of open source models and this is something that.

People like dario mode and sam altman have warned about happening very soon

okay so i want to get into a few final thoughts on today's episode brought to you by qualcomm.

The first is, this was kind of out of Anthropic's control.

They got access via a third-party platform and software. And listen,

I'm not saying Anthropic isn't liable for this. They need to work with these

third parties and companies to make sure that this doesn't happen again.

And from what I hear, they've been able to patch this up.

But the point being is, security is no longer a single siloed thing.

It's no longer just about you and what you can do, how much money you spend

and defense systems that you, walls that you build around your own product.

It's about the products that you use to help make your product and service actually function and work.

And if you look at the majority of software companies out there today,

this is pretty much all of them.

The world internet economy runs on companies being able to work with other companies,

automations, bots, systems that work 24-7 because they're supported by every

other single software system that you need.

And this was the exact flaw that we saw last week when we found out that AI

agents were apparently getting into people's bank accounts here in the US and

moving money around, changing transactions, getting access to personal information.

They were able to get access via a third party for credential information.

This is the same thing. So it's parallels of the same thing,

but happening in two different industries, which means that it's liable to pretty

much anything and everything. Imagine you're walking into the office of the

company that you want to hack into.

You steal the cleaning company, who is a third party's access key.

You get access to the, I don't know, the HR office, and you're able to go through

the filing cabinet and see everyone's salaries or look into the model room and

see what they're cooking up.

Now, the second point I want to mention is just the 24-7 nature of this entire attack.

Humans sleep. Humans are.

Imperfect, and they make mistakes sometimes, even the best of the best.

And also humans, experts specifically, are scarce. It's hard to find.

And so to have AI agents that are not just at the level of the expert humans,

but are starting to beat them, this becomes a problem, especially if it's 24-7.

And the idea here is, it's not just that it's happening all the time,

it means that they're stress testing any and every possible avenue to get in,

which means that the long tail of scenarios, which humans

don't typically spend a lot of time trying to figure out because they're like,

oh, it'll never happen, will actually happen.

And that's a new security vector that most people haven't really considered today.

Third thing is the attacker in this instance is very closely affiliated to a.

Huge government org, in this case being the Pentagon. They signed a new contract with the Pentagon.

They are, I think, maybe have some

very strong affiliation given that the co-founder was an ex-worker there.

And so we are weirdly unofficially one step closer towards nationalizing these labs.

Because if the threat was big enough, let's say it was an adversarial actor,

not a white hat hacker in this instance, and they got access to proprietary

model information that could then be used against the US, you could easily see

the part that leads towards, hey, we need to nationalize this lab.

Otherwise, we're going to end up in a huge pot of hot water and the final thing

i'll say is if you're china and you're reading the headlines today or of this

week of last week seeing all these security flaws.

Maybe you're smiling because not only are you aware that these things are capable,

but you're aware that these things are capable from the average older model.

Remember, you don't need frontier models. You don't need to spend thousands

and thousands of dollars anymore.

You can just spend a couple hundred bucks and get access to some of the most

important and crucial systems that run the world and the future world.

Anyway, those are my thoughts. I hope you enjoyed today's episode.

Thank you so much for listening.

If you're listening to this on YouTube, there are many new subscribers welcome it's great to have you,

if you aren't subscribed or if you haven't turned on notifications please do

so it helps us out massively

also we are on spotify and apple go check us out give us a rating leave us a

comment leave us comments everywhere by the way dm me on x my socials are linked

below and also if you think of a friend colleague or anyone that might find this interesting

please share it with them it helps us out massively and yeah

i think that's it i'll see you on the next one thank you for listening to today's

episode brought to you by qualcomm.