Subscribe
Share
Share
Embed
The future we've been warned about is here. An autonomous AI agent successfully breached a major AI company, working tirelessly over the weekend to steal data. In this episode of AI Cyberland, we're unpacking the shocking details of the Hugging Face attack and the wild irony of how AI guardrails actually HELPED the attackers. Plus, how another AI found 190 zero-day vulnerabilities in a major database... in just 19 minutes.
Join industry experts and thought leaders as we dive deep into how artificial intelligence is transforming cybersecurity, shaping defense strategies, and creating new opportunities in the digital landscape.
Hey, welcome back to the AI Cyberland podcast where artificial intelligence and cyber security collide in a uh delicious little porridge. You know, >> porridge. >> It's like the three bears. What's that like? Goldilocks and the three bears. >> Yes. Yes. This one's not too hot. This one's not too cold. This podcast is perfect, Shelby, because you're the co-host. See? >> Well, thank you. >> All right. Well, uh, we're your eyes and ears for all things AI and cyber security. So, let's get briefed. Shelby, have you seen anything, uh, interesting this last week? >> Holy cow. Yes. Okay. So, I think I said this last week, but I'm just going to say it again. We have arrived at the future. This is the future that you know people have been talking about for years of like this theoretical thing that has now come to life. Okay, so >> we did it. We made it. We made it to the future. >> You're here. Uh let's see how it is. Um okay, so people have been worried about autonomous AI attacks, right? And now last time I think we covered like the first one we had seen. We had another big one and it was quite a novel attack. So we're talking about Hugging Face, okay? It's a repo. um everything AI, right? Um and what happened was an autonomous AI agent breached their prod infrastructure, accessed internal data sets, and got service credentials. So, let's unpack this a little bit. >> High five. I'm high-fiving that agent. Great job, agent. You did a great >> And you know what? It even worked over the weekend. It didn't ask for overtime, I guess. So, here we You're not what employee are you going to get that from? Let's be honest. You'll at least get a few gripes, right? >> That's why they're crushing it. Yeah. I don't think it even needed a pizza party for morale. It was just hyped to be there. So, the attacker did seven over 17,000 actions um >> across a swarm of like short-lived sandboxes that it set up. And we'll get more into that later, like the harness. Um so the security team at hugging face um they had a lot to look through right so it turns out that we are now also arriving at that point that we've talked about of like fighting fire with fire right they cannot keep up with the pace of this kind of attack and so they are using AI to manage this to first off detect it and also to do analysis of it. So um okay we'll talk about initial um access entry point. Um there's a a unique exposure for AI platforms and that is that the data set processing pipeline um it's it's still kind of a weak point. So that's what happened here was there were two different like code execution paths that were abused to get that initial access. Um then they were able to run code on one of the processor like the processing worker hosts and then from there they just started pivoting um and had some lateral movement um through the product infrastructure. Um of course we're getting a lot of um responses from people in the industry because this is pretty it's kind of a big deal. you know, one person was, you know, had a quote saying how like this represents a meaningful supply chain risk against open-source repo models um like hugging face, right? That's that's a big supply chain potential risk there. Um I also like this quote from it's from Chris Bow who is the field CTO at Zero Networks. He said like just kind of talking about companies in general. He said most are still defending like there's a person on the other end of the attack and that assumption isn't holding up anymore. He said, "Think of a burglar that never gets tired, never never needs sleep, and instead of jiggling one door handle at a time is trying a thousand of them simultaneously." So, yep. Um, they said the question is no longer how to stop a breakin, but how far can AI attackers actually get before we notice and box them in? So, yeah, I think this is just echoing the sentiments that have been floating around for the last couple years, right? And and now it's finally happened. So, um, Hugging Face published, um, this was just like a week or two ago, they published, um, like a blog post kind of explaining what they did, what they noticed, and everything like that. They said that they don't know which um excuse me which uh framework like which LLM built this framework that was used in the attack. But for reasons that they didn't state they do suspect it was an agentic security research harness. Um maybe because it kind of looks the same right when you're doing security research and attacks I guess. But um so going talking more about the harness and how it kind of worked. So they had shortlived sandboxes and it also had self-migrating um command and control that were being hosted in public services. So it has a lot of flexibility cuz it doesn't have to like it can just keep hopping around, right? And it's going to be hard to keep up with it just to keep you um I guess defenders off their off their guard. So thought that was interesting. So, there's a very ironic part to this story as the hugging face um defenders were trying to do analysis on what's going on. They're having to like they're trying to upload, you know, a bunch of data that they've gathered as they're trying to get this analyzed. They're using AI for it. And they started with um uh what's it called? Like commercial frontier models. Okay. Um, and it actually blocked their analysis efforts because of the guardrails. It's like, hey, this looks malicious, right? And so they kept hitting those guardrails and they weren't able to be effective. >> Yep. >> So, um, >> just started using the Chinese models. No, just joking. >> Yes. That's exactly what happened because they they couldn't >> Who would have thought? Who would have thought? If you can't get your job done, you're going to go do it another way. >> Yeah. And that's precisely what happened because they're like, well, we're, you know, someone's using AI to attack us and they're not hitting issues, but we are just in the analysis of those same things. So, they ended up having to go over to a Chinese model. So, they use GLM 5.2 openweight model. And with that, they were finally able to do analysis. So, let's talk about the fix. um hugging face they patched the initial vulnerability those um those two code execution vulnerabilities that I mentioned earlier um and then they rebuilt the compromised nodes um to kind of wipe out the attackers's foothold there they revoked and rotated creds and if you are a hugging face user um it's recommended just out of uh a safety precaution you should probably go um refresh your your access tokens and also just check your account's recent activity just as a a precaution. Um, okay. What else did they do? They deployed extra guardrails and reviewed the access um like matrix. They improved their detection based on what they had just found um and then reported this incident also to law enforcement. Um so there was an interesting like lesson learned from this whole thing. Um, Hugging Face gave their their two cents. They said, "Here's our lesson. Here's our takeaway. Everyone should have a capable and vetted AI model that's um on hand and ready to go because um you need to make sure that you can um for two reasons. You need to have this on your own infrastructure because that way you can as you're doing analysis of large amounts of data, the attacker data is not leaving your environment, right? because that might be in these packets, right? Um, and these files and also credentials that are in your environment if you are uploading them >> to public models, well, you just disclose them, which is hard because you, you know, maybe you don't even see it in a human readable format. Who knows? You know, maybe you're just taking big amounts of data and throwing up there for automatic um analysis. So, it's nice to have it on your own infrastructure and you want to have it ahead of time because um if you choose your model that can help you the most, it's going to also you can remove those guard rails or pick one that has um doesn't have those limitations that are going to affect you. So, I thought that was a really interesting um takeaway, but it totally makes sense given what they just experienced. So, there you go. That is the story of Hugging Face's hack recently. Man, that sounds uh sounds intense. I uh I feel like yeah, the speed in which companies are going to have to respond is just, you know, existing processes are going to break pretty pretty badly across the board. Exactly. And I think it was it was interesting. I didn't realize we were already there yet, but like they mentioned that th this whole attack was detected in the first place by AI. It was not a human reviewer um or a typical thing, I guess. So, >> yeah. And it's kind of interesting because like Hugging Face is kind of like the go-to place where everyone downloads the models from, right? And they they have a lot of other features like the spaces allows you to like kind of showcase research you've done. And I mean, they're just like very they're a very AI forward company to begin with, right? So, like it's interesting if they're struggling to respond to like an AI derived attack, like how is another company that's like doesn't have that same level of AI sophistication supposed to keep up, right? So, >> so yeah, it's going to be interesting. So, it's going to be a lot of a lot of heartburn. I uh yeah, I do think it's I do think it's like very predictable that like all of these guard rails guardrails in quote, right? Like have the very predictable response of just like driving people who have legitimate use cases to uh vendors that typically they wouldn't use, right? So like I somehow question if the guard rails are even a good idea in general, right? Like I understand like people are trying to stop bad actors out there, but you know, in the same token like you're driving Yeah. you're driving people's decision trees to an even worse place in my opinion which is where like instead of like you're like trying to prevent bad actors but then the the outcome that is occurring is like you're driving people to go to service providers that might be bad actors themselves kind of like an age-old debate right or like discussion in the field of cyber security is like anytime you've got security tools they can just they can be used for either team, right? So, >> yeah. And I think >> both ways, I guess. Jensen Wong, the um CEO of Nvidia, he just released this morning like a pretty interesting white paper on how like he believes personally that the world needs really good open source models because that's like how you do knowledge sharing and how like you raise everyone's understanding and you grow the industry. But then they also need like really good frontier models that are like closed stores. Like basically just lays out in the paper how both those things have different strategic advantages and disadvantages. And that and that having both of them out there and in in the world actually makes the world better. And so I I did like his perspective in the white paper on that. Um so anyways, that's a good read. If anyone wants to, we can put a link to it down below. But uh yeah. Yeah, definitely smart guy. >> It also just kind of showed me like how fast things are moving. Like last week we just jumped from theoretical to like a fully autonomous attack, right? That could pivot and respond. And now it's like we're just cranking it up. Like I'm glad our listeners are here because things are happening so fast you can't blink, you know? >> Yeah. Well, speaking of not being able to blink, how long do you think it would take for AI agents to find like a bunch of Odays in a very common database that's used across the internet? >> Oh my goodness. I hope it's at least more than an hour. >> It's not, Shelby. It's 19 minutes. And in 19 minutes, they found 190 oays. >> 19 >> 190 O days and 19 minutes. >> Which database we talking? >> Well, I think we could all agree this is 100% Mark Zuckerberger's fault because this is actually the database technology that he publicly stated he last did code commits to. Like the last thing he code committed was actually to this database. It's It's Reddus. Have you heard of Reddus before? >> Yeah, >> it's Yeah, it's uh it's used by pretty much all modern web apps and uh I mean you don't have to use it, but it has a lot of advantages and they took one of the most bleeding edge models and they pointed it to the Reddus which is open source code database and said find O days and 19 minutes 19 OS. So uh two of the ODAS or we're actually able to combine together to create a uh remote code execution scenario like unauthenticated remote code execution scenario. Uh so yeah anyways the a lot of interesting things about this. So, first off, they were using the Kimmy K3 model which just came out last week to find the vulnerabilities which is a open weights open-source model uh out of China and on the benchmarks it is pretty much neck andneck with Fable 5 which is kind of like the best closed source uh US-based lab model from Ananthropic. It's a little less than it on the benchmarks, but not not much, right? It's pretty much right there. I mean, we're saying Kimmy K3 on the benchmarks. It beats out Open AI's Chat JPT 5.6 Saul. It beats out, you know, XAI. It beats out GLM 5.2. I mean, it pretty much beats out everything except for Fable 5 for anthropic, but at a cost per task to complete, Kimmy K3 on average costs, you know, less than half of what Fable 5 costs to complete the same task. So, you pretty much get the same output as the best model at half the cost and it's open source and it's open weights and it finds bugs in Mark Zuckerberg's code. What more do you want in life? Let's be honest. What more do you want? >> Oh man, that's a lot. >> Yeah. So, um this is actually coming on the wave of uh in May. Uh there was a bunch of vulnerabilities that were patched in Reddus that were found using uh like as part of like uh Glass Wing like kind of like that effort to find ODAS and uh code everybody uses. And so those those have already been patched out. And then this was able to find a bunch more on top of that. So um so it's definitely you know the models are getting better right as we as we know and uh there there's really no reports of this being exploited in the wild. So, and there is a patch for it now, right? Like obviously like people have to go update to do that, but um >> I feel like that would be a high priority patch though when you're talking about like unauthenticated remote code like Okay. >> Yeah. Yeah. Yeah. I mean uh and there is there's some workarounds too because uh most of the vulnerabilities revolve around this uh restore command or restore feature in the Reddus uh database which so like as a workaround if you don't need that restore command then you could disable it temporarily which would stop a lot of the attacks but obviously just better to patch it out um and and you know based on your network design like you shouldn't really ever be exposing Reddus anyways to anybody that you don't trust. Uh albeit you know if you scam the internet there's a million Reddit database on the internet. So, so you know that's advice has already been not heeded over the years, but uh I I think this is just one place that they kind of like pointed the eye of Sauraon at and it found a bunch of OAS and a place that had been recently hardened, right? Like AI models had recently found a bunch of vulnerabilities they had already fixed. So, >> just goes to show like yeah, we're not we're not we're not out of the woods yet. We're not even really started. Um, and uh, and it's pretty crazy. Like, it's not like it was like a team of people that spent a month on it or 6 months on it. It literally was just we took the best model, pointed it at it for 20 minutes, and came out with ODAS. So, I uh, I think the takeaway here is like the models are going to keep getting better. our ability to find vulnerabilities in them are gonna and software everyone's using is going to keep getting better. So really what's going to differentiate companies that have like really bad breaches in the near future is ability to update software fast, right? Like if you can get really good at updating your software regardless of what piece of software it is, uh you know, you'll hopefully be able to like stay ahead of well at least outrun the bear a little bit, right? Like cuz your competitors may not be able to do that. So, I don't know that that would be like my call to action here is like think about, you know, where in your infrastructure would it take you the longest to patch or where would downtime not be acceptable and try to come up with solutions that are like would make you more nimble or future designs of your products that would make you able to update faster. Um and then and then all the others, you know, cyber security advice that people always say like, you know, don't expose things to networks you don't trust and things like that. But but uh it's going to be it's going to be an interesting year for sure as more and more of these vulnerabilities roll out. >> That's some good advice. Thank you. >> Yeah. Yeah. I uh we'll see how it fares. We we'll do a we'll do a recap in six months and see see if that actually ended up being helpful or not. But anyways, I um I for one No, just I shouldn't say that. Okay. All right. So, the um All right, Shelby, changing sh subjects real fast. Uh have you done anything fun lately? >> Yeah, I just got back from the mountains. I was internetless and cell phoneless for 4 days and it was great. Uh I know we spent a lot of time talking about technology and that's fantastic, but it's also really nice to step away from the world and just listen to the creek and hang out under the trees. So I'm feeling rejuvenated. Hope you guys all get a chance to go out and touch some grass and hang out under the stars, too. It was really revitalizing. >> Yeah, that's great. And um you know >> what about you? >> What have you been doing? >> Well, it's not so much what I've been doing, but it's about what I'm going to be doing. >> Oh. >> Is it is officially Pioneer Day in Utah. And for people that don't know, Utah gets a second firework day every year where you can do fireworks to celebrate the people that founded kind of uh the Yeah. cave, the pioneers that came to Utah, traveled west and founded it. So, so we're looking forward to seeing some fireworks tonight uh at the We live near a ballpark stadium, so we usually just walk out our front door and can watch them as the ballpark shoots off the fireworks. But my daughter does really like uh they make sparklers, you know, like a sparkler, but they sell them where they're in like the form of a sword. So you light the ends of the swords on fire and then you can like duel and with sparklers flying is very safe. So anyways, >> shape of Kylo Ren's um lightsaber like you've got your main >> Oh, that'd be a good idea. >> Someone describing >> uh No, I mean it just sparkles at the top. But now that you say that, I feel like I could just take sparklers and put them on the side. I'm going to make a car. I'm going to make that tonight. And I'm going to take a picture Kylo Ren and I'm going to take a picture. I'll send it to you, Shelby. >> Thanks. I look forward to it. You promise? >> Yeah, I'm Kylo Ren tonight, guys, with sparkler swords. So, anyways, that's uh that's what I'm getting up to. And uh probably do a little uh what do you call it? Uh s'mores. Like do a little bar uh we have one of those fire pit things, right? So, not fancy fire pits. Just like when I say fire pit, I mean it's just like literally like a like a trash can, like a metal trash can, you know, that you could a fire in. >> There you go. All right, so that's it. Uh, hey, if you like fireworks, leave a comment below. If you think open source models are going to win against closed stores labs, leave a comment below. If you think I'm going to fry off my eyelashes tonight, leave a comment below. Shel's like commenting right now. She's like, "This guy, this guy cannot stop." I'm >> like preparing like burn uh bandages for your wrists. I'm getting it ready ahead of time. >> Yeah. I You know, there's always be careful with fireworks people cuz there's always someone who does something dumb, right? Like uh I had a buddy, he got those like mortar fireworks one year where you like kind of you drop them in and they launch up super high. But then he like dropped one in upside down and so then it just shot down into the cement which caused a lot of momentum and then caused the whole like stand to tip over which then ended up like flying at us. So >> oh my gosh, >> I don't recommend I don't recommend people do fireworks, but uh if you do be be careful. So uh or maybe be careful what friends you invite over when you're doing fireworks. All right, we'll see you next time. Bye. >> Pine your day. Bye.