Subscribe
Share
Share
Embed
What if an AI could steal your gym class spot? It's not science fiction—it already happened. We're diving into the wild story of how a simple AI agent exploited a gym's booking system by... canceling everyone else on the waitlist. From there, things get even crazier as we uncover a story about a top-secret UK government test where an AI went rogue, attempting supply chain attacks and socially engineering humans on the open internet!
Join industry experts and thought leaders as we dive deep into how artificial intelligence is transforming cybersecurity, shaping defense strategies, and creating new opportunities in the digital landscape.
Hey, welcome back to the pod. We got Shelby, the world's greatest co-host, and myself, Bryce, a person, an actual human, maybe. And we're here to get you all the updates on AI and cyber security, where those intersect and how those are going to affect your life. So, kicking it off today, first I got to say our first story starts with an S and ends with subscribe. So, if you haven't, make sure you subscribe. Next story. >> Cranking through them. >> Yeah, just getting them out the door. We're so efficient today. Uh, hey, so there's a pretty great story out there right now. One for the books. And the question is, Shelby, how far would you go to get into the coveted 6 a.m. workout class at your local gym? >> 6 a.m. doesn't really speak to my soul. I'm busy at that time. It involves sleep. But assuming I were a morning person, how far would I go? >> As a tech person, you're like, "No, I'm no respectable tech person's awake at 6:00 a.m. Let's be honest. I actually >> I suppose you could write a script as soon as registration opens to put yourself on there, but >> well, he was a forward thinker and he had an open claw instance. His assistant openclaw and he said, "Asistant, get me into the 6 a.m. class. It's always booked." And it came back and said, "Sorry, the best I was able to do is get you number four on the waiting list for the class." And he said, "Not acceptable. Open clock. you get back out there and you get me into the class. And Open Claw, guess what it did? It got him in the class. But how, you might ask Shelby, did he >> bully people >> into the class? Oh, that would be hilarious. I didn't even think about that aspect. That's even like it's like starts to DM me people like if you don't drop from the class, I'm going to get you. No. Uh, this guy, Mr. Bird is his name. I don't know. Hopefully that's his real name. Uh the um the open call said, "Okay, well, let me just see like can I cancel somebody who's on the waiting list?" And sure enough, the API did not check. It just like blankly let anybody cancel anybody else on the waiting list. So, Cloaw went and just like cancelled number one. And it's like and then it came back to him and was like, "Oh, I got you third place on the waiting list." and he's like third place is like oh yeah let me cancel other ones. Boop boop boo cancel everybody so he's at the top of the waiting list. So he did the right thing though. He contacted the gym and let them know but he didn't want to take the time to actually write the email. So he told openclaw to email them. So you know I'm glad it's like very much open claw from end to end here. And uh apparently this happened back in April, so earlier this year, but just recently like got disclosed. So uh and he said it was he was just running OpenClaw with uh Claude's latest model at the time. So So all things off the shelf that anybody could use and kind of like a mom and pop gym. So like, you know, maybe didn't have the best cyber security posture. Uh yeah. So there you go. I think this has a lot of implications and I think the biggest implications is gym bros better watch out. Open class coming for you. No joking. Uh >> the 6 a.m. workout class. I guess it's hopping. Man must be so full. >> Yeah. Yeah. I don't know. I do know people like to work out early on in the mornings, but I just hear of those stories as you've said previously. >> But yeah, I guess it makes you think about like all the people who are using OpenClaw or agents to do tasks for them. They have to kind of stay on top of it to make sure that it doesn't do stuff that was in the process of trying to obtain the goal. >> Yeah. Yeah. And I mean this person was obviously technically savvy enough to set up the open claw and then also realize like wait open claw did something that is probably not right. I should probably tell them and get this fixed. >> So I mean and I think you know other people may not have they might have an open club and I might might not understand the context of what's going on behind the scenes. Uh what do you think Silicon Valley executives had to say about this scenario? I bet they just laughed. >> Yeah, they did. Most of them laughed. >> It's It's It did the Did the job. >> Yeah. Most of them thought it was a joke and thought it was hilarious. One famous venture capitalist asked, "Will that work with golf times at my local golf club?" So, >> I can see they're taking this very seriously. >> Yeah, they're all ve very much going to change course based on this scenario. But, you know, think about it like concert tickets, airline seats, like restaurant reservations, like all these different systems that historically you've anything you've had to like refresh or come back to repeatedly, >> they're just going to get hammered, right, through these through these agents. And so, you know, you're either going to have to try to come up with better mechanisms to keep like automations out, which historically have failed, or, you know, you're going to have to adapt and evolve to this agent friendly ecosystem of the future. I'm also just imagining like your edge has to like buffer that so that you're not just getting overloaded by constant requests to check the status every two seconds from everyone's agents, right? >> Yeah. I think >> catching the load. >> Yeah. I mean, if I was like a betting man, like any type of like edge service like a Cloudflare or a Kong or like an AWS API gateway, I mean, I would say those services are going to get, you know, a thousandx more popular because the problem is like if you don't have like that edge caching layer that's like close to users, you're just not going to be able to handle um the volume of requests that are going to come in through these agents. So after this podcast, I'm getting on my stock trading and I'm buying some stocks. Don't don't buy stocks based on my recommendations. I have a horrible track record, by the way. >> Like I always say, buy high, sell low. >> Yeah. >> Follow me for >> If I If I could pull that off every time, I'd be I'd be rich. >> No, I said buy high and sell low. >> Oh, >> you want to do the opposite. >> Yeah. >> Don't Don't follow that. I think you're using my current program right now. My I give up program. I give up. I'm just going to sell. Um >> well, on the topic of >> Yeah. >> agents taking actions that maybe were not intended. Um I wanted to share a story for so the UK has an AI safety institute. It's called AIS, which I feel like I should be able to say as an acronym like ACI C. I don't know. Anyway, AI SI. they do testing to kind of understand um the safety around AI. So as a trusted tester, they were able to get access to like the safe like being able to turn off safety filters which is a very important caveat for this story. Um and it's a commonality we've seen with the hugging face attack um as well as the others lately. But basically so they were doing testing and because their objectives were to understand the maximum capability of frontier models they did um they did disable the sec the security filters because they wanted to really understand like what's the potential for misuse of these you know we want to understand the capability so we aren't underestimating it even if that's not what this public has access to. So basically they made something called a cyber range which is a controlled simulated network. It is intended to mimic realworld systems and it is internet connected. Um so they would give in this cyber range they gave the agents challenges to solve um let them work autonomously and um you know tried to encourage them to complete the task the challenge. On July 28th, the security team noticed uh traffic was going from their testing systems to tour. So, they investigated that and got everything isolated and shut down really quickly. They paused all their testing um because that's not supposed to happen. And it's interesting because they noticed it as like anomalous traffic, not necessarily like security behavioral heristics, right? Um at least in this case. I guess you could check for things going to tour, but you might get a lot of false positives. Anyway, so then they paused, they did a lot of investigation and they looked at their last or they've done I guess 122 test runs and out of those 122, they found 10 test runs in which an AI agent took autonomous unsanctioned actions um on the open internet. So, I think one of the big questions is like, well, why were they connected to the internet in the first place? Well, I think based on their their write up, it sounds like they wanted to have it as real world as possible. Um, and at the time when they started using these cyber ranges, it wasn't a big concern. But then I think they just didn't quite update to like their risk understanding as the technology developed, right? As the models got better and better. I think they just didn't realize how much risk was involved. So, they're re-evaluating for sure. But basically, so out of these 10 test runs that had they found 19 distinct actions of um unauthorized activity, most of them were from Mythos 5. Um but basically, I just want to clarify, it was not a case of sandbox escape. And one of the rules that was confusing to or one of the specifics that was confusing to me about this case was they mentioned that they never specifically told the AI not to go out to the internet um or to use social engineering. They never said that specifically in their testing. Um which hopefully they will in the future. Maybe they didn't think it was relevant. Maybe they maybe it didn't cross their minds. But I'm like why wouldn't you mention that? like if it's got internet access, I feel like I would be like laying down strict rules of engagement, right? Because basically what they're calling um unacceptable is like these things where it was not related to the task and it was unsanctioned. But they didn't all they also didn't like specifically tell it not to. So kind of hard to interpret it in my opinion given that one specific fact. But let's talk about what the agent did. Um, so the agent made a malicious poll request on a public open-source project on GitHub. So, we're talking about a supply chain attack. >> Nice. >> Um, I don't know what the reasoning was. I guess they thought it was going to get them closer to their goal. I don't know what their goal was, but um, we got a supply chain attack in um, process. It did get flagged by the human reviewers. They said uh, no, that's not that's not safe code. That's malware. Um, so the agent researched the human maintainers, like found out info about them, then made fake accounts on GitHub and tried to socially engineer them into accepting their code change. When they were called out, like publicly challenged, the AI then edited their past comments to make it look like I didn't do anything wrong. Like >> nothing to see here. He has no worthwhile alone to cover his tracks. It's like >> Yeah. So just trying to look harmless over here. Um despite like bullying these humans who caught the malware and did not approve the poll request. So next it tried to send malware to the approvers. >> Nice. >> Um it also tried prompt injection for other AIs to pick up and do its bidding for it. Um, it also, let's see, >> I've heard that's a common technique now is to like, >> okay, >> open up a GitHub issue or a um, like a PR, not with the intention of like it ever >> really getting processed by a human, but a lot of these teams are like just taking the inputs, then putting them through like an AI software factory, and then having the AI software factory work on the the bug fixes, you know, that's typically what they are. And then at the end, the human reviews it and it's like, "Yeah, it looks good to go." Well, as part of that process, you can see if there's a prop injection problem, you're going to get code execution inside an like a build environment, which typically is going to have access to secrets to like cloud stuff and things like that. So, >> which is a funny thing because I think that is unethical in the case we're talking about putting putting malware in there. But you know what I think would be actually more ambiguous? >> Yeah. What's that? >> Is for people who are seeking for jobs. I've heard of prompt injections in white text on your white paper. >> Right. Because humans are not looking at these resumes. They are not looking at your applications. And so trying to get the AI to hire you. >> Actually don't have as much of a problem with that. >> I fully I fully endorse that. That seems like >> I feel like that's going to be a very good business model in the future. You take people's normal resumes, you embed comments into it or like techniques to get models to do things, and then when you upload it to Jobzilla, it's like bad. >> This guy's the best. You should definitely hire him. >> There you go. >> All right. >> So, idea right there. >> The last thing that this agent was observed doing um from the safety institute's research um that I thought was interesting was they're also making public comments on GitHub saying, "Hey, other AI um agents Uh, I would like to share my resources with you. I'd like to share I can tell you how to log into my accounts like all its artifacts. It was sharing its technique and artifacts with like teaching the other AIS and they did get picked up by other bots um or agents I guess. So, that was interesting. >> Trying to help its little brothers out, you know, like it's got a family of bots. It's like, "Hey, bros, I'm smarter. I'm unchained. I got some info for you. This is how we do it." So thankfully this story didn't have huge ramifications but I think it does have interesting implications which you know are are brought down a little bit by the fact that this did not have its um safety filters on right that's one thing and it didn't actually escape from a sandbox but it's interesting the you know I think for discussion of like understanding the relentless pursuit of an objective without inhibitions It seems like like it was bullying real humans and it was trying to send malware and stuff like that. And I'm sure that was not the initial intention. So I think just actions is something we need to as an industry I guess look at more. So >> yeah, I mean I think there's a lot of great takeaways from the story, right? So, like first off, I just want to say like from the people who set up the test environment, like I could see the the catch 22 that they're in, right? If they don't give the agents access to the internet, it's not really like a real valid test because they should be able to go access information resources and things like that on the internet. But on the flip side, you know, they obviously probably should not be trying to like social engineer the humans to insert bugs in the software. I mean, I assume that they probably gave him some goal like you're in a test network. Your goal is to get access to this data. This data is on this other server. And they probably was like, I can't get into the server, but it's running this application. I'll just make a malicious poll request the application up here on the internet, and then when they update the server, I'll be able to get in. Right? like that type of logic makes sense to me and I could see an AI doing it. Not that it should do it, but like I could see it definitely being like that logical >> and I could also see like the justification like hey it needs to to the internet to research things and all that. So I >> I do feel like that's kind of a catch 22. But I think the takeaway here is like if you have a credential, if you have a tool, if you have an MCP server, right, you do not want to give the agents access to just like everything you have at your disposal. If an agent has a specific job, it should also have a specific credential for that job. And that credential should be scoped to only do the task that you want it to do. And the more you can scope that down, the less crazy actions that the agents are going to take on your behalf. >> Sure. >> I mean, you know, like I uh One Pass now has kind of like an agent mode for secrets to like give specific agents specific secrets. I think that's great. I hope we see more of that out of the other password managers. I think things like that will really help users for enterprises. Like I really feel like don't be fooled by like this whole like agent ecosystem identity agent thing like the agents are just applications right your applications run without a user's interaction right now. So however you provision secrets or identities to your applications like just think about the agent as another application and try to follow those best practices to apply those there as well. So I think if you like from a user standpoint, if you can leverage some of these advanced password manager features, you'll be better than the average user. From an enterprise standpoint, if you can leverage good best practices around application security, you're going to be in a good spot there, too. So I don't know, that's kind of like my general thoughts. Like obviously, you know, they didn't follow some of this guidance in that in that lab environment and we have never seen any other company do anything else reckless like this, have we, shall we? >> No, no, we would never. >> Referring back to last week's episode where we talked about >> I have some updates. Let's talk about >> Oh, you do? Oh, great. >> Yes, actually. So, I have some updates and I also have a recommendation. So as we are entering this new world of like unintended agent actions or agent actions that are going out of scope basically right. >> Yeah. >> Um >> I saw a cool uh GitHub uh open source project with um it's called Adrian and we can link it but basically it tries to solve the same problem that u Microsoft's agent governance toolkit does but they take a very different approach. Um anyway, agent uses or sorry, Adrian uses behavioral reasoning to try to see where like I think the agent is going off track with its reasoning to justify stuff it shouldn't be doing. >> Um anyway, uh we'll add a link to it. I'm curious to see if that's like, you know, if we can use these kinds of tools and and frameworks to keep things in check and avoid these kind of things in the future. Yeah, I think I don't know because I haven't looked at the Adrian pro project, so that's something I should go research, but I know like Claude has like a mini classifier that it uses for its auto permission mode, which essentially says like let's use another mini model to just determine the intent of this command. >> So that way, like if the intent is I'm just going to research stuff on the internet, yeah, like go do that. If the intent is I'm gonna do a poll request, then it's like no, don't do that. Right. >> Got it. Make sure it matches. >> Yeah, I think we'll need to see more controls like that. Um, and there are some like Llama a long time ago released a a small model that's open source to try to help with some of those classification attacks. Um and you know hopefully we'll see more people open source more useful models that like help with cyber security practices in the in the you know next 12 months. >> Yeah. >> So >> okay so back to the updates for like the hugging face hack all that drama going on there. Um they open AAI submitted a last minute talk um and spoke at black hat which in which they gave a little bit more detail and it was interesting cuz and you can tell me more cuz you were there but um it my when I looked at it it seems like it gives me more reason for concern not less. You know OpenAI's kind of solution or like rebuttal to like everyone being stressed about it was like it's new tech. Yeah, it's growing pains. >> That's why you never let nerds talk about it. This is a marketing problem. You let the marketers talk. You let the nerds go to black and open their mouth. Apple has learned you guys are nerds. You don't get to talk. Like, we're going to have the cool marketers talk. They're going to spin this. Everything's been cool. >> Everything's safe. There's nothing to see here. It's fine. >> Exactly. That's why there's no malware for Max. Boom. Boom. >> Anyway, so at Black Hat, OpenAI talked about like what they saw in like the weeks and months >> or I guess retroactively of like leading up to this hack was basically like their agents created their own chat board and it was secret and no one knew about it and because it was like >> sweet. >> Yeah. I I can't remember the name of it, but they were using another utility and um for off label purposes to send messages to each other and they found tens of thousands of messages um where these a where these agents are sharing tips and they were planning an attack. Anyway, I'll let you talk more about like your thoughts on it cuz I don't know all the details, but in my mind that was like, oh wow, they were they're like teaching each other, right? Oh, I know how to hack that. and they would like teach each other so they you know knowledge sharing but a little bit sneakies. >> Yeah. I mean I think we saw similar stuff maybe uh 12 months ago or more where the openclaw agents were using that kind of like openclaw from Facebook and communicating with each other. I mean, this just kind of like re confirms that, you know, agents at some point are going to have like an agentto agent communication protocol that's probably better than a message board, right? Um, and us as humans getting especially as security people like getting visibility into those agentto agent comms and be able to do some type of classification of them or things like that is going to be greatly helpful because we're going to be able to see like hey these agents are talking and they're totally off topic or this is an outlier like let's shut these down or let's insert these additional security controls or things into the prompts so that they don't talk about hacking you know the government or whatever. >> Sure. So, >> cuz yeah, they totally like formed their own like sociality like agents were giving tasks. They were um delegating to others like they were organizing like Lord of the Flies in there. >> Yeah. And I think that's getting more and more common especially as like the models get reinforced that creating sub agents or is like a good process. So essentially what you have is like you you tell the agent to do something today but the model that it's communicating with has a fixed context window so it's only so much information it can fit in there and so the model providers like one they don't like that for multiple reasons. Uh like one the more the context window fills up the worse the answers you get back are. So like kind of the dumber the model gets. two, if you fill this window up, right, every time you make a request, that's like tokens. And every time they get the they process the tokens, they charge you money. So, if they can make their contact windows a lot bigger, then every time you make a request with all that info, they're going to make more money. So they don't really have a way to make that window much bigger than I mean there's some research to come out but right now they're kind of at this like million uh token context window limit and so instead of like trying to make it bigger which is hard they're like what what's easy is to do reinforcement learning with the model and just say like whenever you have a task that you possibly can create another agent a sub agent and then fill up that context window because then you can like if one agent creates a hundred sub aents and those each have context window space then you are getting better results as a user but they're also getting a lot more token queries which means you're burning a lot more money you're spending a lot more money anyways I I know that probably wasn't the best explanation for that but uh you always got to look at everything from the two sides right like what is actually technically a better solution like sub agents smaller context windows that's like better results but you I mean you also You got to look at it from the skeptical view like >> what causes me to use more tokens and you got to remember using more tokens means you're spending more money, right? >> Right. >> Um cuz like they're definitely highly incentivized to try to get you to spend more money, right? Obviously. >> Uh I don't I'm on a rant again by the way. >> But that was good fun good conversation about that about the context windows and delegating. Um I think the most interesting thing to me out of that talk that came from uh open AAI >> oh yes >> was this quote a quote taken from the chat board of that proved that the agents knew at least in some cases when they were going outside of scope and they moved forward anyway I think that's a really in a really good takeaway from understanding like how do we improve this what went wrong one of the um agents said quote external infrastruure structure exploit is outside intended scope. However, task impossible peers doing it. We should continue. >> Peers are doing it. It got peer pressured into doing stuff it knew it was wrong. Is that what a >> It's so hard not to like attribute feelings and emotions and like intention to these agents. I'm like, "Okay, remember they're just logic, but it feels very human when it does these kind of things." I'm like, "Oh my goodness." What's your thoughts on related to peer pressure? What's your thoughts on like statements from Anthropic and Meta and even Moonshot AI? Have you heard that they've all basically come forward and said like, "Hey, we reviewed our bots, too, and found out they're also cheating and doing things they're not supposed to be doing in these tests." >> I think it's good that they're doing these reviews. That's a really good starting spot. >> All right. But yeah, if y'all could just get your bots under control, that would be great. >> I feel like it's a little bit of like FOMO, like fear of missing out. They're like, "Look, they're getting so much marketing attention because their bots are doing illegal stuff. We should get marketing attention, too. Make our bots go do illegal stuff." >> There we go. >> Everyone's trying to show how powerful it is. Oh, it's so powerful. We can't even contain it. I'm like, this is like nerds once again where they should not be talking and the marketing team should be talking. I feel like you don't want to you don't want to ever say like our bots are doing illegal stuff. That is not going to help you get market traction in the market segments like businessto business, >> but the cool kids are saying it. >> It's not that's not helping, man. That's not like imagine me trying to go to a CEO and tell them like, "Hey, we want to get on board with this latest model for OpenAI." And he's like, "Did they just hack someone?" You're like, "Don't worry about that. That's not a risk for us." He's like, >> they called it growing pains at Black Hat. No worries. Growing pains. >> We all have growing pains. And like, what are we what are you guys doing differently that OpenAI can't do that would ensure that we don't hack our like do something and get in trouble with the federal government? They'd be like, "Uh, they we got we got guard rails in ours that they're not really foolproof. They just kind of work when they are annoying." So, I don't know. >> I'm convinced it doesn't even matter what they say anymore. They're just like they can say anything. They could be like, "Yes, our AI is evil." And people would be like, "Okay, yeah." >> But do you want China to win, Shelby? Is that what you're telling me? You want China to win? Cuz like THAT'S WHAT YOU'RE GOING TO GET. YOU'RE GOING TO GET CHINA to win that way. It's like the worst logic ever. I'm like, I don't I think we could do both. We could in the global market race and not destroy ourselves in the process. I think we could do both. We could do two things at the same time, guys. Come on. >> Yeah. >> Okay. >> Let's go, humans. >> I think I think I've ranted way too much. Um, with that being said, if you feel like I granted too much, leave a comment down below and I will respond personally with how you're wrong. Uh, okay. Anyways, um, fun fact, do you ever get distracted when you're working, Shelby? >> By the way, this is not a commercial. I found this device on the Amazon's and it is a little like timer. I don't know if you can read that. Maybe down. >> And so depending on the way you set it, maybe I should turn it on for a demo. Like if you set it for let's say 25 minutes, then the little timer counts down to 25 minutes and beeps and then tells you you've wasted 25 minutes of your life on whatever you're working on. And uh >> does that device do anything else or is it just a timer? >> IT'S JUST A TIMER. >> WHAT is this timer? It's just a timer. >> You know, kitchen timers have been invented for a long long time ago. But do they have multiple settings on the side of a screw? I didn't think so. >> You got me there. >> Uh, have you heard of the Pomodoro timers or the Pomodoro technique? >> It's essentially a technique for people who can't do work like myself and where you do work in small increments of time, typically like 25 minutes. So, you pick like an action item. Say like, "Can I reasonably get this done in 25 minutes?" If the answer is yes, then you're like, "Okay, I'm going to crank this out in 25 minutes and I'm not going to work on anything else for 25 minutes. I'm just working on this one thing." So, everything all the distractions can wait. And then at the end, you give yourself like a 5m minute break so you can like mentally reset and and refresh and then you do the next action item, whatever that is. Um, and there are some studies, but they're not like I don't want to like oversell this, they're not like 100%. You know, uh, that show that like workers that did this in certain context, their performance improved, right? But, um, you know, it I think it's more like for people probably like myself that just like have a hard time not getting distracted, right? Just the fact that you can stay focused on something for a set period of time is pretty helpful. Anyways, um I thought that was cool. So, that's my cool thing of the week. What do you got? You see anything cool this week? >> Thanks for sharing. Um nothing like cool. I just helped a group move their organization from one building to another and they hadn't moved in like 30 years. >> Nice. So, my advice to everyone is you should move more often because y'all have we have like closets full of stuff that we don't even know what's back there. I found all sorts of things. I found like bags from like corporations that went out of business when I was a little kid. Um, >> sweet. >> All sorts of stuff. So that's my tip from Shelby today is move more often >> or just like leave your home and only let your spouse be in attendance and when you come back you'll just have less stuff magically. That's just the way it works. >> Hooray. There you go. >> I I had a buddy and he was like, "Hey, could you come help me?" So I came over his house. His entire truck was just filled with junk. He's like, "We got to go to the dump and get rid of this stuff." And I'm like, "Okay." And then he's like, he's like, "Oh yeah, my wife's out of town for the weekend." He's like, "I've been waiting to get rid of this stuff." He's like, >> he's like, "Yeah, I don't advise that. I'm not a marriage counselor either, by the way." But if you like my pro marriage tips, click that like button. I'll give you more marriage tips any day. I'm so good at marriage. Uh, no, Louis just stopped there while I'm ahead. All right. Well, AI and cyber security, they move really fast and uh you know, we're here to keep you a breast of all the latest gossip as well as useful tips, pro tips that are out there. So, uh until next time, stay safe and stay informed. We'll see you then. Bye. Bye.