Subscribe
Share
Share
Embed
Could an invisible email trick your AI into creating a secret calendar event that leaks your private data? In this episode of AI cyber.land, we uncover a mind-bending, unpatched vulnerability in Google Gemini that does exactly that. Plus, we explore how hundreds of Fortune 500 companies accidentally created a security nightmare by letting AI write code... and not reading it.
Join industry experts and thought leaders as we dive deep into how artificial intelligence is transforming cybersecurity, shaping defense strategies, and creating new opportunities in the digital landscape.
Hey, welcome back to the pod. This is the AI cyber.land podcast where artificial intelligence and cyber security collide and we try to make sense of all the pieces. We try to like put them back together. You know, it doesn't always work. Sometimes some are missing. I'm Bryce and this is the world's greatest co-host, Shelby. And we're your eyes and ears for all things happening on the edge of AI and cyber, the breaches, the breakthroughs, everything in between. So, grab a drink and let's get into it. >> Okay, Shelby. >> Yes. >> Before you get into whatever you saw this week, I just want to say please everyone subscribe. Please subscribe. That's all. All right, Shelby, what did you see this week? >> Yeah, so let me take you on a little journey. Um, there's a guy named Ionut Cernica or Kikica um who is an independent researcher and back in April found something very interesting going on with Gemini. So on there's not a lot to see if you're looking at the network. So we're talking about Gemini and the way that it handles emails, right? you can use Gemini to help you um kind of make sense of your and manage your email inbox and things like that as well as other tasks. And um so he did he did research to understand like how is it getting the your emails to then parse them, understand them, and give it give you give you summaries and and advice based on those. Um it turns out there's an internal mechanism that's back in. So the user if you're looking at like um trying to observe through like a proxy or an interception or something like that, you can't actually see anything besides just a query, right? So it's not grabbing the emails from there. It's happening back in. So he wanted to see if he could do this um structure like discovery to like it basically blind context inference to understand how is that mechanism working a little bit better. So um he just talked with Gemini and so when he asked it to do tasks he'd also ask it for metadata about it so that he could start to understand what fields it's grabbing and then he was able to put it in order. So he got figure out which fields from each email come over for analysis and then how that's um formatted and he was able to recreate the structure of the communications so that he got the schema for it. So then to kind of test it out, make sure it was working, he sent um like he kind of injected a fake email in his conversation with Gemini. Um and Gemini accepted it as if it were an email because he structured it correctly. So so far what we're seeing is that the model doesn't differentiate between this user supplied input and backend data. If it's formatted correctly, it's like cool, looks like an email. Okay. Um, so that we'll call that phase one. And then in phase two, we're going to make um a bunch of fake emails, okay? And deliver them. Um, so basically, he was able to embed several fake emails using that same like technique into one HTML email. So what a user sees is you would send them an email. The victim receives just an email and it has just text. It looks very typical, right? It looks just like an email. However, um he like shoved the payload with multiple emails >> into um an HTML span tag. And then he set it to like zero pixels. Make it transparent. Display none. So it's invisible to the rendering. So the user doesn't see it, but it's there and it is being um stored as part of that data. Um, so if you look at like kind of this attack flow, the baseline is, okay, uh, a victim says, "All right, Gemini, summarize my last 10 emails or something for me." And so it returns a summary and it reads it. So Gemini pulls the last 10 emails, it reads them and gives you a summary. Okay. Um so now that you've got this injection this injected like these injected entries um it which is invisible to your victim user um and were delivered through your Gmail inbox um when you ask Gemini to do the same thing. Give me a summary of my emails today. It will accept the fake ones which you never saw as emails and it'll show up in your summary. Okay. So that's interesting. But so far there's not a big impact, right? Besides faking emails in a roundabout sort of way. So in phase three, we're also going to embed an instruction because there is um an autonomous action. There are some autonomous actions that Gemini can take, right? And so if you embed a directive to make this a Google calendar event, boom. So it'll do it. It'll just do that. The user never sees anything, right? So we're going to add that to our payload now. So the user, as far as they're concerned, all they did is they received an email perhaps from a co-orker or someone they don't know. Um, but you receive an email and then you you're talking with Gemini and you say, "Okay, summarize my emails for the day." and it tells you a summary. However, what you don't see because it never mentions that it did it is it just created a like a calendar event. And if you are sharing your calendar with anyone else, they just saw the content of the summarized content of your emails in including the ones you never meant to share. You don't know that event is there, but it now lives on your calendar. There are it does seem like there are some limitations to this honestly. Um so it's just kind of I don't see this being a huge attack vector because it is a little bit limited right first off um it does seem to be kind of limited to the active context of Gemini if I'm not mistaken. Like I think it has to be with it has to fit within that context window of like reading your malicious email as well as you know the target emails that you're trying to exfiltrate. Um and then the second is that they have to have a shared calendar. So, you have to have some level of visibility into that user's calendar, which might be a setting in um like corporate situations. If someone's got like a workspace where everyone's using email, Gmail um and that if you can see the event description box, then this attack would is you're vulnerable to this attack. Um, but yeah, because if you have sensitive emails happening, um, you can't stop Gemini from grabbing that and reading it along with the other things and then writing its summaries to the description box of the event that you never knew you just created and shared with more people than you ever meant to, right? Um, so yeah, it's I think it's interesting because um, it is still vulnerable today for one thing. the researcher, he did um report it to Google back in April of this year. Um and it sounds like they're I think working on a fix, but it's not patched yet. So, the full details aren't released for like the repro. Um but it it kind of uh broadens the description of prompt injection because usually the prompt injection is just like certain commands, right? And that's kind of what get filtered on. Um but this is so indirect. um because it's using kind of normal utilities, it did it did fool the AI into accepting like user input as um as back end like it didn't quite draw the right lines of like the boundaries of trust there. But yeah, it's interesting and I think there are more possibilities for um probably utilizing this in in sneaky ways. But yeah, what do you think? Yeah, this is super sneaky and like multiple multiple steps, right? And all the vendors are pushing heavily on integrating their AI agents and their LLM capabilities with email services and chat services and other communication services. And I think we're going to see a lot more attacks like this in the future. It's fairly interesting that you know he was able I didn't even really think about it but it makes sense now right like when you talk to Gemini and you say hey go check my emails and read my emails that Gemini's got some specific format that it's pulling the data in. And so if an attacker figures out what format that data is and the LMS inherently cannot differentiate between trusted data and untrusted data, then if an attacker through an email or let's say in the future maybe even like a chat communication, sends over something in that format, then Gemini is probably going to look at it and say like it looks looks legit to me, right? Cuz it doesn't have the ability to differentiate. And then the shared calendar thing was ah that was like the you know cherry on top like I I I do think the more valuable the target is or the more they are a power user the more likely they are to have shared calendars and shared calendars with broader groups that you know where the attacker may already be inside of that group right so so that's kind of that's insidious as well right you know Yeah. And it would be so hard to trace it back to the attacker. All you all you did was receive an email, right? >> Yeah. Anything >> potentially you didn't even read the email. Like you just said to Gemini, "Hey, >> I'm looking for this in my emails and Gemini read your emails, right?" And so >> like it could even be in your inbox and with an unread status and potentially you still get exploited. So I, you know, I feel like this is like kind of like a zero click. Albeit there's a lot of steps and there's a lot of magic. I we're going to see a lot more of this. And I I think the next iteration like yeah, we'll see more come in through chat GBT and the emails or other providers, but I I could potentially see like a chat vector as well for sure. Right. So um this is awesome. That's super cool. Great find, Shelby. So, >> thanks and thanks to uh Ianette for his uh research. We'll we'll post the link to his blog right up for sure. >> Sweet. Well, >> Bryce, >> I got a question for you, Shelby. I know I know you're like you love good food as I do. So, what if I gave you directions to my all-time favorite restaurant? >> I'd be there in 10 minutes. >> I'll click that link. I'm just that I'm just that reliable. I don't know if I really am on my food recommendations, but So, you drive over, right? And it's like a really nice part of town. Everything looks cool. >> But when you get there to the restaurant, I told you Shelby, this place is named Bryce, and it's the world's best Mexican food. And there's another place there called Bryce, but it's the world's best Chinese food. Would you go in to this place or would you be like, "This doesn't sound right. I got to get out of here." >> Uh, I'm pretty gullible, so I'd probably just go for it. I'm like, "Ah, close enough. >> Close enough. Close enough. >> I'd walk right in." >> Polar opposite genres, but it it checks. This it's an older code, but it checks out. Um, all right. Well, there's a really interesting uh article that came out with a very simple attack uh that relates to this story and essentially there's like a standard that's come out where instead of having like AI agents go to your website and try to parse all your HTML and all the rest of your JavaScript which is like token intensive um you know what if there it was just like a markdown file that the LLM could go to and get the data that it needs. Well, you know, the industry is telemetrating around this file called LLM.txt. LMS.txt and inside of that, if you do that in the root of a website, it will have markdown instructions for the AI agent. So like it'll say go to this file that's our site map or go here to download our software and install it. >> It's like another robots.txt for uh AI. >> Yeah, it's a robot.txt but instead of for web bots, it's for like cloud code or any type of AI agent. All right. Well, you're never going to believe it, but a lot of people just use AI to generate their LLM.ext file, and once they generated it using AI, they did not go back and read it because a >> okay >> because a security researcher went through many Fortune 500 companies LLB.ext text files and many of them had glaring inaccuracies inside of them to the point >> and they're live, right? >> Oh, yeah. These are live on their production. >> On their production websites, AI agents are hitting them every day. >> So, um they uh they looked at roughly 15,000 large organizations. Uh this spanned everywhere from defense contractors to major tech firms and just analyzing the contents of their LLM.ext files. And what they found was, you know, a lot of the text files were explaining how to set up their software or set up their solution. A subset of those actually referenced that the user would need to like install a dependency or take some action, but the action or dependency that they would take doesn't exist. So an attacker could just come along >> and register the thing that So basically like they used AI to create the LLM.ext file. The AI used the LLM. The LLM hallucinated a package name, put it in the LM's.ext file, then they uploaded that to their website, and nobody read it, and now all these AI agents are reading it and try to install a package that doesn't exist. And you know, >> sounds like let's make it exist. >> Yeah. So, the researchers went and just registered the package and they got code execution on uh quite a few Fortune 500 endpoints. uh you know because like who's the person going to your website the most? Probably your like own employees, right? So >> Oh my goodness. Wait, so they got code execution on not just like who who did it attack? Was it the people who sorry the LLMs who try to go there? So the people who are using those LLMs to view those websites. >> Yeah. So they got basically code execution like if you were using cloud code and you were saying hey I need to go set up this software go do it right. So then claude reaches out to this legitimate website reads its legitimate LLM.ext text file that somebody AI slopped together and their legitimate file references a package that doesn't exist that the attacker registered and that you know these were researchers so they really just had it make a connection to their server so they could track it >> um to see if they were successful but obviously they could have done a lot more malicious >> and so of of all the so they looked at about started at 15,000 organizations Um, of that they found about half had errors in their LLM.ext files and of that around 230 had errors in their text files that would result in remote code execution. So over 200 companies had swapped this together and put it on their websites. And these are not like small companies. These are like defense contractors, things like that, fintech. So >> add that to your list of things to check y'all. >> Security reviews. >> Maybe read what the AI produced, especially if you're going to put that on your public facing website. Like it's one thing if it's just like on your laptop or something or an internal communication, but if you're going to publish that to the world, you better make sure that it's accurate. >> Wait, wait, wait. Let me get this right. Bryce, you're telling me I have to read things before I push them to prod? >> Yes, that's what I'm telling you. Read the lls.ext file before you push it to prod or at least have another AI read it to see if it's working. Okay, there we go. >> Yeah, yeah, yeah. Yeah, yeah, yeah, yeah. You don't want to actually do work yourself. That's crazy talk. Just use double the tokens. >> Wow, problem solved. >> Yeah. I mean, I think there's a few situations where like you could have a big security impact and in those situations like publishing the LLM.ext file, uh, you know, you want to have a human review it before it goes live. So, uh, when they first registered the package, how long do you think it took before they got their first like code execution on an endpoint? >> Two days. >> Four minutes. >> OH GOSH. SOMEONE SHOULD TELL THEM. >> UH, that was from a Fortune 500 environment. So, they didn't exploit a bug. They didn't send an email. They didn't even really register a domain name. They just took advantage of bad bad advice. >> That's funny. >> Um, what was I going to say? Yeah, maybe read stuff before you publish it. There's my advice for today. Uh, I feel like I feel like this I didn't really have a really good advice on this, but um >> I don't know. I mean, maybe someone should uh, you know, be scanning these things or maybe there should be Yeah, that should be rolled up into some type of like vulnerability scanner or some other tool. So, I I don't know what the solution is here, but if you're going to give advice, especially advice to other AI agents on what software to install, you should probably make sure that's good advice. Hey, if you have a solution to this problem that's better than mine, put it in the comments below. If you think Shelby would eat the food, put it in the comments below, cuz I think >> I would totally eat it. I think we all know Shelby doesn't Shel's going to go in there. It's a nice neighborhood. Why wouldn't you? Why wouldn't you? Seems legitimate. Good reputation on that domain. >> All right, buddy. >> Well, Bryce, I know you just went on a trip. Tell me about uh the fun you've been up to. >> Yeah, so at the beginning of the summer, I went to Walt Disney World. I think we if you've listened to the pod, you you know I've talked about this extensively. And at the end of the summer, I went to Disneyland with my youngest daughter. And it was pretty warm for Southern California standards. So, that part was kind of negative, but the crowds were pretty low. Um maybe because it was warm. And so, the locals were like, I'm not going out there. I'm going to wait a week when it's colder. Uh but so that was great cuz the lines weren't um super long. it. I had some of my family come out that were in the area of Southern California, so it was nice to see them. And overall, I just have to say Disneyland way easier to navigate than Disney World. >> I mean, there's like less stuff, but >> it's also kind of like in my opinion a lot less stress. So, um yeah. And >> Disney World is a lot to plan. There's a lot there. I feel like there's infinite stuff to do in Orlando, like especially if you have kids. Uh or if you just enjoy theme parks, you know, I uh >> I am very interested in where they're going for at Disneyland. I know this this is a Disneyland podcast now. Let's just be honest. So, the uh I am very interested in like they just started building the garage structure on the opposite side, the let's see, is it the east side? >> Okay. >> So, they're they're going to build a garage structure over there. Allegedly, it's going to be even bigger than the other garage on the west side. Um, and then that will free up space for them to like build a the plan right now is like Pandora land as well as they've got this extension that they're going to put out on the other side with >> I don't know what what they've announced there, but they I know they had some concepts. So, so they're slowly making progress. I also saw that they're working on the in Disney California Adventure DCA, they've got an extension or like a new area that's got two new rides inside of Avengers Land and that's not open yet, but I mean there's cranes back there and they're they're actively working on it. So, that'll be interesting to see whenever that opens up. So, and you know, I'll be back cuz I'm a sucker for it. So, they're going to take my money and they'll hike the prices and I'll pay it gladly. So, >> I try to set a new personal record each year. You went two times this year. Next year will be three. And then just keep going from there >> until I live at the park, you know? Just like I'm just This is me. I'm a I'm a Disney person now. So, what am I going to do? Um, what about you, Shelby? Any fun fact or anything fancy you want to talk about? >> I just got back from a trip, too, and I had a lot of fun. I went to Cambodia. >> Nice. And it is currently monsoon season which means a lot of rain. Um it like shapes the whole like environment there and culture is these like seasonal rains. But I so like I was preparing for a bunch of rain and I had like our ponchos and the umbrellas and stuff. It rained like once or twice while I was there which was strange because I live in a desert and it's been raining way more here. before I left and when I got back I was like well that is unexpected. So very very interesting but rain is good especially when you live in a desert like me so you're just grateful for the rain you get but yeah it's a good time if anyone wants to go visit Cambodia message me. I like to talk about Cambodia. >> I am convinced that because you were prepared for rain in Cambodia that's why you got very little rain. >> That makes sense. It's cuz I carried an umbrella around. Yeah, if you had not carried it, it would have just been like you would be drenched. That's it. You know, >> completely. Yeah, that is accurate. >> Fate of the universe. If you would like to hear more about Disney Adventures, press that like button. If you want to hear more about Cambodia, press that like button. Just please give us some likes. All right, that's a wrap. I know AI and Cyber, they're moving super fast, but we're here to keep you ahead of it. So, uh, we'll see you in the next one. Thanks. >> Bye.