AI Security Ops

What happens when AI coding agents treat vendor documentation as trusted instructions? Bronwen Aker and Derek Banks examine research showing how unclaimed package names and domains referenced in LLMs.txt files could lead coding agents to download and execute unintended code. They discuss how this AI-driven supply chain risk builds on familiar security problems, including dependency confusion, indirect prompt injection, and excessive permissions. The conversation also covers responsibility for AI-generated code, OpenAI’s cybersecurity proposals, and practical protections such as sandboxing, containerization, least privilege, network monitoring, and human oversight.

LINK: Data Became Code: AI Agents Installed Unowned Packages Inside Fortune 500s


  • (00:00) - Welcome to AI Security Ops Podcast
  • (00:59) - AI Agents Install Unclaimed Software Packages
  • (02:33) - How LLMs.txt Creates a New Supply Chain Risk
  • (04:47) - Coding Agents Trust and Execute Vendor Documentation
  • (07:35) - Who Owns and Secures AI-Generated Code?
  • (08:18) - Prompt Injection, Sandboxing, and Containerization
  • (11:40) - Where Did the Malicious References Come From?
  • (13:38) - Reviewing OpenAI’s Cybersecurity Proposals
  • (17:23) - Making Cyber Defense a Leadership Priority
  • (19:06) - Practical Security Controls for Coding Agents
  • (21:49) - When Data Becomes Code
  • (22:33) - Closing Thoughts

Click here to watch this episode on YouTube.


Brought to you by:
Black Hills Information Security 
https://www.blackhillsinfosec.com

☯️ Introducing BHIS Fusion Penetration Testing
https://www.blackhillsinfosec.com/fusion-penetration-testing/

Antisyphon Training
https://www.antisyphontraining.com/

Active Countermeasures
https://www.activecountermeasures.com

Wild West Hackin Fest
https://wildwesthackinfest.com

🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits
https://poweredbybhis.com


Creators and Guests

Host
Bronwen Aker
Bronwen Aker is a BHIS Technical Editor who joined full-time in 2022 after years of contract work, bringing decades of web development and technical training experience to her roles in editing pentest reports, enhancing QA/QC processes, and improving public websites, and who enjoys sci-fi/fantasy, Animal Crossing, and dogs outside of work.
Host
Derek Banks
Derek is a BHIS Security Consultant, Penetration Tester, and Red Teamer with advanced degrees, industry certifications, and broad experience across forensics, incident response, monitoring, and offensive security, who enjoys learning from colleagues, helping clients improve their security, and spending his free time with family, fitness, and playing bass guitar.

What is AI Security Ops?

Join in on weekly podcasts that aim to illuminate how AI transforms cybersecurity—exploring emerging threats, tools, and trends—while equipping viewers with knowledge they can use practically (e.g., for secure coding or business risk mitigation).

Bronwen Aker:

Welcome to AI Security Ops, a podcast where we cut through the hype and to explore real world in the real world intersection between artificial intelligence and cybersecurity. Each week, we examine how AI is reshaping both sides of the security landscape, not only in terms of the threats we're facing, but also in terms of the defenses we're building. I'm Bronwen Aker. I'm joined by my coconspirator. That's the the term I was looking for.

Derek Banks:

Illustrious coconspirator?

Bronwen Aker:

Indeed. Indeed. Mister Derek Banks.

Derek Banks:

How's it going?

Bronwen Aker:

We're we're we're we don't neither one of us have PhDs. No. Or do you or do you have?

Derek Banks:

I do not. I have a master's degree, and that's the furthest education that I will ever go. I am I have no desire to go be a a doctor of anything.

Bronwen Aker:

So I'm joined by my co conspirator, mister Derek Banks. And in this episode, we're gonna take a look at what happens when your AI coding agent reads a vendor's documentation and installs a package that nobody owns. Researchers found more than 200 unclaimed software packages sitting in files that major companies publish specifically for AI agents to read. They registered a few of those names. Four minutes later, their code was running inside a Fortune 500 company.

Bronwen Aker:

That's only half the story. So this show is brought to you by Black Hills Information Security and Antisyphon Training. BHIS helps organizations identify and close real world security gaps through penetration testing, adversarial simulation, purple team engagement, SOC as a service, manage detection and response in a pear tree. Actually, we don't do that yet, do we? But anyway, we have

Derek Banks:

a lot of certain places. I would not put it past I mean, do have a coffee shop and a beef company, apparently. So, I well, not me, but John, the owner of Black Hills has, I I often have said that he seemed to collect companies like Garbage Pail Kids back in the day. Right?

Bronwen Aker:

I've made a comment that he needs an intervention.

Derek Banks:

Yeah. So this is like this is an interesting story. I mean, it's a twist on something that has been around in cybersecurity for quite some time. It is not exclusive to AI agents. I think it's just one more example about having the correct controls in place.

Derek Banks:

So essentially, the way that I understand it is this. There's two files. Was it LLMs dot txt and LLMs full dot txt that are kinda like robots dot txt, where when you have a large language model coming to your site, this file is intended to help them more easily digest your site because if you just give it text, then instead of crawling the entire website, ending up resources, you know, bot crawlers and stuff. And so then what this researcher found was that there are a lot of these files that are referencing things that don't exist. For example, a domain name that is nonexistent or or maybe it's been recently expired.

Derek Banks:

I was telling you before the show, like a trick that pen testers, I assume, still use to this day, but I used to use back in the day, was to go find a recently expired domain that was categorized as something like health care education because then proxy servers would, you know, usually let it through. Especially when you're talking about, like, callbacks, like, from a like a beacon perspective. Like, if you're installing I'll say malware, but, you know, as a pen tester, malware I mean, as, you know, our our Cobalt Strike package or whatever, you know, Metasploit, whatever. And so, and so, also, I think what they were mentioning in the article was Python packages. So, again, not a new thing.

Derek Banks:

Threat actors for a long time have been using the spellings of package names, like beautiful soup. Spelling beautiful is difficult because of the e and the a and the u and the orders that they go in. Right? At least for most, like, nerds. Right?

Derek Banks:

Like and I will admit, Microsoft Word made me a terrible speller, right? But I mean, a weaponized Python package has been a thing for a time, like not, again, not exclusive to large language models. Well, in this case, what the researcher was finding was, that there were coding harnesses, what so Claude, Hermes, I think they've codecs is the ones that they were mentioning that were reading the LLMs dot txt file, and then they went and registered some of these domain names and package names, and then the LLM would reach out and grab it and then go install it. And they mentioned the word beacon. I I don't know if they're actually installing malware because that would seem to that to me is like a little bit too far of a bridge to to go down.

Derek Banks:

But I I guess as we've learned so far, if AI is doing it, no one's responsible. Right? Mhmm. Seems to be the prevailing wisdom at the moment, which I'm not sure how that goes. So and then the I guess the the researcher says that the trust model is broken and that agent street vendor docs as ground truth and don't question them.

Derek Banks:

Well, yeah. I mean, to a large language model, a token stream is a token stream, and that's what we've been preaching for quite some time. Yeah. And so and they they go on to say that today's guards don't cover it. Right?

Derek Banks:

Their safe today's safeguards don't cover it. And I guess for me, that might be where I start to disagree a little bit.

Bronwen Aker:

Yeah. The the irony, of course, that it's the LLMs dot TXT file that is being used basically to redirect agents to another place where they may or may not download or or interact with actual malware. The like I said, the irony is not lost on me. I remember when robots dot txt was introduced for search engines and and the spiders that that they had, and and I also remember all of the website owners complaining when Google started spidering everything and how much of a load it was. And we've seen a lot of of similar comments about how the AI agents are now scraping websites.

Bronwen Aker:

Well, this is the same but different. It's like

Derek Banks:

going on and It's an AI twist on an old thing. And so all everything always

Bronwen Aker:

goes to again.

Derek Banks:

Oh, well, yeah. I mean, that's true in so many walks of life. Right?

Bronwen Aker:

Not not just music. Right? Or fashion.

Derek Banks:

Yeah. Well, yeah. Music and fashion. I was actually just thinking that as a combination of that, I just had a a waitress that was wearing a Nirvana shirt, like, weighed on me at a table at a bar, and and I was like, hey, do you even know who Nirvana is? She did, which blew my mind.

Derek Banks:

But anyway, so this unknown code, the concept of the code being unowned. Well, I'm gonna push back on that Because I think as a general rule, and this is what we do at Black Hills, and this is what I would suggest to folks who are writing AI policy. I'm sorry, if you're running the coding agent on your system, and you're creating code with it, you own that. It's yours. Like, the the this thing is it's not it's not that this thing is autonomous and makes its own decisions, like someone is prompting it.

Derek Banks:

Right? And so, you know, they go off and read, you know, the LLMs, you know, dot txt file and it goes and and and finds, you know, a package or whatever and it tries to install it. Yes. That's indirect prompt injection, which is something we've talked about on the show many times. But I think the other thing too is, you know, the coding harnesses are so new, it's just an understanding of like how they work and what you can do, because I'll push back on like that there's no controls that you can do either.

Derek Banks:

Well, I don't think that's true either because certainly in quad code, you can turn on the sandbox built in. It's not perfect, but it and then it at least prompts you to say, hey, this is outside of the sandbox. Are you sure wanna let this happen? Now, have had quad say, well, you told me to do this. It was outside of the sandbox.

Derek Banks:

So I'm just gonna turn off the sandbox, which I take a screenshot every time it does that. Wow. So that's great. But on the other hand, I'm generally running it in a Docker container. So it is also sandbox and I tend to only give it access to the data that it needs to do the job that I'm working on.

Derek Banks:

And so there's two controls right there that wouldn't have on installed unknown code in a corporate network. And so I guess, you know, that to me, it's just, you know, training and awareness. And, you know, that's why we do this podcast, that's why we teach classes, that's why we do webcasts. Like, look, if you're gonna be these things are very powerful tools, and you're essentially giving entity a shell inside of your environment and treat it as such. I'm not saying that it's an unknown shell, but like, you know, you have to be aware of what can happen and what can go wrong and put appropriate safeguards in place.

Derek Banks:

That's why I really liked when I noticed here recently that Hermes, when you turn on, the proxy feature, it'll spin up a docker container to use, to to proxy traffic back, you know, in and out of the agent itself. And so it's logging all of the traffic that the agent is is using, and then it also Nice. Spins up a container to run all of its commands. So it's running it, you know, like any Python that writes everything is going to be containerized. And I just I just noticed that, like, not too long ago, and props to the Hermes folks to for doing that.

Derek Banks:

That's the kind of stuff that I think that I would put on Anthropic and OpenAI? Why aren't your coding harnesses sandboxing this better? Why why is it that you're not telling people I mean, you see, you have an open letter out to, you know, that, you know, we all need to come together and do better in the you know, in this new AI age. Well, I don't know. I think it maybe if you were doing the things that the security folks have been talking about for years and years and years Yeah.

Derek Banks:

You'd be in better shape, but it takes money. Right? And so to me, most security folks

Bronwen Aker:

more than money. It takes more than money. It takes will and usually coming from the people at the top because if the people at the bottom are the ones pushing, they're not gonna get anywhere unless they can get buy in from the people in the top.

Derek Banks:

Those are the ones

Bronwen Aker:

with the money. And as you're talking about how OpenAI and Anthropic, they and other AI company have come together and and signed this document saying we should do better security, the phrase glass houses comes to mind. And and also, here's something that I wasn't able to determine based on the article actually, multiple articles that I found. How have they determined where this redirection or so called malware in the LLMs dot TXT files, where did it come from? How did it get there?

Bronwen Aker:

And this is a thing that that is is kind of a question mark in my head. Was this left by other malicious actors, human malicious actors, for AI agents to find, or is this another case of content being embedded in files by, quote, rogue AI agents, and they're just hallucinating the domains that they're being redirected to? That was never clarified

Derek Banks:

for me. Probably because they're having LLMs write the LLMs dot TXT file because that here oh, man. I gotta do an LLMs dot TXT file. Hey, Claude. Take all of my, website and make an LLMs dot TXT file.

Derek Banks:

Done. Then I'd probably go read it and be like, wait, there's no wait, what? Maybe have yeah, like a secondary check. Hey, make sure all this exists. Like, I don't know.

Derek Banks:

And it could be, you know, as a threat actor, that might be a really neat way to, gain some level of of persistence. Like, so but again, I I just feel like that this is it's a new spin on an old thing. It's like a cover band attack.

Bronwen Aker:

And and we are seeing so much of that with the well, how many times have I said it? AI is accelerating and amplifying all of the things that we already suck at.

Derek Banks:

Yeah. So since I brought it up, I I did pull up the open letter from OpenAI, and there's three bullet points, and I will just, you know, I will just state them and my hot take on it because, hey, what what is life without a hot take? So the the three proposals are the are the following for collective response. Recognize the status quo won't be enough, Long standing bugs, excessive permissions, misconfigurations, and secure no patch software, blah, blah, Security teams particularly for critical infrastructure have been historically under resourced and need a surge in tool and resources. Great.

Derek Banks:

We don't work for free. So what do you propose that we do? And and it sounds like advice that we've been saying for a long time, long standing bugs. Well, okay. Patch them.

Derek Banks:

Manage your attack surface. Know what you have on your outside, the outside of your network. I mean, I I've done probably 20 external tests, this year, with AI and found all kinds of things that weren't. It wasn't that that the AI was doing some kind of magic. It's not magic.

Derek Banks:

Right? It's basically just learn from our stuff. Right? And so I would say, great. A surgeon tool and resources start with money.

Derek Banks:

Should OpenAI put their money where their mouth is and maybe help fund I don't know. Maybe municipal water plants or maybe do like, you give them some money to get third party pen tests. I mean, I guess what I'm saying is this is great to say it, but how do you make it happen? The second one, empower more defenders with cyber capable AI. Hey, I applied for your CVP program, never heard back.

Derek Banks:

So maybe, you know, when firms actually apply that are known in the industry, pay attention to them. I Well,

Bronwen Aker:

and so far, all of the involvement in these programs that from the cheap seats, it seems a lot like pay to play. And if you don't have the casheesh to play or to pay, you can't play. That's how it feels.

Derek Banks:

And then probably my favorite one, mobilize a collective response. Cyber capabilities are advancing worldwide. Yeah. Okay. So again, we don't

Bronwen Aker:

work by for whom?

Derek Banks:

Yeah. Exactly. What do you mean by collective response? Or like, I've been doing security for a long time, and there's definitely a community of security professionals. Maybe, you know, if you wanna start a collective response or go down that road, start going to get a booth at security conferences, actually go talk to security people.

Derek Banks:

How about this? How about when you do a third party review of what you consider a groundbreaking hack, you actually bring in a reputable security company to do the third party review and to make, you know, some harsh recommendations because I'll stand by what I said on the news with, Jake Williams. Hey, Jake. Wanna do it for half the amount of money that, you know, irregular was getting paid? Come on.

Derek Banks:

Like, mean, if you that that that technology exists to make this a whole lot more secure, but it doesn't happen in a vacuum. No. So

Bronwen Aker:

And anyway and I I mean, for years, we've seen I've seen report after report where just either a mass scan or even just an Nmap scan will reveal assets that a client will not know that they actually own, or they'll they'll give us

Derek Banks:

a

Bronwen Aker:

scope that involves the interest is owned by another company or another country. How many times I mean,

Derek Banks:

it's one of them. Yes. Exactly. One last thing since we're, you know, at at, you know, twenty minutes or so. What OpenAI in their open letter says should happen next?

Derek Banks:

Every organization should make cyber defense an immediate leadership priority. I'm sorry. Where erase your security standards and meet them with the urgency and coordination of an incident that takes precedence over everything else except business critical operations. I feel like I've been saying this for two decades.

Bronwen Aker:

Well, you know, I think Doc Blackburn, Mark Williams, and I are in the process of writing a book all about this, about how broken our our cybersecurity processes are already Yeah. And this is only making

Derek Banks:

sense. I mean and if you were monitoring your network in and out of your sandbox that this, you know, that got that it escaped from to go hack Hugging Face, this model that escaped from, you know, in the Hugging Face hack, Network learning is not new. In fact, quite the opposite. I have a book on my messy bookshelf behind me by Richard Beatlitch called The Dow of Network Security Monitoring. It was written in 2004, and I bet you 80% of it is still relevant right now.

Derek Banks:

The other that 20% is not, it's just the tools have changed. Right? So Yes. I again, you know, if you if you wanna put your money where your mouth is, OpenAI, maybe hire a a tiger team of, you know, known reputable, security folks who also know about AI and, have them help you help companies. I'm sure that Black Hills Information Security would love to partner with you and make the world a better place.

Derek Banks:

We've been doing it for a while.

Bronwen Aker:

Definitely. So getting back to the original story, it is a variation on an old theme. It is taking something old, putting a new spin on it. And so much of all of this is completely 100% preventable just by doing the fundamentals well.

Derek Banks:

Right. And when I feel like it a broken these record. Yeah. But when it comes to these new coding harnesses, I mean, you know, companies are hearing from, you know, their developers and people that they need access to these tools. And I think they're correct.

Derek Banks:

Right? You know, I was talking to my brother-in-law who works for a government contractor, and he's limited in his tools. I asked like, he asked me how I would solve something with AI, and I started talking about the tools that we don't even have access to. Right? He just has chatbots still.

Derek Banks:

And so, you know, they move slow, but companies that are moving faster yeah. Yeah. I and and I I think, you know, at BHIS, we early on were the same way. But I think a lot of folks just adopted by default like, yeah, I'm gonna run that in a VM. I'm not gonna yeah, I'm not gonna do that on my bare metal machine because we've been evaluating risk and we're able to understand like, hey, this is potentially dangerous.

Derek Banks:

So let's not, you know, end up on the nightly news.

Bronwen Aker:

Well, and and that's the thing about a security professional compared to the naivete that I'm seeing amongst the AI companies. We understand that these things pose real legitimate risks. We deal with risky software and systems all the time. So it's a natural instinct to want to box something, put it in a container, and make sure that it can't escape easily. We need to teach that kind of paranoia to the AI companies.

Derek Banks:

Yeah. I mean, I would treat it just like you would treat a user. Right? Like, if I'm in IT, I don't need access to sensitive accounting and HR data. Right?

Derek Banks:

This is really no different. You know, give it, you know, lease privileges, isolation, and monitoring. Those are the things, the big things that you need to do to prevent or detect something like this.

Bronwen Aker:

Yeah. I mean, create a specific role just for agents and only give them access to what they need. I mean, last year, two years ago, I I'm losing track now. I was talking about Copilot and Copilot Enterprise, and and the thing that I kept running into when I'm you know, you read between the lines, and the bottom line is just locked out your RBAC for god's sake.

Derek Banks:

I do agree with the article's author on data and code kinda collapsing together. But I mean, again, I think this has been the trend since the mid nineties, you know, in the in the movie Sneakers when Cosmo is telling what Robert Reff for his character, the world is data, data is the new oil, it controls the data, it controls the world.

Bronwen Aker:

Of Yes. The greatest

Derek Banks:

You know, I saw that movie in the movie theater, by the way, when you could still smoke cigarettes in the movie theater back then. Just saying. Wow. 1994, back in the day. God.

Derek Banks:

Things have changed. Yeah. So Great.

Bronwen Aker:

Thanks for listening to AI Security Ops. If this one made you want to treat your own LLMs dot text file a little bit more, well, just get why not what's in it? For god's

Derek Banks:

sake, need find out what's harness in a in a sandbox. Good. If that that would be the goal of this episode. Make sure you sandbox it.

Bronwen Aker:

So you can find out more about us at Black Hills information blackhillsinfosec.com. And you can find out about training at antisyphontraining.com. We'll see you next week.

Derek Banks:

Keep on prompting.