AI Security Ops

In this episode of BHIS Presents: AI Security Ops, the team tackles a deceptively simple question with some very complicated answers:

Can you actually ban an AI model?

Not access to an API. Not the chips used to train it. The model weights themselves — files that can be downloaded, copied, modified, quantized, fine-tuned, and redistributed around the world.

As governments consider restrictions on Chinese open-weight models, the security argument cuts in both directions. There are legitimate concerns around national security, guardrails, model capabilities, and foreign technology dependence. But those same open models are inexpensive, locally deployable, and can give defenders capabilities that commercial frontier models sometimes restrict.

So what would a ban actually accomplish — and could it even be enforced?

We dig into:
- Where U.S. restrictions on open-weight models currently stand
- Why banning downloadable model weights is fundamentally different from restricting an API
- How procurement rules and hosting restrictions could create a “soft ban”
- Why the economics of open-weight models are driving adoption
- How restrictions could disproportionately impact startups and smaller organizations
- Whether modifying, quantizing, or fine-tuning weights makes model-specific bans impractical
- The national-security argument for restricting Chinese models
- Why guardrails on hosted frontier models matter to the security debate
- How Hugging Face turned to a locally hosted open-weight model during incident response
- Whether banning open weights could put defenders at a disadvantage
- How existing government actions can indirectly limit access without banning a model outright
- The hardware and operational costs of self-hosting large models
- China, AI infrastructure, market competition, and industrial-scale distillation
- Anthropic’s argument for mandatory safety testing of sufficiently capable models
- Why safety testing gets complicated when open-weight guardrails can simply be removed
- What realistic AI policy might look like when the technology cannot easily be recalled

This episode explores a central tension in AI security: the properties that make open-weight models difficult to control are also the properties that make them useful.

You can run them locally. You control the data. A provider cannot revoke your access. You can modify the model for your own use case.

But once the weights are released, those capabilities are also difficult to take back.

For defenders, the bigger question may not be whether open-weight models should exist. It may be whether restricting access leaves security teams with fewer tools while attackers and foreign competitors continue developing the same capabilities elsewhere.

https://www.anthropic.com/news/position-open-weights-models


—

Learn more about Black Hills Information Security:
https://www.blackhillsinfosec.com/

Check out Antisyphon Training:
https://www.antisyphontraining.com/

#AISecurity #CyberSecurity #OpenWeightAI #ArtificialIntelligence #LLMSecurity #AIRegulation #DeepSeek #InfoSec #BHIS #Antisyphon

----------------------------------------------------------------------------------------------
🎧 Subscribe to the Podcast:
https://aisecurityops.transistor.fm

About Brian Fehrman - https://www.blackhillsinfosec.com/team/brian-fehrman/
About Bronwen Aker - https://www.blackhillsinfosec.com/team/bronwen-aker/
About Derek Banks - https://www.blackhillsinfosec.com/team/derek-banks/
About Ethan Robish - https://www.blackhillsinfosec.com/team/ethan-robish/
About Ben Bowman - https://www.blackhillsinfosec.com/team/ben-bowman/

  • (00:00) - Intro: Can You Actually Ban an AI Model?
  • (01:26) - Where U.S. Open-Weight Restrictions Stand Today
  • (05:20) - Why Cost Makes Open-Weight Models Hard to Replace
  • (06:32) - What Would an Open-Weight Model Ban Actually Look Like?
  • (13:06) - National Security, Guardrails, and the Case for Restrictions
  • (15:02) - Hugging Face and Why Defenders Need Open Models
  • (20:02) - Soft Bans, Model Access, and the Cost of Self-Hosting
  • (23:14) - China, AI Competition, and Model Distillation
  • (27:09) - Anthropic’s Proposal for Open-Weight Model Safety
  • (31:19) - Final Takeaways: Competing in an Open-Weight World

Click here to watch this episode on YouTube.


Brought to you by:
Black Hills Information Security 
https://www.blackhillsinfosec.com

☯️ Introducing BHIS Fusion Penetration Testing
https://www.blackhillsinfosec.com/fusion-penetration-testing/

Antisyphon Training
https://www.antisyphontraining.com/

Active Countermeasures
https://www.activecountermeasures.com

Wild West Hackin Fest
https://wildwesthackinfest.com

🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits
https://poweredbybhis.com


Creators and Guests

Host
Brian Fehrman
Brian Fehrman is a long-time BHIS Security Researcher and Consultant with extensive academic credentials and industry certifications who specializes in AI, hardware hacking, and red teaming, and outside of work is an avid Brazilian Jiu-Jitsu practitioner, big-game hunter, and home-improvement enthusiast.
Host
Bronwen Aker
Bronwen Aker is a BHIS Technical Editor who joined full-time in 2022 after years of contract work, bringing decades of web development and technical training experience to her roles in editing pentest reports, enhancing QA/QC processes, and improving public websites, and who enjoys sci-fi/fantasy, Animal Crossing, and dogs outside of work.
Host
Derek Banks
Derek is a BHIS Security Consultant, Penetration Tester, and Red Teamer with advanced degrees, industry certifications, and broad experience across forensics, incident response, monitoring, and offensive security, who enjoys learning from colleagues, helping clients improve their security, and spending his free time with family, fitness, and playing bass guitar.

What is AI Security Ops?

Join in on weekly podcasts that aim to illuminate how AI transforms cybersecurity—exploring emerging threats, tools, and trends—while equipping viewers with knowledge they can use practically (e.g., for secure coding or business risk mitigation).

Brian Fehrman:

Hey, everyone, and welcome to this episodes of AI Security Ops. And in this episode, we're gonna talk about can you ban models? That's the actual question that some of the government is wrestling with right now, not talking about chips, but the actual models themselves. And it's a bit of a, interesting question both from just kind of a philosophical standpoint, I would say, and also a practical standpoint in terms of enforcement and implementation. But before we dive into that for today, let's talk about our sponsors.

Brian Fehrman:

Black Hills Information Security. If you or organization are in need of any kind of security testing, whether that's external, internal, AI testing, AI assisted testing, physical testing, wireless, SOC operations, basically anything security related that you could possibly think of. Sure. We could help you out. Blackhillsinfosec.com.

Brian Fehrman:

Additionally, we have our training branch, Antisyphon Training, where many of our consultants take their daily knowledge, and package it up into an affordable and easy to digest format to share out with the world, so you can help, level up in the current job you're at, get the position that you want, or maybe just learn something new that you didn't know before. So check them out at antisyphontraining.com. So let's hop in. So where are we at today with bands in terms of The US?

Derek Banks:

So right now, in terms of open weight models, I'm not aware of any kind of like hard or soft band. I don't think there is one today. However, there are talks that probably stem from a couple of of things. One being the export control of Fable five and Mythos, and I don't know. And if I say FUD, fear, uncertainty, and doubt that surrounded mythos, is that accurate?

Derek Banks:

Because, I mean, I think we've probably talked about that a bunch. But but either way, you know, the the the access to those were controlled by the government. But I mean, I guess in my opinion, Anthropic was basically asking for it. Right? But I think, you know, now that, you know, OpenAI had an accident where they had a quote rogue model that went off and hugged hacked Hugging Face.

Derek Banks:

And then Anthropic said, oops, we looked back through our logs and oh, man, we accidentally hacked some stuff too. I, you know, I don't know. Like, I think that the the government's going to start taking a lot closer look at at what's going on.

Bronwen Aker:

I wouldn't mind, honestly, if they had people in the government, at least in the administration, who were a little more practically minded. I I just it a lot of the decisions about any of the AI technology have felt very reactive and not really looking at the actual issues with a lot of understanding about the repercussions or or even whether or not these bands are gonna be effective. And I think that's gonna be a bigger question in the long run. Just because someone imposes the ban, this is software, and it's high demand software. So it's going to be a a very difficult challenge, I think, to keep a lid on anything, especially because there's so much pressure from within the technology companies themselves to go further, faster, farther.

Derek Banks:

Well, actually, so last week, I I I saw that there were like two open letters. I think last week, there were like three or four like open like AI letters to the government, the world, the community, whoever. One was, if I'm recalling correctly, an open letter from about 1,700 or so companies. And and Anthropic and and OpenAI were on there. Was a bunch of technology companies.

Derek Banks:

What they call it, little tech, I guess, you know, the technology startups and stuff. And so I think there are a couple open letters. The first one, I think Anthropic didn't sign on to, unless I'm mistaken, and that was, don't ban Chinese open weight models, you'll kill the startup industry. Right? Then there was another open letter and I don't know what the Venn diagram is between companies on each one of these open letters.

Derek Banks:

It was basically saying that they wanted the government to come in and slow down AI progress. And I'm sitting here thinking, why would you ask for that? Like, please come regulate our our industry because I don't do do you all know like what happens with like regulated industries like in general? Basically, their product goes up in price. That's typically how that works.

Derek Banks:

Right? And so, we get less AI for more, like, you know, for more money. That sounds terrible.

Brian Fehrman:

Yeah. And I think that's a great segue kind of into the next talking point of some of the difficulties surrounding, the actual practicality of trying to quote unquote ban some of these open weight models because I think it's important to look at, well, why what's really driving some of the popularity of these open weight models and cost is a huge portion of that, I would argue. When you've got a model that can perform at 8090% of the, the efficiency output of these big frontier models, but at one tenth the price, that's a game changing for companies. That's the difference between we literally can't afford to use this frontier model. We'll go out of business to, hey, we can actually incorporate this new technology into our normal workflow and still be profitable.

Brian Fehrman:

And so, by taking some of these open weight models out of the equation, it's gonna essentially absolutely cripple, certain smaller companies who just can't afford access to the larger, the frontier models on the scale that they really need to on a day in and day out basis.

Ethan Robish:

Can we talk a little bit about what banning might look like? I don't know if there's any actual proposals out there, but Derek, you you were talking about something earlier and I've got some thoughts as well. But like, what does it even mean to ban an open weight model?

Derek Banks:

Backstage in the green room. Yeah. And we were chatting before the the show. So, yeah, I mean, I think it could take a couple of, like, forms, right? I suppose the US government could say, no US company can use a Chinese open weight model and you can only use these, that that's unworkable.

Derek Banks:

Like, I don't know how at this point, like, you could probably I mean, remember the days of m p threes, like, and all and like, did they ban m p threes? I I you go download them off of like, well, Mega Torrent or whatever it was at the time. So I guess, I I don't think banning anything works. I don't think banning software works. I don't think banning firearms or drugs work.

Derek Banks:

People are gonna do what they need to and want to do. Right? So I don't think I I I think that, you know, the current administration would be smart enough to realize that they can't just say you can't use it. What I think they could do, would be to create an environment where companies have no choice but to not use, Chinese open weight models or open weight models to force you into a provider like Anthrop or OpenAI as a quote trusted partner. They could have said they definitely could say all government employees, and this actually might be the case right now, can't use Chinese open weight models.

Derek Banks:

They could say government contractors can't use Chinese open weight models. They could create an environment where it would be deemed risky for corporate enterprises to use an an open weight or a Chinese open weight model by saying, here are the risks that are involved to your company and if something happens, it's on you. So I I think that's kinda like a soft ban. And there was actually I can't remember the gentleman's name. There's someone who was tweeting about this, I think, week before last, that was involved with the Trump administration, essentially, not really advocating for that, but saying that's, you know, a likely course that they could take, which, you know, I think is kind of a little wrong.

Derek Banks:

Like, I it just seems like not the right way to do it. And I I also feel the same way about, like, slowing down AI development. I mean, my opinion is the genie is out of the bottle and I I don't know how you put it back in at this point.

Bronwen Aker:

It's also one eighty of the drumbeat that has been pounding ever since OpenAI first released ChatGPT. I mean, it's been we need more. I can't think of a single application on any of my devices or or computer systems that hasn't had AI shoved into it to some degree. And now they wanna try and put the genie back into the bot.

Ethan Robish:

So for what Derek what you were saying about the soft band, I think one step further and maybe a kind of middle ground. Because I don't think anyone would argue that it would be very effective to ban the weights. Like, you release the weights. Like, do you ban people sharing the weights back and forth? Like, you could try, but people are still gonna do it.

Ethan Robish:

Right? So I think maybe what they could do is they so not only for government, but they could they could ban the hosting, like the the service providing of the models. Like, no, the models themselves, you can have them. Sure. Go ahead.

Ethan Robish:

You any company not allowed to provide, like, host it for anyone else. So if all of a sudden you don't have easy access to, like, that Rock, OpenRouter, Cloudflare, Vercel, like, you can't access the open models anywhere, I mean, you could host it yourself, but then that's restricted based on hardware. You're probably not going to get unless you're a a huge company, like the the biggest models, you could I mean, depending on how they phrase the sanctions and stuff that if the model is hosted by a non US company, like overseas, that could be an option. Like, you could still access it that way, but that would put a huge bottleneck that would effectively cripple like the access because one, they're not gonna have all of the hardware that The US would have. And if everyone is bottlenecks to going to the overseas ones, like, it's gonna kind of denial of service they're they're offering.

Brian Fehrman:

Yeah. I think it's, I think it's difficult because I mean, even if you so talk about banning like the hosting. So at what point can you I mean, how did they define the exact model? Right? I mean, if you've got something that is literally billions of parameters, how many of those how many of those internal weights could you slightly modify to the point where it's no longer considered that original model.

Brian Fehrman:

Right? Because a lot of the a lot of the weights within models aren't aren't even used depending on like what what you're activating. I mean, you've got these dense models and, a lot of times the stuff doesn't get activated, especially when you look at like mixture of expert models, the non dense models, where there are 30 something billion parameter, but they only activate 3,000,000,000 parameter at a time. But like, okay. So now you start going in and you just start manually modifying like the very last decimal point on some of these so that the the output of it is like virtually, it's unchanged, but the weights have now changed.

Brian Fehrman:

So it's not really the original model. And so then it gets into like the legally distinct like portion of where, you know, like, people, I don't know, make references to trademarks and change it just enough so that it's not it's no longer legally distinct as that entity.

Ethan Robish:

What you're describing it essentially the same as quantization. Right?

Brian Fehrman:

Yes. Yeah. Like, you're just Basic check.

Ethan Robish:

I mean, instead of changing it, like, you're chopping up, but you're either way, you're saying this fraction Yeah. Is

Derek Banks:

Any any kind of supervised fine tuning is going to change some of the weights. Right? Yeah. And so, like, I guess, like, if if you were to say you can't host these open weight models, like, how much innovation would you really stifle? I mean, a lot.

Brian Fehrman:

A lot.

Derek Banks:

What

Ethan Robish:

do you guys think is the motivation behind banning the models? Like, we talked about why they're useful. Safety. Yeah. What's what's the thing that they're actually gonna publish is the reason?

Ethan Robish:

Wait.

Derek Banks:

Do you want the cynical answer or what they're gonna say?

Bronwen Aker:

I think I already gave the cynical answer.

Brian Fehrman:

Yeah. So when we're talking I think we're talking specifically like Chinese open weight models, what they're going to say is national security concerns, and which it it.

Ethan Robish:

So so it's more the it's more the bigger model. So like, you could have so like, Quinn comes in, you know, whatever trillion parameter size. They could ban that version. But like, once it's down to, you know, the 30,000,000,000 model I mean, maybe they'll ban like any open weight model above a certain parameter account or something like that. Like, with that reasoning, they could probably justify like drawing that line.

Ethan Robish:

I don't know.

Brian Fehrman:

Yeah. At least at least to the non I would say, at least to the non tech savvy folks, because, you know, we've had an episode before of kind of what what are the real security risks. And basically, I mean, if it's not, like, if you're hosting it yourself and you've got your different, kind of guardrails, in place in terms of agentically what can and can't happen, the data can and can't access, I mean, like, there's not a lot of risk.

Ethan Robish:

Yeah. Well, risk to yourself, but risk risk to other people, like, how you could abuse that model to attack others, like, that I mean, that does have a point in the guardrails that these companies have up. Because like, Anthropic OpenAI have guardrails in front of their hosted model that we can't just disable. You can try to jailbreak and get around them, but that's I mean, sometimes that doesn't work either. I mean, I think that we've got a point on that here.

Ethan Robish:

And did you guys talk about this last week, the hugging face?

Derek Banks:

Yeah. Bronwen and I talked about it from the angle of, like, step by step, like, essentially, like, going through the MITRE framework of how here or the the kill chain of, like, what the model did, and how you could have detected or prevented it from doing the thing that it did to to get to be successful.

Ethan Robish:

Did you touch on this particular point where Hugging Face tried to analyze the breach? They didn't know at this point that OpenAI was the one responsible. They just knew like, they were being attacked and it seemed to be an automated fashion. So they were trying to analyze the incident and feed in logs and exploits and stuff to to on the defending side. And the frontier models were like,

Brian Fehrman:

nope. We we can't do

Ethan Robish:

that for you because that's too risky. So And so they talked

Brian Fehrman:

about it.

Ethan Robish:

An open weight model.

Derek Banks:

Yeah. So we did talk about it. And I guess, I had a couple of multiple takes. My first one would be is I've used Frontier Models for both offensive and defensive work and I I suppose I'm probably in the cyber verification program. I definitely have gotten guardrail before, but I also get Frontier Models to do a lot of things that like would be, you know, considered hacking.

Derek Banks:

And so I I don't I'm not saying Aker is wrong. I'm saying that I I I'm kinda surprised from the defensive standpoint, because I've usually gotten a lot of mileage basically saying, hey, I'm on an authorized pen test, or hey, I'm doing an IR. And then I've pretty much been able to do what I've needed to do. I mean, that said, I mean, I'm probably referring to Opus four six, because I

Ethan Robish:

pretty much have penned to Opus four six. So The Frontier Labs have gotten more strict with the guardrails as they release more

Derek Banks:

capable models. As a four eight

Ethan Robish:

The whole point of like, this news story where, you know, this model escaped and like started attacking other, Like, they didn't have those guardrails on it, like, on purpose because they were doing internal testing. So I mean, the guardrails seem to be somewhat effective. Like, they I mean, they're not perfect. I'm sure by any means, like, people can get around them. But there's a lot of difference between Frontier model with no guardrails versus the Frontier model with guardrails versus, oh, I can get this open weight model that has no guardrails.

Ethan Robish:

And like, yeah, maybe it's not better than the what the Frontier with no guardrails, but it's better than what I can get, like Yeah. What they're serving me.

Derek Banks:

So I I definitely agree with the sentiment. If I I I think that if I run an IR team, like, you know, at a at a company, I would definitely want to get, like, a Spark DGX and and test my whatever I'm doing with AI now and using Frontier Models, I would wanna get a local model and test out my my playbook, whatever you wanna call it. I'd wanna make sure that was in place before I had to figure out that I couldn't use my AI solution to do what I needed to do. I don't know that's any different than IR preparation with other tools though. I I think it's just something that I I definitely recommend but, so that is definitely if if GLM, I think five two is what they were using, right?

Derek Banks:

I mean, it were banned, what how would how would you so if you ban open weight models, do you, really shoot defenders in the foot? It's kind of the thing. And I think the answer is ultimately, yes.

Brian Fehrman:

Yeah. I mean, yeah yeah. I mean, both it just security community as a whole, I would say, it severely restricts, our ability to quickly, find and mitigate issues, to do post postmortem analysis. All the tools that we need as well intentioned people to do the jobs that we need to do to try to stop the bad people from doing the jobs that they wanna do. Right?

Brian Fehrman:

Like, we we need to be able to be on equal footing. Otherwise, we're automatically at a disadvantage and that, that is going to become a real problem because, you know, we we ban the the technology here. Well, people are still using it elsewhere, and now they have the advantage that we don't. And I I think that's that's a real problem.

Bronwen Aker:

Unfortunately, defenders are usually at a disadvantage anyway. And if this ban of open weight models were to truly come into play, we'd be even more disadvantaged because of these additional restrictions. And I don't think that any sort of open weight band is is going to have the net effect that the lawmakers want to achieve unless, of course, they're acting counter to what they're saying they're acting for.

Derek Banks:

Yeah. So here here's I I actually found what I was looking at last week. So in June not in June, the US Department of Defense, added Alibaba, section two o twelve sixty h list of Chinese military companies. They add added Alibaba to that list. And then shortly after that, I noticed that a project that I was working on that was using Quinn Models at Nvidia.

Derek Banks:

The Quinn Models had been removed. I was getting a four zero four and I think NVIDIA's official stance is it's part of normal rotation of deprecating models. Seems kinda coincidental. Right? And so I guess that's like my example of like back to earlier talking about a soft band.

Derek Banks:

I mean, you don't have to say, you general public can't go download and use this. Right? But if you're gonna, you know, you general public, like like Ethan was saying, yeah, I can go get a small model and run it on my laptop or on a Spark. And the barrier to entry for that, it's $5. Right?

Derek Banks:

If you wanna do something useful with it, if you wanna do something agentic with a, you know, like right now, I think it takes 86 gig of RAM on my Spark to run the Quinn 27,000,000,000 parameter fully like dense model with the full I think it's 200 k, maybe it's 256 k of of KV cache. And so it works pretty well. It's definitely slower than, you know, using an API, but it's also it's not like I'm running like the full model, the full, you know, as a two point two point four trillion parameters. It's probably, you know, anywhere from 800 gig to a terabyte on disk. You need like 1.5 terabytes of GPU memory to run that.

Derek Banks:

Well, that's that's $500,000. Right? And so the general public is not doing that. That's what companies do.

Ethan Robish:

Even small companies wouldn't be able to do that. Like Yeah. You'd you'd basically be key be gatekeeping for larger companies. Like, larger companies could get by with, alright, we can just host it ourselves, like, we'll have the capital expense of the hardware, but but but then smaller companies, small businesses are just like out of luck.

Derek Banks:

Oh, yeah. And then on top of that, like hosting, it's one thing. Wiring it into all your business processes is completely different thing.

Bronwen Aker:

And there's all the care and feeding of having to to maintain things and yeah. I mean, anytime you roll your own, anytime you're self hosting, you've dramatically increased the amount of labor involved. And that right there is a strong deterrent for smaller companies, especially to try and and do things on their own if they can't access the frontier servers, or they get priced out of the frontier servers. And now by cutting off the less expensive option of these open weight models. I I don't know.

Bronwen Aker:

This there's so many different ways this could go.

Derek Banks:

Yeah. There's one thing we didn't talk about and that's what if China decides that the West can no longer use their open weight models? Like if they make a decision to where we're no longer gonna release these open weights. Because I personally, I think the the reason that China is doing what they're doing while releasing open weight models is not because they're altruistic and wanna share and make the world better. I don't believe that.

Derek Banks:

Right? That's kinda what they're saying, like, you know, open source and open weights make us all better. But I think they're trying to undercut The US market. And then, you know, I think they're also behind a lot of the FUD surrounding data centers. Like, there's a big data center debate in my state right now.

Derek Banks:

And to the point where the governor is pausing building data centers and even my wife and I disagree with it like each other on this. She's like, screw AI and data centers. They use all the water. Like, no, they don't. What do you mean they don't?

Derek Banks:

I'm like, well, I mean, listen, if you wanna talk about water usage, look at golf courses. Are you up in arms about how much water golf courses use? Because it is an order of magnitude larger than all the data centers combined. How about the almond industry? Now to be fair, there are people who protest the almond industry.

Derek Banks:

Right? Because it takes a lot of water to make almonds. And it's like five orders of magnitude more. I actually have the stats somewhere because I was, you know, arguing with somebody as people do. Right?

Derek Banks:

And so but I mean, I'm not saying you shouldn't be against data centers or putting them in neighborhoods and stuff. I'm just saying water usage isn't a good thing. So I guess what I'm saying is I also think the Chinese are trying to make trying to help us lose the AI race by pushing for data center bans and and things like that. I really do. I mean, why wouldn't they?

Derek Banks:

I mean, you know, like, that that would seem to be a good campaign, like, let's, you know, let let them build them.

Brian Fehrman:

Yeah. And I mean, under undercutting the market is that has been their general economic strategy for a very long time because they have the population, the means of production, they have, the shipping ability, they have everything that they need to be able to basically do that to undercut the market, to make, to increase your liability upon their production.

Ethan Robish:

The the lack of restriction on stealing IP. Yes. Exactly. Yes.

Brian Fehrman:

All of that.

Derek Banks:

Well, and so and I actually do sort of agree with the administration on industrial scale distillation. But I also don't think that all of the Chinese models are always distilled from cod. Right? Because I saw a claim that the most recent I can't remember if it was Kimmy. I think it was Kimmy k three.

Derek Banks:

There was someone saying that they distilled it from Mythos. I'm like, well, or from Fable. I'm like, well, Fable was online for like seven days and then got restricted to like and so I I don't I don't know about that. So I think not all of know, them are

Bronwen Aker:

here's certainly seven days is a long time though.

Derek Banks:

Sure. Sure. Right? But I guess what I'm saying is is that I I think we can't just always say, hey, they're just stealing our stuff. They definitely are stealing our stuff.

Derek Banks:

And I definitely think that industrial scale distillation is part of that. But also, don't think distillation should be outlawed. Right? It's a valid technique. It's and and so and to your earlier point, Bronwen, if, you know, I do think that in the government as a whole, they've always not had folks who really know about technology and I would really like to see them making a better effort to bring in people to, you know, like, explain the technology to them.

Derek Banks:

And maybe they are behind closed doors, right? I I don't know. But like, I I don't have faith that the government will do things that are the and gonna make it better for us. So

Ethan Robish:

I shared this link, but I think it's I think it's very interesting. The CEO of Anthropic has a a blog post, I guess, our position on open weight models. And I mean, of course, he'd say that, you know, their motivator is not profit or, know, like competition or whatever. But I mean, it is it is interesting to read what he does say in Clayton. Like, his concerns are abuse by authoritarian governments and cyber attacks and biological attacks by, I suppose, But the I mean, those are scary concerns, but also I don't I don't know how we avoid that.

Ethan Robish:

Like, he he has some points about, okay, we shouldn't sell powerful chips to China. We should crack down on distillation operations, what what we were just talking about. And the third one, which is interesting, I I don't know how we would enforce it, but he says, all sufficiently capable models. So there we were talking about, like, size wise probably, but also, I guess, advancements and techniques and stuff. But either both open and closed should go through mandatory safety testing.

Ethan Robish:

And I suppose that directly addresses his points, his concerns about like cybersecurity and biological warfare. But

Bronwen Aker:

There there is one fly in the ointment regarding large language models and safety, and that is that the longer the LLMs are around, and the more people think that they understand how they work inside, the more it's become obvious that large language models are never going to be not susceptible to social engineering attacks to some degree. And this is this is not just me talking through my end. I've seen at least three different studies that have all come to the same conclusion. So in the face of that, the question of open weight versus frontier models kind of becomes moot. Because if they're all going to have these same vulnerabilities, how do we honestly defend against that?

Derek Banks:

Well, I think you have a point. And then also, like, I'm I'm I'm with him in the article until that last point, all sufficiently capable models. So first of all, who decides what's sufficiently capable? Right? I mean, certainly in the case of biological or chemical attacks, I get it.

Derek Banks:

But if I was a a neuroscience researcher, I I I bet I kinda want some of the things in there that can be used for good or evil. Right? And then and and then, you know, you say you're four open weight models, but you're gonna release an open weight model after it goes through this testing. Well, then I can just remove the safety guardrails. Like, I I mean, Brian and I have this as a lab in our class.

Derek Banks:

Right? You can just ablate a model or obliterate a model and remove the safety guardrail. So it's open weight. So what would be the point of it going through safety testing? And then my last point is he and he admits that the CCP would have to be on board.

Derek Banks:

I'm sorry. Do we really believe what our global, you know, competition says to us? Do you think they would potentially lie? No, never. Right?

Derek Banks:

I mean, come on, it's naive. Right? That they would say one thing and do another? Right.

Ethan Robish:

Okay. I just I mean, I wouldn't trust our our own government or any No. Any any government to not lie about. Yeah. Yeah.

Ethan Robish:

Hey, we're gonna do that. Or we're say this thing. Tell politicians. How do you test them out? Secret, you know we're

Derek Banks:

gonna lips are moving.

Ethan Robish:

Yeah. You know we gotta do the thing in secret because they distrust everyone else. And if we don't do it, then we're falling behind everyone else.

Derek Banks:

Yeah. Unfortunately, like, the more I've thought about this, and I drove a long time yesterday, so I thought about this a lot. I think the only real answer is we have to get, we have to maintain our superiority in in in AI research and and also hosting AI. Like, it's one thing to to make the models, but it's another to use them on a large scale because you have to run inference. Right?

Derek Banks:

And that's where I do think things like chip restrictions do do make sense. But, you know, what an interesting world we live in.

Bronwen Aker:

May you live in interesting times.

Derek Banks:

It's like an ancient curse, an ancient Chinese curse. Right? Or yeah.

Brian Fehrman:

Yeah. So, you know, as we said earlier, there's not currently any bans in place in The US in terms of open weight models that we're aware of. There's discussions surrounding what that looks like, what they're gonna do going forward, and we just kinda have to wait and see see what happens. So with that, I hope everyone enjoyed the episode and learned something new and see you next time and as always, keep on prompting.