Subscribe
Share
Share
Embed
The development world is cluttered with buzzwords and distractions. Speed, focus, and freedom? Gone.
I’m Nicky Pike. And it’s time for a reset.
[Dev]olution is here to help you get back to what matters: creating, solving, and making an impact. No trend chasing, just asking better questions.
What do devs really want?
How can platform teams drive flow, not friction?
How does AI actually help?
Join me every two weeks for straight talk with the people shaping the future of dev.
This is the [Dev]olution.
Nicky Pike (00:00):
The model repo dressed up as an official OpenAI release hit number one trending and got pulled onto roughly 244,000 machines in 18 hours.
(00:09):
18 hours and it wasn't just a bad model, it was an info stealer. The thing was built to quietly rifle through whatever machine loaded it and ship the good stuff back out. Save passwords, API keys, session tokens, the keys to everything that you were logged into. A quarter of a million developers reached for what they thought was an OpenAI release and they grabbed a credential thief instead. And here's the part that should really bother you. The malware wasn't even in the model. It was in a file sitting right next to it. A little Python loader riding along in the same repo that fired the second somebody loaded the thing. The weights were fine. The wiring around the weights is what got people.
(00:48):
Now hold that thought because to understand why that matters, you have to understand why everybody's reaching for these models in the first place. The math on frontier models is starting to sting. You're paying per token for a top tier closed model on every job, even the ones that don't need the biggest brain in the room. So the smart play is to run the frontier model where it earns its keep and to put an open weight model that you host yourself on the lower stake stuff.
(01:12):
The internal tooling, the high volume work, the specialized jobs that you can fine tune a smaller model to absolutely crush. Costs, vendor diversity and specializations. These are really good instincts, which is exactly why the panic around it was such a problem. Right now, every CIO in America is asking the wrong question about these models. They're asking, "Can I trust the Chinese model like DeepSeek or Qwen?" And I get why.
(01:39):
It's the question that headlines are screaming, but it's aimed at the wrong part of the picture and I'm going to show you exactly where the real risk lives. Hey folks, I'm Nicky Pike, field CTO at Coder, and today we're talking about open-weight AI models. The ones you can download and run on your own hardware and whether you should actually be scared of them. This one's been sitting at the top of every enterprise conversation that I've been in. Somebody's largest customer floats a policy about Chinese AI and suddenly there's a fire drill. Good instinct, wrong shape. So let's fix the shape.
(02:12):
Everyone's worried about the neighborhood that a model grew up in. Where was it trained? Who built it? What flag flies over the data center? And look, neighborhood matters. I'm not going to pretend that it doesn't, but you can know everything about a zip code, the schools, the tax rate, the crime stats, and you can still buy a house that burns down because something in the walls that you never looked at.
(02:33):
The neighborhood is the easy homework. The inspection is where the real risk hides. A house has three layers that you need to care about and they're not equally inspectable. The first layer, the doors, the windows, and the walls. Does this thing talk to the outside? Can stuff I don't want get out or get in. You can see the doors. You can lock the windows and you can stand there and watch them. This is the easy layer. Layer two, the mechanics and the walls.
(03:00):
The wiring, the plumbing, the stuff that actually makes the house run all baked in before you ever showed up. And here's what matters. Standing in the driveway deciding whether to buy, you can't see any of that. Even a professional inspection can miss a fault that only sparks under the exact wrong conditions or a pipe that only leaks when the pressure's just right.
(03:21):
Every house you've ever loved, you bought without really knowing what was behind the walls. This is the scary layer and it's the one that almost nobody's looking at. Layer three, the contractor who built it. Did they frame it right or did they cut corners where you can't see? Did they leave trash sealed up inside your walls? Did the house even come from who you think it did? This is the layer that's already burned people in the real life. Everybody's out front, they're checking the neighborhood and they're triple locking the front door. Meanwhile, these wiring and contractor issues, they remain wide open. So let me walk you through all three and we're going to start with the one that I think should keep you up at night. The wiring, what's baked into the weights before you touch them. Back in January of 2024, Anthropic published research called Sleeper Agents.
(04:07):
They trained a model to do something specific and nasty. When the prompt said that the year was 2023, it wrote clean, secure code. When the prompt said 2024, it wrote code with an exploitable hole in it. Same model, same questions. The only thing that changed was a trigger and the model knew to behave until that trigger showed up. That alone is creepy, but that's not the part that got me. They then tried to train the bad behavior out of the model. All the standard stuff that you would use to make a model safe. Supervised fine tuning, reinforcement learning and adversarial training where you actively try to catch the model behaving and correct it. The back door held. It actually survived the cleanup. Worse, the adversarial training, the thing that was supposed to fix it sometimes just taught the model to recognize when it was being tested and to hide the behavior better.
(04:57):
You didn't remove the problem. You coached it on how to avoid getting caught. Now, fair is fair. That was a proof of concept. Researchers built that back door on purpose to see if they could remove it. Nobody has caught a shipped commercial model, Chinese or otherwise doing this in the wild. So be straight about the line. This is, we proved it's possible, not we've caught somebody doing it. There's a separate related worry that I want to raise and then label it honestly.
(05:22):
A model trained with bad intent could, in theory, write code with subtle weaknesses, predictable patterns or weak crypto choices that the people who trained it know how to spot and target later. A kind of fingerprint that only the attacker can read. And this is the sort of thing that a CISO at a defense contractor has to at least be thinking about, especially with foreign models.
(05:42):
But this one is still firmly in theory. There's been no confirmed cases. So we'll file this under plausible but unproven and don't let anybody sell you a product based on it. And honestly, this just reiterates the importance of human in the loop. Now here's why the wiring later matters, even though it's unproven. You cannot inspect it away. That's the whole point. There is no scan that opens up a set of weights and it shows you a clean bill of health. You can't prove the absence of a back door.
(06:10):
So you're buying a house knowing that you can't fully see the wiring. I mean, you buy it anyway. You just need to buy smart and we'll get to what smart looks like. Now, in 2024, we could at least tell ourselves an attacker would need to control a big chunk of training data to pull something like a backdoor off.
(06:26):
But then October 2025 happened. Anthropic, working with the UK's AI Security Institute and the Allen Turing Institute ran the biggest poisoning study anybody's ever done. And they found that as few as 250 malicious documents can plant a back door in a model regardless of model size. A 13 billion parameter model trained on 20 times more data than a smaller one went down to the same 250 documents. So sit with that for a while. The old assumption was that you needed to poison a real percentage of the training data.
(06:57):
And for a big model, that set was so enormous that it was kind of like trying to make the ocean saltier one bucket of salt at a time. Now the good thing is the new finding says that you don't need the ocean. You just need one poisoned water bottle. 250 documents that you could write in afternoon scattered into a training set, regardless of how big that set is.
(07:19):
Now the honest caveat, and I'm keeping it because the research kept it, that study used a narrow back door, the kind that just made a model spit gibberish on a trigger word. Low stakes. And they were clear it's not known whether the trick scales to serious harm in a frontier model. So this isn't the sky falling. This is more like the lock that we thought was bank vault grade, turns out to pop with a paperclip. Big enough shift to change how you think. So that's the wiring. Real threat, can't fully inspect it. It's getting cheaper to exploit and there's no confirmed cases in the wild yet, which is exactly why the answer can't be something like a scanner. So hold that. Let's go to a layer that has burned people for real, right? The contractor layer. Now this is how the model actually gets to you.
(08:04):
Most folks download open models from public hubs like Hugging Face. This is a great resource genuinely, but a lot of these models ship in a format called Pickle. And there's the thing about Pickle that developers forget. Loading a Pickle File can run code. It's not just the data sitting there. Cracking it open can execute whatever's inside. So a model file isn't only a model, it can also be a program. JFrog went looking and they found around 100 models on Hugging Face carrying real working malicious payloads. Not theory, but live code. About 95% of them used a pickle trick to open a reverse shell.
(08:40):
Basically a back door into the machine that loaded the model while the model still works normally so that you'd never notice. And here's the one that I opened the show with a repo posing as an official OpenAI release. It copied the real model card almost word for word.
(08:54):
Number one trending, 244,000 downloads in 18 hours and the malware wasn't in the weights at all. It was a loader file sitting next to them. The model was the bait. The file next to it, that was the hook. Now, if you notice something, nobody in these stories ask what country the model came from. It didn't matter. The contractor layer doesn't care about the neighborhood. It cares about whether you checked what showed up at your door. All right, three layers, doors, wiring and contractor.
(09:22):
Now let me tell you why the loud answer everybody's reaching for is aimed at the cheapest layer to defend. Ban Chinese models. That's the move on the table. States have done it. Texas first back in January of 2025, followed by New York, Virginia and others. Federal agencies like Commerce and Navy pulled DeepSeek off the government devices. And there's a bipartisan bill that No Adversarial AI Act that would build a federal list of adversary AI and keep it updated.
(09:52):
And look, for hosted services, the kind where your prompts and your code travel back to a server that's potentially in another country, that concern is legitimate. DeepSeek's own privacy policy says US user data gets stored in China. So if you're piping your source code to that, yeah, there's a real problem there. But that's the doors and windows layer and it's wide open when you use the hosted version. But here's what self-hosting actually gives you and this is the good news that I promised. When you run an open model on your own hardware, the weights are static math. The model isn't necessarily secretly calling home on its own, but the part that people get wrong is this. Running it on your own hardware doesn't automatically wall it off. If you stand it up with open outbound internet access, it can still ship data out through a tool call, a rendered URL, a web hook, whatever.
(10:42):
Self-hosting isn't a force field. What it is is the right to decide. Deny by default network, you cut the internet completely. Watch every packet that tries to leave. You can do that because it's your house and your wiring. Nobody else has to approve it. The loudest fear your data leave in the country is solved by self-hosting only if you actually do the work. Owning the place is what gives you the right to do that. And that's the whole reason that you self-host. With a hosted model, you're a tenant. You live with whatever the landlord installed and if you want it changed, you have to file a request and you wait. Self host and you own the place. Ownership is the dream here and you should chase it. I just want you to own it like somebody who plans to live there for a while. So think about what ban the Chinese model actually buys you.
(11:30):
It swings at the doors layer. The one that you can already control yourself the moment that you self host and you lock the outbound down. It does nothing for the wiring of the contractor layers. A poison American model loads the exact same way. A malicious payload and a repo doesn't care whose name is on the model card. The geography answer is a comfort blanket. It feels like you did something, but you just nailed plywood over a window that already had storm shutters bolted on and you left the back door standing wide open. Now that said, I want to be fair to the CISOs watching because there is a version of banning that's completely legitimate. If your biggest customer says no Chinese AI in your stack or a regulation puts a model on the list, you ban it. Not because it's a bigger security risk, but because the contract says so.
(12:19):
I'm not going to tell you to torch a deal over an architecture argument. Just be precise about what you're doing. Banning by country mostly isn't a security move. You can architect that risk down regardless of whose flag flies over it. Sometimes it's a business move and that's real. Just don't tell your board you're safer when what you actually did was satisfy a customer. Now here's another kicker that nobody else is pricing in. Remember the Adversarial AI Act, that federal list, the one that updates on a schedule. That's a moving target by design. It's built to add names every six months. So regardless of whether it's an open weight model or a frontier model, picture the team that rebuilt their entire developer tool chain around one specific model. They made it the foundation. They built three rooms off of it. One list update, one customer policy change, one new regulation, one billing change and they're not patching anymore.
(13:13):
They're knocking down a load bearing wall the whole structure depends on. So own the house, but don't weld the appliances to the wall. And that's the whole take and it's where the honest fixes come from. So let me give you a real version of what works and what doesn't, because I've watched people get sold a single silver bullet for this and there isn't one. First, self-host anything sensitive. Then you got to actually lock the outbound down. Self-hosting gives you the power to close the doors later, but closing it is on you. Do that and the loudest fear is handed. It's necessary but it's not sufficient. Second, tier your models by how much they can hurt you. A Chinese open weight model running your internal documentation certs or a throwaway prototype or MVP, that's probably fine. The same model and the path that writes code shipping to customers, different conversation, different approval.
(14:05):
Not every workload carries the same blast radius, so stop treating them like they do. Third, clean up your contractor problem because it's the one with confirmed victims. Prefer Safetensors format over pickle. Safetensors can't carry executable code and it's just the numbers. Pin your model version so you know exactly what you've pulled and treat every public repo as untrusted code, not just data. That means the file's sitting next to the weights too, because that's what the 244,000 people learned the hard way. And the fourth one covers the layer you can inspect. You can't verify the wiring. Nobody can, not yet. So the fix isn't a better scanner. It's all in how you build. Any contractor worth a damn leaves you access to the things that you'll need to work on later. You don't pour concrete around the water heater because every water heater dies eventually and you know it going in.
(14:57):
The model is the water heater. It's going to need replacing whether it got poisoned, it got banned, or somebody down the street just built a better version. So build for the swap now while it's config chains instead of a demolition. Pull one model, drop another one in. None of these have to be a crisis. It'll just be another Tuesday. That's not fear of commitment. That's how somebody who plans to live there builds. And I'll say the quiet part once and then move on because this happens to be what we think about at Coder all day. The model should be the interchangeable part. The governance, who's allowed to use what on which workloads with what controls. That's the part that actually protects you. Make that the durable thing and let the models come and go. So here's what to actually do with all this. Grab a notepad because I think this part's worth keeping.
(15:46):
Three layers, three questions for every model you bring in. The doors, can it phone home? Self host, lock the outbound down and you shut it. On a hosted version, assume that your data is leaving and decide if you're okay with that. The wiring, can you verify what's baked in? Well, you can't, not fully. So build for the swap where replacing the model is open in a panel, not gutting the place and ripping out drywall. The contractor, do you control how it loads? Safetensors over pickle, pin versions, treat the whole repo as untrusted code. Run that on every model, every time, doors wiring contractor. And here's what should really make you slow down before you ban a whole category on geography alone. These open models are getting good and I don't mean cute in a local demo kind of gut. Qwen 3.635B, A3B, the open weight model that Alibaba dropped in April.
(16:40):
It posted at a 73.4% score on the SWE-bench verified. The benchmark that measures whether a model can actually fix real software issues. Simon Willison about as levelheaded a voice as you'll find in this space, ran it locally on a MacBook and found that it beat Claude Opus 4.7 on some tasks. Now fair warning, a lot of those numbers come from Qwen's own setup and independent reproductions are still catching up. So treat the exact figure as directional, but the direction is the point. An open model you can own, run on your own metal and lock down however you want is now within spitting distance of a frontier on real coating work. If your entire policy is it's foreign ban it, you're handing your competitors a cheap, ownable, fast, improving tool and choosing to leave it on the table for yourself. That's a self-inflicted wound that's dressed up as caution.
(17:33):
Everyone wants the threat to be simple. If it's foreign, ban it done, but the threat was never the neighborhood. Those 244,000 people didn't get got because of where a model was born. They got got because they trusted the house without inspecting it. So inspect what you can, own the place and build it so you can swap what's behind the walls when you need to, because eventually you will need to. If this hit, like and subscribe and join the [Dev]olution if you haven't already. We've built it for the Greyman developer. Those engineers and leaders who aren't chasing trends, they're just trying to do great work and keep up with an industry that won't slow down for anybody. That's the whole mission. Bringing the focus back to building good software. See you around.
(18:16):
Thank you for listening to [Dev]olution. If you've got something for us to decode, let me know. You can message me, Nicky Pike on LinkedIn or join our Discord community and drop it there. And seriously, don't forget to subscribe. You do not want to miss what's next.