Subscribe
Share
Share
Embed
For long-form interviews, news, and commentary about the WordPress ecosystem. This is the companion show to The WP Minute, your favorite 5-minutes of WordPress news every week.
Eric Karkovack (00:00)
Hi everyone, and welcome to the WP Minute. I'm Eric Karkovack. Today's episode is all about security. Specifically, we'll look at the recent supply chain attacks impacting dozens of WordPress plugins. Austin Ginder is here to help us sort things out. Austin is the owner of Anchor Hosting and the author of WP Beacon, a new tool that aims to identify supply chain risks. Austin, welcome to the WP Minute.
Austin Ginder (00:27)
Hey, thanks for having me. yeah, ⁓ I'm happy to talk about this subject and particularly this last month has been a wild ride for me. I just need to say upfront, like I'm not a security expert. I'm just a developer armed with AI and it is just, it's crazy times we're living in.
Eric Karkovack (00:48)
Yeah, absolutely. And I think it's interesting that, you know, I could go back like the first time I heard about one of these supply chain attacks was actually, I was looking it up, it was 2017 and there was a plugin that changed hands that had, ⁓ you know, some malicious code added in there. ⁓ But we'll get to that. But I wanted to start with, you know, you did a lot of research, it sounds like on what was happening. ⁓
on these supply chain attacks. What got you interested in this and ⁓ can you give us little bit of background on what you found?
Austin Ginder (01:28)
Yeah, so I think it's important to note that most of what I've been finding is completely by accident. I've been very lucky.
So the backstory really is story about AI and how everything kind of changed at the beginning of this year. For me personally, ⁓ like I was a very heavy user of generating code with AI all of last year. used Google Gemini for a while, but I didn't really use an agent for the first time until around February. When I used an agent, Cloud Code specifically, that's kind of when everything
Eric Karkovack (02:02)
Okay.
Austin Ginder (02:07)
just started to get really crazy. I'm for my business, if I take a step back, so I manage around 3000 WordPress sites. I'm a solo developer. I have some people that help me out from time to time, but the business is me. I'm a one man shop.
I had a huge influx of sites getting hacked and malware ⁓ as a byproduct of thanks to everyone having this superpower called AI. ⁓ so ⁓ February was a month of patching things up, getting things in a better place. ⁓ March was a lot of, all right, what can we do to...
⁓ actually go on the offense, like they'll redo my entire security system. ⁓ then in April, I started to reap some benefits from that. going back to the supply chain attack, the first one that I discovered was just a byproduct of me doing malware cleanup on my own customers. And ⁓ what?
Eric Karkovack (03:18)
Okay.
Austin Ginder (03:19)
you can do with Cloud Code specifically ⁓ with malware cleanup is just absolutely phenomenal. You can just keep digging and you can keep uncovering stuff. So, ⁓ previous malware, if WordPress sites get to malware infection, you can restore from backup, you can fix things up, you can get a developer involved, but you really don't get to the...
⁓ very rarely do you get to uncover the whole story. Like, this is how the hacker actually got into your site, and this is what they did. And this is the line of code that was the reason why everything was vulnerable in the first place. Well, with AI, you can just do crazy forensics level... ⁓
Eric Karkovack (03:49)
Sure.
Austin Ginder (04:07)
uncoverings to the point of, okay, this is exactly where it happened, and this is how it happened, and we can just patch those codes. And it's all in a conversation with working with Cloud Code. So anyway, the...
Eric Karkovack (04:23)
That's wild.
Austin Ginder (04:25)
The supply chain tax it was literally just kept digging like how did this happen? All right, let's expand the scope. Let's look at the plugins Let's look at the author of the plugin. Let's look at what's going on. ⁓ I think there's something here I should probably reach out to the WordPress the WordPress plugin team To report this as an issue. So that's that's how I got started and that was like a month ago
Eric Karkovack (04:53)
And so that's quite a story. honestly, I'll tell you, I don't feel as bad now that I've had sites hacked, I know you're a pretty darn good developer. So if you've had sites hacked, I don't feel nearly as dumb now. So thank you for that.
Austin Ginder (05:08)
Yeah,
the playbook for keeping your site secure just completely changes this year in particular because I was doing all the right things. ⁓ I run a very strict...
⁓ I update all my customer sites every week. I run backups, run long-term backups, I run an entirely different ⁓ revision history system so that I can catch file level changes immediately. So I was doing all the right things and I would say that was good advice up until about October, November of last year.
Eric Karkovack (05:49)
Okay.
Austin Ginder (05:49)
And that
advice started to fall short because people got really good AI and you could just scan for vulnerabilities and you could find stuff. Going back to my main point, like I'm not really a security expert. I've historically not submitted things to security teams to get patched. Well, with AI, I'm finding them all the time. Like you're on a scan on a site and it's like, oh, this block of code is...
Vulnerable, I should probably report that so I tell Claude hey draft this findings up to the author It kicks out an email draft. I'm like that looks good send Well, if they're real team they get back to you right away. They're like, thank you for letting us know This will be patched in the next few days so Yeah, like it it I think ⁓ the big takeaway is things have changed
So like, not just me, but like everyone needs to stop their game when it comes to security. And the rules are just getting rewritten right in front of our eyes.
Eric Karkovack (06:57)
Yes, it seems like the same old cat and mouse game that we've been playing for years, but now with maybe on steroids with these AI tools that, ⁓ as you said, I mean, you can actually go in and see that what vulnerable code there is, but also a hacker could do the same thing, right? If they're, well, if they want to target a certain plugin or a certain site, they absolutely can do that.
Austin Ginder (07:14)
Yep.
Yep. Some of these discoveries have been dormant for years. And also, part of my new process is I aim to get 100 % code audited for all of my customers. And that's kind of difficult for me to do, but I'm making progress. I'm getting there.
During this process, it's like a house cleaning. Like I'm uncovering things that have been planted there for years, like PHP back doors, ⁓ things that all the classical security systems have missed. Because a security system at the end of the day is just as smart as whatever. ⁓
Fingerprint you give it like hey look for things that do this But with AI it's kind of like human level intelligence to evaluate is this thing a real threat or not and ⁓ That's the difference
Eric Karkovack (08:12)
Sure.
I was going to say, mean, the security tools that we have now, and I know you're into hosting and, you know, hosts have their scanning tools that they use at the server level. We have things like Cloudflare. ⁓ And at the, I guess at the ⁓ basic level, we have security plugins that we all put on our sites and those all do something, right? But yet thing after thing after thing is missed and sites still get hacked.
So you can't always find these little back doors that are hidden in there. like Anything surprise you with this or is this kind of what you expected to find?
Austin Ginder (09:07)
I was shocked at how bad things were. I am still very optimistic because I feel like things are gonna get bad and then things are gonna get really good because this is all stuff that was hidden in the dark and once we just
Shines and light like I didn't realize how bad it was. Well, that's not really the problem The problem is we just we need to clean it up and then we're going to be set for the future. So ⁓ And also this is not a wordpress problem All code is suffering through this like if you look on any security bulletin board any software linux any os
We're all, they're all doing with the same thing. It's just like, we didn't realize how much bad code there was out there in the world. And now we can just press a few buttons, a few conversations with the chat bot and like see it. It's all there.
Eric Karkovack (10:04)
So tell me a little bit about ⁓ the genesis of WPBeacon. So it's WPBeacon.io, right? This site ⁓ goes and kind of looks for patterns, right? It's looking for ownership changes and SVN changes and things like that.
Austin Ginder (10:13)
Yes.
⁓ So ⁓ WPBeacon is a name I've had for a while. ⁓ I've lots of names. ⁓ But the idea for WPBeacon really came a couple of weeks ago when I was dealing with the larger findings about WordPress plugins that were being operated by either bad actors or other things they were doing that were harming people's websites. ⁓
After about two or three of these major findings and getting a lot of press around that, like, you know, I want to actually find one of these on purpose. Like I'm just stumbling in the dark and finding stuff on accident, like cleaning up a site or finding one of them I uncovered because I wrote some tools to scan my customers plugins for variants. So I was specifically looking for
Is there a batch of plugins that are running a version of the plugin that isn't the official version? And I found one it was it was I'm like, well that's weird and then that led to what would it happen? So
Eric Karkovack (11:34)
Yeah.
Austin Ginder (11:38)
⁓ A lot of what happens with WordPress plugin repos, ⁓ when there's bad actors, a common trait that they do is they will try to hijack the update system, meaning they sneak an updater into the plugin. You download it off of WordPress.org and then they hijack it off of WordPress.org so that they can give malware. So if you're trying to scan for malware on WordPress.org, it's very difficult because it's not there.
what's happening with the supply chain attacks, your clean version got replaced with a version that has malware. ⁓ So after about a couple of these stumbling through, I'm like, I want to see if I can try to find one. So that's why I started to work on WPBeacon last month. ⁓
Like I said, the name I've had for years, I've always wanted to do something in that space of either code verification or finding, and it just felt like the right idea for the right name. ⁓ So I started to write very crude AI. Some people would call it AI shlop. I'm just throwing ideas at AI. I'm like, hey, download everything we can on ⁓ from the...
WordPress.org repo. Download everything we have information about the committers. Here's some code patterns of common ways that bad actors hijack their own plugins. Let's look for them. So I just like pulled it all into my laptop and just like created this thing trying to scan for things. And I had my first hit and it was ⁓ when I was scanning the top...
2,000 plugins on the WordPress.org repo. For context, that might not seem like that many plugins, but it's actually a lot of plugins. I forget the numbers off the top of my head, but there's over 60,000 plugins out there.
Eric Karkovack (13:43)
Yeah,
that's a lot of installs.
Austin Ginder (13:45)
It's a lot, but like if you take let's say the 10,000th plugin, the top 10,000 plugins, well it's only got 400 installs. So what I'm saying is by targeting the top 2,000 plugins, it's actually hitting most websites. And then beyond that, it starts to trickle off.
So I ran a scan, it found something. This is just using cloud code on my laptop. I'm like, wow, I actually found one. And it's live. it was...
pull it up. The school top was at 20,000 installs. So it was like live and running a hijacked version on 20,000 WordPress sites. I'm like, this is insane. So I reported to the WordPress plugin team and by now they know me because this is like the third or fourth time I've interacted with them in the last month. And they're like, ⁓ thank you for the report. And they closed it out and ⁓ issued a... ⁓
Eric Karkovack (14:34)
My gosh.
Austin Ginder (14:53)
patch to kind of follow what I was seeing. They were doing some weird stuff with version numbers. ⁓ So the version on WordPress.org was a certain point, and the version that got installed was a couple points above that. So the WordPress team followed my guidelines or some of my ideas and decided to bump it way up above both of them. So the compromised version. But the thing about these, the WordPress plugin team
Eric Karkovack (15:08)
Okay
Austin Ginder (15:22)
is kind of limited in what they can do, right? They are very able to, ⁓ they've been awesome to work with. They're very quick to shut this down once it's discovered. They're very quick to even put a patch out. But a patch in this scenario is unfortunately not gonna hit most of these sites because most of these customers are already hijacked off.
So the cleanup process is left to the site owners. And that's what these bad actors are going for. They want to compromise sites and we want to keep them compromised for as long as possible so that they can get whatever they want out of it, whether that's serving spam ads or SEO junk, whatever they're trying to do.
Eric Karkovack (16:12)
Yeah, so that patched version, just to be clear, we still have the back door in there, right? I mean, the plugin itself is no longer vulnerable, whatever back door may have been installed is still gonna be sitting there dormant until it's not, and then you're getting casino spam on your site or your Google results are poisoned by that type of junk.
Austin Ginder (16:36)
Right, so the trick is...
catching some of these before they're activated. So this was hardwired to do a content delivery system where it will display content, but that server was never active. It was dormant. It was hardwired. It was plugged in. The bad actor maybe even sold positions like, oh, I've got this, cover my servers, but it was never activated. So when we shut it down, I also reached out to my friend, Sal. He used to work for Kinstub, but now does.
⁓ some security stuff on the side and ⁓ he was able to get all those compromised servers turned off so they don't have any foothold on those sides anymore.
Eric Karkovack (17:20)
Well done, well done.
my gosh, it's nice to actually hear a happy story about this stuff.
Austin Ginder (17:26)
Yeah,
so some of the times you can do something about it ⁓ that's kind of outside of my expertise, but if you ping the right person, you can get things done. yeah, like going back to WordPress security team and WPBeacon, project itself. So I built WPBeacon to try to find some myself. And I see them solving, they're coming at it from...
two different vantage points. Like they are very complimentary. WPB can I feel is like a how much data can we harvest out of things out of these supply chain attacks that previously would have been impossible to pull out. So running one of these ⁓ audits, it's able to chain together.
many layers of details that was previously missed. Like, here is the exact pattern. Here's when they created the WordPress account. Here's the domains they're using. And here's how it's all interlinked together. And even over the past weekend, I uncovered one ⁓ a supply chain attack that WordPress plugin team closed off about a month ago.
But it was able to correlate it to other ones over the years and different WordPress profiles have been pulled together. So that one's wild. I mean, that's like...
Eric Karkovack (18:49)
Wow.
Austin Ginder (18:55)
this guy's been operating a compromised servers for 13 years and no one's correlated the data because it would have been impossible to sift through all the information to figure it out. But if you have AI and you can point it at the right direction, it can start to pull it all together. So, I mean, that's kind of my hope for WPBeacon is just to like expose some of these incidents and maybe start to like have... ⁓
Eric Karkovack (19:00)
13 years? ⁓ my gosh.
Austin Ginder (19:24)
⁓ It would be great if it could proactively find situations where bad actors have things wired up and take them down before any damage is done.
Eric Karkovack (19:37)
Yeah, I mean, because right now, mean, have, you know, plugins change hands all the time, right? I mean, we have, you know, acquisitions all the time. But we, I don't know if there's any process in place to really, ⁓ you know, verify any of that stuff, right? I mean, like anybody can take over a plugin and do something, you know, malicious with it, right? Do you see anything else we could be doing that, you know,
When a plugin changes hands like this, is there something that maybe the plugins team could do or AI could automatically check that we're making sure that this person is on the up and up and that the code that they're committing is safe?
Austin Ginder (20:24)
Yeah, I think.
I think this is eventually going to be implemented kind of across the industry, and that is having AI audit the code. ⁓ So like from the WordPress submission point of view, I think they can shut it down future cases by having AI do a multi-level check. Are there any updates, libraries that goes against the codes of services or the requirements that are outlined on WordPress.org?
⁓ Are there any obfuscated update libraries? Because some of these are very well hidden. It's not like you just search for a certain filter and you see all the updaters. No, they're doing crazy stuff. They are doing bad things and they're trying to hide their tracks. And some of them are doing a phenomenal job. I think from, yeah, two-phase, like the plugin repo or the plugin team
They do have the ability to stop this happening at the front door, but that doesn't mean like it's but what's really hard is like figuring out the stuff that's already submitted because to do that that's that's a full code audit. That's that's going to be a massive project. ⁓ just like
Eric Karkovack (21:37)
Yeah.
With 60,000 plugins, it's not going to be kind of impossible,
Austin Ginder (21:47)
Yeah, so I guess one of two things, I think the low hanging fruit is, let's see if we can stop it.
upfront and what that would look like is every code that gets pushed up to SVN has to get AI code audit look over and see the diff and see if there's anything harmful being applied and now that'll stop it right out of the gate. And then there's the big project and this is going to be a very expensive project. Run the whole WordPress.org plugin repo through a full 100 % code audit and that's going to uncover a whole bunch
of mess. ⁓ Because these guys use different techniques, ⁓ so it's just going to require ⁓ more thorough adept. Now, from WP Beacon, what I'm mostly interested in is how many of these cases can I search and let the AI do the investigation report?
and surface that because the more that that gets publicized, it's going to be a lot harder in the future for someone to pull off hardwiring a whole bunch of WordPress sites to a compromised server.
Eric Karkovack (23:09)
Yeah, that's definitely the hope. you can, it is, I put something in our newsletter about this and I thought, well, if you can detect these patterns, certain patterns of what's happening, cause it seems like there is like a same basic backstory, right? With these, with these compromised plugins, if you can kind of spot that before it really gets deployed, you have at least a shot to stop it in its tracks and save a lot of people, a lot of money and a lot of trouble.
Austin Ginder (23:37)
⁓ One quick side story. ⁓ because of... Ironically, the post that I wrote that went viral, I had... I didn't do anything. I mean, the side story was one of my customers reached out because he saw the notice from WordPress.org.
Eric Karkovack (23:54)
You
Austin Ginder (24:01)
So someone in the community reported an issue and got the essential plugins suite of plugins ⁓ closed down. So it's like a batch of 30 plus plugins.
Well, one of my customer sites got compromised. Now, my malware scanner would have picked it up within 24 hours and would have sent me a notice and I would have cleaned it up. But it just happened that morning that one of my agencies, Ricky, the developer on staff, he caught the thing. He saw the notice from the WordPress plugin team, notified me. I ran the scan. was like, yep, it's compromised. The WP Config file has been tampered with.
kept digging in and I did the investigation work and then put out the story but I didn't do anything like it was all Claude Code I was just doing malware cleanup and put it out there ⁓ and ⁓ like I literally didn't do anything I I I was there with Claude Code having to
Eric Karkovack (25:00)
Yeah
You had the curiosity
to look though.
Austin Ginder (25:05)
I was having a conversation at the end of it, I'm like, this is wild. Like, Cloud Code did the research. It went and found the Flippa article where this team purchased the suite of 30 plus plugins for six figures on Flippa and from, ⁓ I think it was a team in India ⁓ that sold the company off.
and then they weaponized most of the plugins. And ⁓ I think the WordPress.org team is high praise for this because from what I saw, most sites did not get compromised. Like it was a huge, like I can't see it any other way. I think it was an epic failure for the one who's joined to compromising because ⁓ high profile, lots of sites wired up and
Like it was like one in 20 actually got compromised on my site and most of them Like they didn't even have a chance to roll the the compromise out to all their plugins They were in process of rolling it out when WordPress shut them off So it wasn't that like all 30 of them got weaponized, but he was systematically going through it and doing that ⁓
Eric Karkovack (26:12)
Okay.
That's just insane
to think about it. then, you know, obviously they paid a good bit of money for those plugins. to, always talk about here like, well, how do we make money in the plugin business? Well, this is one way I guess, but not the way you really should be making money in the plugin space these days. ⁓ It's crazy because I always think of it as, okay, it's just one rogue actor with one plugin. But to have a suite of 30 plus of them is just a little bit insane.
Austin Ginder (26:28)
Right.
Eric Karkovack (26:51)
⁓ So bringing this back to the rest of us, as hosts, developers, ⁓ site owners, like what should we be doing? Is there anything we could be doing right now ⁓ to kind of ⁓ tighten the screws a little bit on security?
Austin Ginder (27:10)
Yeah, I think my advice for the plugin repo holds for everyone. Like my process moving forward is everything needs to be 100 % code audited. And what that means is going to look different for the size of organization you're running.
⁓ For myself, I manage 3,000 sites. The way I'm doing it is I have clogged run a scan. It's a vulnerability scan. Malware is a little bit different, but it will pick up malware. That's not what this is for. This is sort of like looking for code that is easily ⁓ vulnerable. And ⁓ you can do this yourself. ⁓ It is if you download a backup of your WordPress site,
and you have access to cloud code, you literally just ask cloud code, hey, look over my files line by line for security vulnerabilities. And you might have to prod it a few times to get it to do the line by line because the line by line is expensive. So most WordPress sites are going to have, I'm not even sure, hundreds of thousands of lines of code. ⁓
So it's gonna burn through tokens is what I mean. I think anyone on a $20 a month plan can get through one site. That's fine. It's a lot of tokens. On my $200 a month plan, I can get through many, batches before I eat up my usage, but almost all my access usage for Cloud Code goes to this. It's like when I'm done for the day, I'm burning scans.
Eric Karkovack (28:29)
Yeah.
Austin Ginder (28:54)
So I'm doing it a little bit more systematic. I am every plugin or theme I create a unique hash for.
So I only run the audit once. If Jetpack version whatever is installed on ⁓ a bunch of my customer sites, I audit that version of the plugin one time and I mark it in a system and it never gets audited again because I've got that. I mean, I might run it through a better AI model in the future, but it's audit once ⁓ and move on to the next.
And it's just wild how many things I've uncovered for my own sites, how many have led to a direct security patch. ⁓ So I think that's what you just need to do. Like if you want confidence that your site's secure, just ask AI to do it. I will say one recommendation tip. There is a very big difference with these AI models in its ability to detect threats.
So I would only run the audits with the most expensive model. So for me, I'm using Opus 4.7. I don't trust anything else. ⁓ I would either that level or higher, not that level or lower. Like, yeah, you can use the DeepSeq Pro models or the Kimi K2 series for other tasks, but not this. The security audits require... ⁓
They just, they catch more things. And for security, you need the best set of eyes to look it over.
Eric Karkovack (30:40)
Yeah, wouldn't want to. I know these models change so much in such a short amount of time. So you probably wouldn't want to use the chat GPT model from three years ago to look at this. ⁓ So, I mean, what do you have any plans for anything else coming to WP Beacon? Any other like ⁓ wishlist items that you're hoping to add?
Austin Ginder (30:50)
Right.
So I think WP Beacon as it stands right now is a gap that's kind of unique because you have, ⁓ there's a lot of good security companies out there doing fantastic work on the code side of things. But.
I want to keep WP Beacon sort of separate from that ⁓ because there's nothing looking for bad actors and trying to learn from that. And I think there's value in there being a database forensics style for the bad actors. So I have lots of ideas around code audits and hashing, and I'm already doing that in my business. That'll probably be a separate project.
⁓ outside of WP Beacon. ⁓ But yeah, like I think there's just value in doing these investigative research into how these guys are operating. And my hope is it just becomes the thing that doesn't get used. Like we find all the bad actors and then WP Beacon doesn't have a purpose. Like that's...
Sort of what I would hope happens over the next three to six months as AI gets rolled out.
Eric Karkovack (32:27)
That sounds like a nice idea. We have this great security tool, but we don't need it anymore. We've already figured it out. But probably not. ⁓
Austin Ginder (32:36)
Yeah,
security goes like whenever it's an arms race, right? Like everyone tries to one up the other. So it's usually a never ending battle, but we'll see.
Eric Karkovack (32:50)
Yeah, I mean, I'm happy to see you using AI in such a positive way to make it like, you know, an actual good difference for, you know, so many people, so many site owners, so many developers, because, you know, I mean, there's always that balance, right? Because somebody's going to use it to do bad things and somebody's going to use it to do good things. So you've uncovered something. I mean, I'm sure six months ago, you probably had no idea that this is where you would be and what you would be doing.
Austin Ginder (33:16)
NARUDO!
Eric Karkovack (33:18)
So it's awesome that you had the mind to just dig into this. I know you give Claude all the credit, but I mean, someone had to type that prompt in there, right? And someone had to have the curiosity to say, I want to know more about this. So kudos to you on that. So I encourage everyone to check out WPBeacon.io. ⁓ Where else can folks ⁓ reach out to you online?
Austin Ginder (33:42)
I am on X at AustinGinder.com or all the places if you search my name. ⁓ Anchor.host is where I mostly do my blog post through Anchor Hosting's site. ⁓
Eric Karkovack (34:01)
All right, well, be sure to put those posts up in the show notes too, because it's really incredible what you found when looking through these plugins. And you cite a lot of examples in there. So I think anybody interested in security or just wants to know what in the heck is running in the background in their site will be very interested to know. ⁓
But Austin, thank you so much for being a part of this and thanks to everyone for watching and listening. Visit us over at thewpminute.com slash subscribe, grab our newsletter and support the work that we do here at the WP minute. Thank you and we'll see you next time.