Subscribe
Share
Share
Embed
At any moment around the world, cybercriminals and nation-state sponsored hackers are targeting businesses, government entities, and nonprofits for financial gain, cyber espionage, or geopolitical advantage. But behind every organization facing this nonstop barrage of cyberattack activity are dedicated cybersecurity professionals working to defend their organizations' networks, systems, and sensitive information.
Dark Reading Confidential brings you rare, firsthand stories from the cybersecurity practitioners in the cyber trenches, as well high-profile industry experts and headline-makers. Subscribe, and treat your ears to compelling stories and insights from the actual people fighting back against these hackers, and the experts helping to shape ever-evolving cybersecurity defenses.
Pen Test Arrest: Looking Back Five Years Later
Becky Bracken, Senior Editor, Dark Reading
Hello and welcome to Dark Reading Confidential, a podcast from the editors of Dark Reading.
My name is Becky Bracken. I'm an editor with Dark Reading and your host for today's conversation, Pen Test Arrest: Looking Back Five Years Later. I'm joined by Kelly Jackson Higgins, Dark Reading's editor-in -chief.
Five years ago on September 11th, 2019, a pair of cybersecurity pros named Gary De Mercurio and Justin Wynn were arrested while conducting an authorized pen test at a courthouse in Dallas County, Iowa. Following their arrest, what ensued was a heated years long battle between the two pen testers; their employer and company contracted to do the pen test Coalfire and its CEO, Tom McAndrew; and law enforcement; particularly Dallas County Sheriff Chad Leonard, who was seemingly bent on making an example of the pair.
Today, we are joined by Gary, Justin, and Tom to look back at the incident and how it affected them personally, professionally, as well as how the wider cybersecurity community conducts physical penetration tests. Welcome, Gary, Justin, and Tom. We are thrilled to have you here today.
Tom McAndrew, CEO, Coalfire
Thanks, Becky.
Becky Bracken
I would like to hand things over to Kelly who really was in the weeds with you on this as it happened as a reporter just to sort of walk us through the basics. Kelly.
Kelly Jackson Higgins, Editor-in-Chief, Dark Reading
Thank you, Becky.
Justin, Gary, and Tom, it's great to talk to you again. I've always wanted to circle back with you after we talked about what happened five years ago and kind of get a feel for how things have changed since then. I think this case, for those who don't know much about it, really was sort of a game changer for the physical pen testing world.
For those that don't know, physical pen testing really does rely on this pact between the client and the pen testing company that you, pen testers will be free from legal and physical peril when you do your job.
But this case really showed how things can go wrong sometimes. And I wanted to sort of go back to that night, Justin and Gary a little bit. I think it was after midnight, September 11, 2019, if I remember correctly.
You were obviously in the final phase of the pen testing engagement for Iowa's judicial branch, and you were breaking into the front door of the Dallas County courthouse with what I remember, a plastic cutting board that I think you bought at Walmart that had a, sort of had retrofitted it with a notch that you could kind of break into the door jam. Things kind of were going along at that point, right? And then things went in the other direction. So, kind of bring us back to that night. Justin, do you want to start just kind of where things went from there and how everything was going as planned for the most part.
Justin Wynn, Coalfire, former penetration tester and current director of cyber security services
Sure. Yeah, just to catch up and recap on some of the other elements of the engagement, even in the months before we had been doing the virtual penetration testing for them, but then that week we're on site assessing, with great success some of the other courthouses, the judicial branch, other facilities that were in scope. Nowhere along the way do we set off any alarms, just open access to pretty much everything with egregious vulnerabilities. We even ran into the state trooper while we were working on a door at the judicial branch. And then he Just kind of chatted and joked with us, went about his way, told him we're working on contract, just kind of business as usual.
So, earlier that night, we'll start on the 10th, since that's when kind of everything started and went down. We're working in another facility, got in using similar vulnerabilities, and then went over to the Dallas County Courthouse sometime probably around 11 pm on the night of the 10th.
And even before we had to employ that cutting board, just due to the positive air pressure in the building, we walked up, and that door was actually unlocked. So, you just pulled on it and opened right up. So that was the start of everything.
Once we got in, we kind of knew for all the places we tested, this was going to be the first place that had an alarm that would likely go off. It did. And then we sat around, waiting for the responding officers to show up as we kind of continued our work throughout the courthouse, just finding other vulnerabilities, getting a vantage point.
And then once officers showed up, took a little while. There's probably about 20 minutes there while we're trying to establish contact, wondering why they aren't in the building yet. And then find out just some funny things afterwards about the security.
But we made contact with them, went down to the responding officers, know, kind of verified and went through the gambit for however long it was, 20 or 30 minutes. They verified us, talked to our point of contact and said, you guys are free to go. And that's when the Sheriff (then-Dallas County Iowa Sheriff Chad Leonard) showed up and everything changed.
Kelly Jackson Higgins
Gary, I know that there's something called sort of like a “get out of jail free card” is the term for it, but something that you show to prove, okay, we're here legitimately, you know, showing, proving who you were. How did that go when the first officers came in? It sounds like Justin was saying things were kind of going okay. What happened at that point?
Gary De Mercurio, former Coalfire pen tester, current founder, Kaiju Security
Yeah, it was going fine. It worked as intended, which was just to show them why we were on site, what we were doing, the name of the company, our names, which of course they verified by looking at our identification and then our contacts’ names and numbers, which they called and they verified that we were working for iowastategov slash courts, I believe is what it was.
And yeah, from there it just it went really well. The entire interaction was very professional They just said, “Sit tight. Let us let us verify who you are. This is kind of strange It's the middle of the night.” And I made a joke something like, “you know, of course, It's harder to break into a place when there's a lot of people in the in the facility during the day.” But yeah, for the most part everything went really really well. It was very professional, and the card worked just as its advertised, which is just to let them know what you're doing and the information that they need to verify that you are who you are and you're doing what you're supposed to be doing.
Kelly Jackson Higgins
I vaguely remember you guys talking, Justin, about what you had done before the police, before the deputies came. You had gotten through some of the actual pen testing, not physical, but regular digital pen testing process. Can you talk a little bit about how far you got there and some of the things that you had found at that point before they came?
Justin Wynn
Officers responded quite quickly. I think the police department is literally across the street and then there's just the audible alarm just blaring throughout downtown. So didn't have too much time. we did find some other security vulnerabilities, obviously don't want to go in and jeopardize and disclose things, that we shouldn't that weren't already made public. And it's kind of Barbara Streisand effect. A lot of this we're able to discuss because they brought so much notoriety into the case and a lot of the documents became public.
But yeah, found some other vulns. I think we ended up on the third floor of the courthouse, in a courtroom. And then we had a good vantage point overlooking that we could see officers arriving to the scene at that point, kind of put things on pause until we could make contact with them.
Kelly Jackson Higgins
So pretty much everything was going as planned, like a typical engagement. So, then things got a little more heated when the sheriff showed up. Take us back to that scenario.
Justin Wynn
100%. Yep.
Well, it kind of paints a different picture now too, when you can see all the body cam footage, but we, we felt it as soon as he showed up, everyone's demeanor changed reviewing body cam footage. You can hear some of the officers saying, “This ought to be good.” And then turn off their body camera as soon as the Sheriff shows up. So, just a lot of implications there, I suppose you read into it, but, he showed up, already kind of immediately irate with the situation.
You know, spouting off cursing that “the state can't do this.” You know, he needs to be notified. This is his courthouse. and just kind of the jurisdictional issues that come up thereafter. Gary, you to add more color to it?
Gary De Mercurio
They got uncomfortable. I think he said something in effect of “Don't you feel kind of stupid now?” When he asked, he said, “Don't you know that the county owns this building?” And I think we said something to the effect of, “No sir, we were hired by the state. We naturally assumed that the state and the court had jurisdiction over a courthouse.” And he said, “Well, they don't. So I bet you feel pretty stupid now, don't you boy,” Something to that effect, if not, if not verbatim.
And he walked away and said, he said something, the effect of, “Hold them, they're not going anywhere.” And he walked away. Oddly enough though, when he walked away, even after that engagement with him, the deputies were still professional. was like, everybody's like, “Okay, yeah, well, that's just the Sheriff, that's how he is.” And as soon as we walked away, we just continued to talk to them about how to better secure their courthouse, to figure out why the door was open, to figure out why they weren't able to get into the courthouse even though we had gotten into the courthouse.
We were answering questions. It was just basically another day for us even after the Sheriff had shown up, which made it all the more bothersome when he came back and obviously told them to and told them to arrest us.
Becky Bracken
So what explanation, Gary, were you given about why you were being detained and eventually arrested? Or were you given any? Seems like things abruptly changed. What was the legal explanation for that?
Gary De Mercurio
The legal explanation from the Sheriff was the state doesn't own this courthouse. It wasn't that we were burglarizing the courthouse. It wasn't that he didn't believe that we weren't supposed to be there. It wasn't that we had done anything wrong. His reasoning was simply that the state doesn't own this courthouse. You can't be here, which to me isn't burglary. And that was one of the things that always bothered us.
We're not law enforcement, we're not lawyers, but because of what we do, we at least have a rudimentary knowledge of what burglary is and burglary still requires intent. And they had the contract, they knew we were there, they knew that we weren't trying to burglarize the facility, that we weren't there to actually steal anything, that we were just two guys performing our jobs under contract. Now, whether or not the state and more specifically the court was able to give us access to that facility, that's a different debate. If that would have been truly what we were in trouble for, that would have been something different. But that night they charged us for burglary, which means once you entered that building, you have intent to commit another felony, which was not the case.
Becky Bracken
So, what appeared to be a simple administrative snafu resulted in your being arrested and held for how long, Justin?
Justin Wynn
I think it was 20 hours before bail was posted and we were out the next day. You know, in between we're dealing with quite a lot inside the jail cells with the officers trying to make phone calls. We were arraigned in the morning with the Magistrate. So, you know, quite a lot of action packed in those 20 hours.
Becky Bracken
So, then Tom, that's where you come in, correct? This is where you have to come up with the bail and come up with a game plan to get your guys out. Is that sort of how you received the news?
Tom McAndrew
Yeah, I think it's not every day does the CEO get a call saying that you got to figure out how to bail employees out for doing their job. Much like Gary and Justin said, when I got the first call, I think it was like six in the morning. And I got the call that the two of them were, for lack of a better for term, caught and were in jail. My initial reaction was I laughed because I knew both of them. I knew about the engagement that we had going on. We kind of expect the unexpected in this. And so, when we do these engagements, everyone reacts a little bit differently, but usually that's resolved in a couple of hours.
And so, by around 10 or so my time, which is around noon their time, when I heard that things were not resolved at that time, that was really when then I started kind of kicking into high gear. We started to understand what was going on. And this one's unique because we've done 10,000-plus of these or so before, but we've never had any engagement that even today, Coalfire has never been part of this. All the charges and everything are actually against Gary and Justin as individuals, not anything against the company. So, it was kind of a weird thing where we said, well, we get our lawyers, we'll protect ourselves, we'll fight this, but we're not in anything. We actually had to do it to kind of save the employees, which really, which is a little detail that I don't think people really understand.
Becky Bracken
So then litigation ensues. You all are out and so you have to then make your way from jail back home somehow.
How did that work, Gary?
Gary De Mercurio
Tom said, “I don't care how you get out of there, just get out of there first class, whatever you need to do, Greyhound bus.” I agreed 100% and we got out of there as soon as we possibly could to get away from the situation. But not before having the worst pizza in the world. I will say that. If you want to know where the worst pizza ever is, let us know and it is in downtown.
It was pretty simple. I was gone the next day. I think Justin actually had a later flight just because he lives in a little town, Naples, Florida. So, it's a little bit harder for him to get there. We were out there I think I was out the next morning. I don't even think I lasted 12 hours
Tom McAndrew
Yeah, I'll just add maybe onto that one nuance of having bond, having lawyers getting that stuff in. was really, that was something that we had to do quickly. I've never done that. I've never gone out and posted bond through a company. We didn't know what these things cost.
This is also a little town. And so, some of the questions we had initially of what's the best way to tackle this, right? Do we get big, big New York lawyers to kind of come down and put pressure on or will that backfire because you know, they're in a small town in Iowa and it's better to kind of connect locally. So those are kind of the things on the back end that you know, we were trying to figure like how do we even, how do we post bond? What do we do for this stuff? And all that.
Kelly Jackson Higgins
So, I think sort of the big thing to look at too is obviously, so firstly, this was a professional, you know, situation, but also affected you all personally too. Talk a little bit about sort of the fallout, like how you, you know, probably processed it. You probably were shocked that you were being walked to jail from a job you were actually doing legally. And then you had to deal with the overnight, you know, trying to get out of jail, trying to get all the legal stuff set up. What was kind of the aftermath like for you the first 24 hours?
Justin Wynn
First 24 hours, probably not as bad as it's been in the last five years since. Gary's got a lot of fallout that he can discuss and you know, I'm very mindful of it and it kind of affects my day to day, I approach career and everything else.
But those initial hours, I don't think the full gravity had set in until we were arraigned in the morning. And it was clear that the Magistrate, the person who actually formally presses the charge against us, wasn't keyed in on the situation. So, the Sheriff hadn't talked to her and said, “Hey, you know, I did verify called that the state something's fishy. You know, we'd still like to press charges,” nothing like that. I mean, she was set up in front of against us saying, “Hey, these guys were arrested burglarizing the courthouse last night.” And then she reads the documents as well in this courthouse and things got a little bit more personal from there.
And then, you know, the input from the county attorney saying these guys are a flight risk, who was working very closely with the Sheriff and he knew the situation. So, then they opted to heighten our bail, heighten the charges, everything they could.
So at that point, yeah, then we're facing seven years of felony prison time. Again, all in the first 24 hours. And at that point, you still assume, you know, the company is going to take care of this. Things will be, you know, resolved pretty timely and without incident. And then, you know, the months started dragging on and things certainly change, but Gary, were the first 24 hours for you?
Gary De Mercurio
Aside from the pizza, they are all right, I guess. Yeah, it was, I think the only thing that was rather nice, Tom, I can't say enough about Tom. He handled it about as well as I think anybody could.
Tom McAndrew
I'll get you some free pizza, Gary.
Gary De Mercurio
Just not from that place. Tom and I had known each other since you were a director, I think, when I came in as an associate. And so, I would like to think that Tom knew that I wasn't a complete jackass and I wouldn't be breaking into places I wasn't supposed to be breaking into. So, I don't know if that had anything to do with Tom's help or not, but I'd like to think that it does. But the whole thing was handled, was handled really, really well.
I was a little upset and I think I went into his office later. I was like, “What the hell took you so long to bail us out?” I like shut the door, you know, and I was like, “With all due respect, what the hell is your problem? Then he explained everything. I was like, “Okay, I get it. I understand. Yeah.”
Tom McAndrew
I tried the best I could.
Yeah, I will say that is one thing that this popped up, you know, and to your guys' credit, you know, when it did pop up, kind of by the noon, we knew it was serious. The first thing we did go through is go through all the contracts and everything, because, you know, when these things pop up, you have rules.
You know what you're supposed to do, but you're always like, is this the one where we didn't dot our I's and cross our T's? Is this the one where we didn't, where we had a verbal instead of an email? And like, we're, know, on one hand, think that Coalfire, we were very lucky that we had Gary and Justin, because had there been others with less experience or just others that didn't have as good of a day, it would have been very different in the outcome.
And you kind of see that from all the everything that got posted, is why this bothers me so much. I mean, when people make mistakes, we get it. But in this case—luck, skill, whatever—the two of them did really everything by the letter of law.
And I think when people in power started realizing it, what really bothered me is they started spinning things politically into messaging. And if it wasn't this, then this, then this, then this. And that was something that even today, when I was going back and looking at some of the news articles, you see this slant on it that two people arrested for burglary. And that's real. That's the sensationalism of this. The background of two people hired to do what they were going to do got in trouble is the big issue that I’m worried about.
And even today, I was re-looking up the apology from the Supreme Court of Iowa apologizing for this. And they showed the contract on there and they say, “Well, we never expected them to physically break in.”
But even on the screenshot they have of the contract, at the bottom it says, “Do you authorize lockpicking?” And it says, “yes.” But they cut that off halfway and they only highlight the other parts above of the building and the location because it matches their story. And so that's, again, I think the part of this is people in power not doing the right thing is the thing that really bothers me.
Becky Bracken
How long did you have to live with this? I can only imagine the feeling of I didn't do anything wrong and yet I am still, it must be a very powerless and anxiety inducing feeling. How long did this drag on? Where does it stand now? How long did it take you to get some sort of resolution?
Justin Wynn
About five years. Honestly still dealing with it just about every day that we're in the public light and talking with people and having to set this story straight.
Because while this all went down, our lawyers were advising us that you guys are totally in the right, but the way things work for the law, you can't make statements, you can't go public. And meanwhile, the state's just putting out all this misinformation. The Sheriff's setting up meetings, responding to people, putting out all this misinformation. So, the narrative, if you read it in real time online, looks like we were out of scope. We did things wrong. Which can’t be further from the truth.
And still today, there are people that we set straight on that story, let alone all the personal ramifications with employment now that we have an arrest record. It's extremely frustrating. I mean, still today, it hasn't ended for us.
Gary De Mercurio
More specifically, they waited until literally the last day to drop charges, which I think is one day short of six months, I believe is what it was. And I think I had to wait one less day because my charges were dropped. But they didn't drop Justin's. And I think we were talking to Brian Krebs at the time, I think it's on video, where they actually dropped the charges for Justin finally. And we were like, okay. But they just drug everything out.
The dropping of the charges was this huge ridiculous rigmarole where the Sheriff, and please correct me if I'm wrong because I never was part of that conversation, Tom, but he, I remember having a conversation with the three of us, Tom and Justin and I, where the Sheriff was like, “We'll drop charges, but I want to talk to Tom.” And so, remember we're like, “This was against us, not Coalifre. So why on earth would a Sheriff need to exact his pound of flesh from our CEO when it has nothing to do with Coalfire?”
They were never brought up on any charges. Nobody was ever going after them for anything. It was just Justin and Gary, but it was almost like he was holding his hostage. Power play, I don't know what it was. I'm gonna talk to Tom before we drop our charges. And we tried to tell Tom at the time, like, “Don't talk to him. Don't give him the satisfaction.” I think Tom's reply was something like, “Who cares? Let him yell at me as long as we get the charges dropped. Let's go for it.”
Tom McAndrew
Yeah, think that so I think to your point, right, that in the system, because this was against you guys and wasn't Coalfire, right? What I kind of told everybody at Coalfire, because everybody wanted to come to their defense, and even others around the security committee really wanted to jump in. But I said, “Hey, it's less about the public appearance and more about like getting the charges dropped and make sure these guys don't go to jail.” So that was kind of priority number one.
But there was that point when nothing was really against Coalfire, right. And we take kind of our NDAs and, you know, our job is basically to find the problems with all these companies and to figure out how bad guys can cut into and make recommendations. So, we take that very seriously. And so, I had known kind of all the facts of exactly what had happened for a while. And we sat on it. And then finally it was that one day when I realized, think they had just, you guys had higher burglary charges, and they moved them down. So, you still had the charges, but they just downgraded them.
And at that point, that's when I kind of got upset. So, I wrote a, I remember it was like 11 o 'clock at night. I was just mad, and I wrote this like manifesto just saying, here's what they did. They did all the right things. Here's all the arguments for hearing the media. Here's why it's all wrong. People are assuming you don't have a letter. You see, this is why you guys have to have this in contract. And I was like, this is, they just didn't understand the situation and the nuances of it. And that, once that became public, people understood the difference.
Yeah, we think of government as one thing, but know, state, local, they don't necessarily like each other. Small towns, big towns, you know, they don't like each other. Law enforcement, know, judicial, administrative branches…. And the people that have actually spent time looking at the contracts and what we did and everything, that's the scary part, is you walk away today and say, this exact same thing can still happen to anyone today, no matter what. And it doesn't matter what the company does. On paper, on anything else, that risk still exists today for every single person that does any of these sorts of engagements.
You have to realize you are on your own doing this. Hopefully you'll have a company that'll support you, but even if they do, it's like I said, you guys are charged with burglary, not as Coalfire employees and not for doing work. And they don't really care about what's on the paperwork.
So maybe the analogies you think about it, digitally, people give us IP addresses and websites and every once in a while, people will mess up and they give us the wrong IP addresses or websites and we go after those, and we have to respond to them.
In this case, we do a lot of data center assessments and stuff and a lot of times when people give us those, we're assuming those are the right data centers, but just like in this case, a lot of those data centers are actually owned by other corporations you may not know. So just because the company you're working with specifically tells you, yes, you can go into these places, whether they're physical or digital, does not mean that they necessarily have the authority and it's very difficult to understand whether they do or they don't before you're doing the engagements.
Kelly Jackson Higgins
This is a great segue to the question I wanted to ask you both five years later, how you're approaching these engagements now differently, what you're doing differently, what maybe sometimes gives you pause before you go into one.
Gary De Mercurio
I think immediately when we got back into the physical pen testing, we'll tell a war story here, I guess, but, but we started contacting the local police and the sheriff's department both. And then we had the client contact the local police and the sheriff’s department both just so there wouldn't be some another overzealous sheriff or something saying, “Well, well, you know, we didn't hear from the company. Maybe these guys are not on the up and up and we should go check that out anyway.”
And we didn't want him to get in another situation like that. So, we had the client call and then we would call. And it was probably our first or second engagement, we were doing a very large red team and we did that. We called the client called; we had verification before we went on site. We called the local police department, the sheriff’s department said, we're about to go on site. We just want to make sure that your dispatch understands that if you get a call that there will be security professionals on site testing the building. So please don't come in guns blazing.
And sure enough, we were on site. We set an alarm off. We were actually on the phone with our contact trying ridiculous things to see if we can bypass the alarm. I think we even bought a shower curtain, like on the Karate Kid with the hoop and everything.
And we were trying to get past some of the rec sensors, it wasn't a rec sensor, but a motion sensor. And we set the alarm off and he's like, “Yeah, okay, yeah, the alarm went off. You've got about X amount of time before it'll call the plant manager.”
And sure enough, a police officer showed up. It was his first response to a burglary. He had his weapon drawn and he was shaking. Luckily the plant manager was there with him, and she knew that there was some testing.
And she's like, “Son, you're going to have to put that weapon away. Like we, we may have some security professionals here and I don't want you to shoot them.” And he's like, “I'm so sorry. It's just my, it's just my first response. I'm, a little bit nervous.”
So even after we take all those precautions, and even though we have changed the way that we do things, you still have issues where information still isn't passed out. And I, and I believe, correct me if I'm wrong, Justin, but I think that, I think that Coalfire records most of those calls now. So, we don't have the issue like we did before.
Justin Wynn
That was kind of the big thing.
All the stuff was on the phone. Yes, we want you to break in overnight. Here are the addresses. Just so explicit that there could be no confusion about what we were there to do. And that's where they had leeway in the public lakes wasn't in the documentation. We didn't expect these guys to break in. So now that's, that's one thing we've changed, but kind of going back to Tom's point, I find it an interesting question. What do you guys do different now when it never should have happened in the first place? And I think our, and I hope our situation is very unique and a one -off because there was a bad egg in law enforcement.
But we had, you know, two prior encounters both with the state trooper and then the responding officers who verified and let us go like happens on every other engagement where we get confronted. There are certainly things that we've adjusted and buttoned up. Some of the documentation I thought was a little outdated and we've done better there, but really in the first place never should have never should have taken place.
Becky Bracken
I'm interested in picking up on something that you all were talking about and specifically about the reaction from the cybersecurity community. Tom mentioned it earlier that everybody was sort of rallying around your cause, but it sort of took a minute for that to kick in. Tom, can you walk us through a little bit about what the initial sort of schadenfreude felt like?
Tom McAndrew
Yeah, it totally sucked. We would deal with this all the time. We do audits and certifications and so, and maybe less so now, but five years ago, 10 years ago, anytime there was a breach, or something happened, you'd want to know who was part of that because they were part of the problem. If you guys audited that or you sign off that certification or someone did the pen test and application was breached, obviously the pen tester didn't find that issue and they should.
So I'd say the community used to be, and maybe newer folks, used to be very reactive when they see something bad happening. They're assuming that they were bad people that made mistakes. I think now with this assume breach and assume that you don't know the details, the community, or at least I think the part of the community that most of us respect and the most, those are saying, “Hey, we never know all the details of everything. And instead, we assume that CISOs are doing their best. Security consultants are doing their best. And let's kind of sit back and look.”
Like these are the questions you should ask. Was it in scope? Is it documented? Did they do it? These sorts of things. But unless you have firsthand knowledge, you really shouldn't comment on them. So, I think that's shifted a lot over the last several, maybe five years, maybe over the last decade. And the second part is then how do organizations react? Like I said, the comment I have that I made public on Iowa was that in 10 ,000 plus engagements, only one time have they ever publicly posted something like that. So it's very unusual.
So, the time when something like this does happen that's unusual or unique, it’s a chance for leaders to step up. Like I get it. I get its initial reaction. I get the initial court judge and her reaction. And I get the initial reaction, but once you have the big picture as a leader, you need to come back and be public and say, “Hey, I messed up” or, “I didn't have all the right information.” And this should happen and that's really what didn't happen here.
I mean, we had like an employee that started last year. I remember he said, “Hey, Tom, just so know, like the only reason I interviewed here is because of how you guys treated those folks.” And for me, it was kind of heartwarming to know that, but also to know that still today, it is one thing that a lot of folks really know about this onto it. And it's a good story, right? You get to hear, you get to learn a lot of things.
But the part that makes it sad to me is there's a lot of leadership failures out there in the greater community that could have done this. And ultimately, the two people still have lifelong repercussions of this.
There's still a wrong in this world that has not been righted. And I don't see that there is a path where that's likely to be cleaned up. And that's why you said, well, how long has it been? It's been five years. How long will it go? I don't know. To the best of my knowledge, Justin, Gary, there is no 12 more months and records are all clean or anything, right? It's still, there is no path.
Kelly Jackson Higgins
So does this kind of mean that the physical penetration testing job or profession is just not quite the same now? Because can you do the things you could do before with the element of surprise, for example? But you can't do a lot of that now. The element of risk on your profession, your professional and personal lives. I'd love to get your thoughts on that, Gary, sort of where you see this field. Like is it going to change forever now, do you think?
Gary De Mercurio
I don't know that it's changed necessarily in too much of a negative way. I think it was very unique in that we were trying to test courthouses for the court system, but the county is responsible for the security of said court system. There's some gray area there. We still should have been able to be on site because we were hired by the courts and the courts are the one that run the courthouse, right?
So again, I think there was a whole lot of politicking going on there. In general, though, I don't think it's changed too much because if a private company hires us to test private property, there's not really much the police can say outside of that unless they just don't believe us. But again, you're arresting somebody that would be the equivalent of a contractor working on private land and being an officer and showing up.
So, we haven't had problems in the past and we haven't had any problems from that point going forward other than some of the close calls. So, I don't think it's changed too much.
I think it does highlight, however, people's fears of allowing a red team to go through their facility, especially when they hear stories like this or they remember a story like this, which is where, I think Justin was the one that came up with it was the, we called the whitelist walkthrough, where we perform the same function as the red team, just not testing the people, the policies and the procedures because we're not doing it real time. We're just testing the physical infrastructure of the facility and then going over their policies and procedures to make sure that they're correct if people would follow them the right way.
I think so with the advent of that, making people feel a little bit better rather than having two guys breaking in the middle of the night. I hope it doesn't change too much because the weaknesses that you find when you're doing a physical penetration test are glaring and astounding. Come up with any big word that you want to think of that shows extreme value in that situation.
There are still vulnerabilities that we found four years later that we reported on to one of their facilities that are still there. And when we walked by they're still there where somebody could just walk in the facility, kind of like we did with the air pressure in the front when we closed it for the courthouse that we were arrested in. Some of those vulnerabilities are still around because everybody was so worried about whether we should be in the facility, they completely forgot that maybe we should do something about all these things that they found.
Tom McAndrew
I mean, I think it's particularly important for state and local government, because I was actually going into my mind, all the states and counties and cities I've done, and most of corporate America, they outsource their data centers. when we look at what we have access to, physical, logical, administrative controls into it, a lot of governments, and you see it, right? Like you go to the DMV, it's not the world's best facility there. I was just at the post office here over the weekend. I couldn't even tell if the building was open, I wasn't totally sure if it was.
So a lot of times you have government and civil servants that are doing the best they can and they're stuck with very old buildings that can't be retrofitted securely. And then someone decides for money or for whatever reason, they're going to stick, you know, Active Directory servers or sensitive things into these facilities. I'd say normal corporations generally don't make those, but it's very prevalent in state local government.
And one of the biggest benefits of these sorts of tests is you understand it and it can really help drive funding and fixing things. That's been a big part that you'll do these long engagements, and you'll show all these technical risks and everything, but it just goes over everybody's head. But you show them you can break into a door, or you show them that anybody can just go in here, or we had one where a homeless person was actually sleeping in a data center one time. Those things get funding allocated immediately.
One of the negatives of when you pull this out, it becomes very difficult for state, local, CISO security folks that know they have glaring physical holes to be able to show that to management to say this is why it's so important. It's just not the same as doing these, you know, whitelist walkthroughs. doesn't have the same impact.
Gary De Mercurio
And I want to piggyback on his piggyback, which is there's the three-legged security stool, right, where you have logical security, physical security, and then the human element, usually through some sort of social engineering or something into that effect. And it doesn't matter how good it is, it’s almost like the stool has to be able to balance on three legs. If you take away one leg, it'll balance for a while, but eventually it's going to fall down.
When people neglect physical security or they neglect the human aspect and they focus everything on their external logical security, it doesn't matter how good it is. If Justin and I can stick a stick through your door and wave it, which Justin has done multiple times, and open the door, and now we can walk into your facility and plug into one of your network jacks, it doesn't matter how good your external security is.
And that's one of the things that holistic idea of security that each leg is equally as important because they help the other one's balance. And by taking this away, you take away one of those legs of that stool. Like Tom said, (we’ve found) homeless people sleeping in a server room. It's that easy. People just walk in and without it, you don't know where those vulnerabilities are. And unfortunately, organizations, whether it's local, state, or even companies, don't test for those vulnerabilities and it makes them exceedingly insecure.
Justin Wynn
Well, apparently that was a popular question, Kelly, because I also wanted to piggyback and tack on to that. But I know you also asked about whitelist, but let me, let me hit that real quick because you asked, “Did this damage our industry or has it had me, has it made things more difficult?”
I think overall it's certainly improved things because we've had conversations and discussions about it. There's been proactive industry movements like Awareness-Con, that was held, in the town that arrested us, focused on just discussing this work that we do and how important it is. Documents were released by TrustedSec that show this is the letter of authorization, here's the terminology and legal descriptions that we use to approve this type of work. And it also highlights the need of transcended industries, right? We usually operate in such a vacuum or a silo and nobody knows that red teaming and breaking in the buildings is a thing. And now pretty much the entire state of Iowa knows, and quite a few people across the world.
But look at how we got into these places and how trivial it was. There are glaring security issues that were never tackled to date for the decades that these buildings were in operation until red team was hired to do proactive offensive security testing. And I think that's the main takeaway for us is how important this is.
And I was, I was a little different. So their reaction was a little bit negative. I think the industry improved, but Iowa did two funny things afterwards.
One: To appease the politics at play, they made the most boneheaded knee -jerk reaction they could make and said, we are never going to do this type of testing again. And now they are permanently in a degraded state of affairs.
All their buildings, like Gary said, we come on site and just walking by, you can see doors held open after hours because of the air pressure. And that stuff's not going to be proactively tested. They said no afterhours testing. And look how important it is. And until they change that, they are still going to be in a giant hole and kind of set back from the rest of the industry.
A lot of the hackers, the community that rallied around us said we're pulling out of Iowa. We're not going to do jobs there if this is how they're treating us. And so I think once, other parties at play realized that, Iowa was the second state after California, which tells you how progressive this is, that introduced a statewide VDP, a vulnerability disclosure program. So they did kind of come back and announce to the hackers—and this was months after our arrest and all this case so I'm sure it had something to do with it—but they basically made a statement to the community saying, “Hey, we recognize the importance of this. You know, things probably there weren't handled great, but now we have a statewide VDP, second in the nation. If you find vulnerabilities on Iowa systems, please let us know.”
So that was one kind of positive change, but they still have a long ways to go. So now we can talk about whitelist if you want to get into it.
Kelly Jackson Higgins
Yeah, tell us a little bit about the whitelist.
Justin Wynn
Okay, so there's generally two different types of engagements. The offensive, which was Iowa, which is what we're all familiar with, and it can encompass social engineering, covert methods of entry. After this, we've unfurled a new offering called, “physical walkthroughs” or “whitelist walkthroughs,” which is fully comprehensive. So, some of the trade-offs, we're not getting to test live responses. We're not seeing if people can be tricked into letting us in.
Targets of opportunity while we're there on site and really demonstrating the impact that Tom was talking about that man when you see that in a report that somebody can walk in your data center, that's pretty impactful. But the whitelist lets us come in and do a comprehensive audit; so, a full 360, we test every door, every security appliance, we can sit down at the computers with them review the configurations. So, it's a lot deeper touch and a lot safer. So, it lets multiple consultants come on site. You don't have to worry about jurisdiction multi-tenant office buildings are notorious for this, you know, who do we need to notify? Who do we get buy -in on this? So, it's enabled a lot more testing. It's comprehensive.
Some trade -offs that you don't get with that red team element, but it's been hugely successful and something that a lot of our clients are opting for nowadays.
Gary De Mercurio
There's another thing that we implemented too that we did on one of our red teams, which was to invite a representative from the client with us. And so, they were actually on site going through us while we were doing the red team portion of it. We would explain what we were thinking, why we were doing it, the thought process. And that actually worked out. If there is a client there that is willing to do that and has the experience needed to understand what it is that we're doing and how it affects their facility, that actually works out phenomenal.
The gentleman that we had with us was outstanding. At one point we had taken one of their yard semi-trucks—we hotwired it, right? And then we moved the semi-truck, backed it up to one of the trailers on their loading dock and moved the trailer so we could try to get into one of their warehouses without breaking into anything because the doors were actually pretty secure.
And he was just like never in a million years would I have ever thought that that was something that we needed to protect. He's like, never would have even entered my mind. So when you bring them on site with you, that's like that middle step between a full red team and that that assessment is bringing them with you and having more of that purple team approach where they can actually get to see what we do, but they more importantly, they get to see how we think and the things that's going through our mind when we're trying to enter a facility.
Becky Bracken
Unfortunately, we have reached the end of our time together. For our audience who wants to learn more about your work or find you, Justin, are still doing penetration testing with Coalfire, correct?
Justin Wynn
Correct. Yeah, scared to apply elsewhere, but no great company can reach me here or red team wins is my handle.
Becky Bracken
Gary, how about you? Where can we find you?
Gary De Mercurio
Well, nobody will hire me anymore because of my arrest record. So, I had to start my own company. I'm at a company called Kaiju Security, which is which is my own.
Becky Bracken
And Tom, course, CEO of Coalfire. Thank you so much for your time today. We very much appreciate it and for taking us back to those dark days five years ago. I learned a lot and I know our audience did as well. So thank you.
Tom McAndrew
Thank you so much for having us.
Kelly Jackson Higgins
Yeah, thank you.
Becky Bracken
I want to thank you again Justin and Gary. Thank you to our audience and to Kelly Jackson Higgins, Dark Reading's editor-in-chief. We appreciate everybody joining our conversation today.
That's been Dark Reading Confidential, a podcast from the editors of Dark Reading. We'll see you next time.