Michael Vance [00:00:06]:
There are some experienced hackers that still like to do that sort of thing just to kind of give back. It's also one of the ways that new hackers on those platforms can establish themselves, get some reputation points, and get invited to some of the actual cash paying bounty programs.
Chris Lindsey [00:00:20]:
Right. And you got to start somewhere. And the nice thing about it is, when you have somebody that's wanting to get into hacking, this is a great avenue for them to have a chance to legally do this without getting in trouble. You've set the boundaries, you have a nice program, and what you're talking about is spot on. Perfect, because really creating that channel initially of, hey, we have a finding. How do I get it to you?
Michael Vance [00:00:44]:
Right?
Chris Lindsey [00:00:45]:
Because in the past, a lot of companies would get upset if you come to them and say, look, I have a find. And the first thing is, wait, what do we do with this? And wait, why are you hacking us?
Michael Vance [00:00:56]:
Right. And let's be honest, how many of us actually have the technically required email address of security domain name going someplace where it's actually monitored all the time?
Chris Lindsey [00:01:11]:
Hello, and welcome to Secrets of Appsec champions. My name is Chris Lindsay, and today we're going to be speaking with Michael Vance. Today's conversation is going to be around bounty programs and penetration testing. Michael is the CISO at Navient. Michael, please introduce yourself.
Michael Vance [00:01:28]:
Hi, Chris. Thanks for having me on. Yeah, so I've been CISO at Navient for almost two years now. Prior to that, I was in charge mainly of building security and things, so had responsibility for security architecture and application security. Pretty small team, but we managed to get some things done.
Chris Lindsey [00:01:48]:
Today's conversation is going to be around external services and combining both bounty programs as well as pen testing together to be able to take your program to the next level.
Michael Vance [00:02:00]:
Yeah, and I think that was actually, as I mentioned, we've always had a small team, so it was especially important for us to get outside help. I always envy these when I hear some of my peers talking about having a red team of ten that they can just throw at whatever they want and having really strong in house hacking resources. We haven't had that. So there's always been some struggles to get good talent. Of course, I think everybody does that, says that budgets are tight. Everybody talks about how they don't have enough appsec talent. That's a subject we could do off on a whole nother podcast if we want to. But, yeah, the way we've addressed that is by getting outside talent to do it for hire.
Michael Vance [00:02:40]:
And that's mainly taken two forms we've had for a number of years in a couple of different flavors. A contract engagement with a well known application security firm. Well, for the moment, shall remain nameless. We'll see to do. However, what do you want to call them? Ethical hacks or application pen testing. An applications focused test. A particular application to hack into it and find the things that are wrong. Combination of automated with manual follow up dynamic testing.
Michael Vance [00:03:08]:
And then we also had an interesting evolution as well with our bug bounty program that I think is actually a pretty good story and I hope one that not very many others have on how you get to a full fledged bug bounty program.
Chris Lindsey [00:03:25]:
Yes. Well, you know, in the value that bug bounty programs bring to you. I mean, the benefits are numerous because really, when you start thinking about what a bug bounty program does for you, it gives you the ability to legally tell the world, okay, I'm going to set up this environment, come attack it, see what you can do, and help us to be better. Because when you start actually doing something like a bug bounty program, really, that shows the world you're trying to be better stewards of the data and with your applications, but it also gives you the chance to say, look, we feel like we've done our best effort. We can only go so far internally. We can only go so far externally. Let's see what people can do. Because you have these twelve year olds and 14 year old kids, or even professionals who do this just kind of on the side for fun to see what they can do.
Michael Vance [00:04:12]:
Yeah, and there's always some fear of, are we sharing too much? Are some of these people gray hats? Are some of them just flat out black hats in disguise? And I think there's always a risk of that. But that's why you don't run your own bug bounty program entirely. That's why you hire somebody like Bugcrowd or Hackerone or is there anybody else, or are those pretty much the only two in the space? They're certainly the big two.
Chris Lindsey [00:04:36]:
Yeah, those are the big two. I mean, there's a few other ones, but those are your two biggest ones.
Michael Vance [00:04:40]:
Yeah, but that's why you hire them to run the program and to manage the people, to vet people, to give them the right incentives to follow the rules. Like I said, the story of how we got to both are almost a little connected. So for many years we were working with an appsec specialty firm to do ethical hacks on our applications, and we were doing that where we paid for a certain number of attacks, a certain number of hacks, and we only got so many, they cost so many per, and of course it's one of those things. We're sure the more you buy, the cheaper the rate gets. Cheaper the unit cost gets. But it was fairly expensive on a per hack basis. But part of the reason we used that model was because with them, that was the only model they had where we could guarantee us citizen and domestically sourced resources for all the hacks. So we took this contract and we converted it to another service offered by the same appsec consultancy, where rather than having a certain number of tests to do, just like, hey, you bought twelve tests for the year, or 15 tests for the year, you bought one testing stream.
Michael Vance [00:05:47]:
Or you could buy as many testing streams as you wanted, but we bought one. And basically the way that one worked was just keep the tests coming, keep them in the queue, and deal is that with having one stream, they would always test up to one thing at a time. We couldn't have them test two things at once, we can only have them test one thing at once. But we could keep it going constantly and as long as we could keep the stream filled. We actually tripled our testing capacity the first year we did it. We tested three times as many applications and kept the stream about 92% filled that first year.
Chris Lindsey [00:06:20]:
I can see that, because the nice thing about having the stream is as you're deploying new, or having new features come out, you could be verifying and validating those as they come out. Now, it's not just, okay, we have three or five or ten specific tests that we've paid for. Now you can sit there and you can move things around in the queue and get things tested quicker and more flexible. I like that idea.
Michael Vance [00:06:43]:
Right. But instead of, so it's not that they're looking at everything all the time, they're still looking at one application at a time. But like I said, as soon as they finish that one, they just start up on the next one that we've got prepped and queued up. So instead of having just that limit on a certain number, number. I mean, if you think about it, even with their most intensive testing protocol, an application pen test would be about two weeks. If you think about that, you do new application every two weeks for a full year. That's 26 applications, whereas before we were only buying twelve for virtually the same amount of money. I think this is our fourth year, if I remember correctly, with the streaming testing service, and we've pretty much maintained that pace of about 90% capacity on the stream and have been able to do a lot more testing.
Michael Vance [00:07:24]:
We need to convince more developers to come over to the dark side. But that was always my pitch. Whenever I was trying to hire anybody. I know we're digressing them, trying to hire a new appsec talent. I pitched it as trying to get developers who were tired of just coding day to day and wanted to get into security. I think that as a general rule, makes for better appsec people than folks who came up through the more traditional network security ranks. So how did that segue into the bug bounty program? So we started off slow. We did what they call a managed vulnerability disclosure program, which is basically, we're not offering bounties, we're not going to give you cash, but we're going to set up an easy communication mechanism for people to report issues they find.
Michael Vance [00:08:10]:
And we're going to do it on a platform that has a pool of hackers available that can still give them some incentive, because if you're a hacker on their programs and you discover something valid and report it, you still get reputation points. Just the non cash incentive of bug bounty programs. It's the dopamine shot of finding something and reporting it and it allows. There are some experienced hackers that still like to do that sort of thing just to kind of give back. It's also one of the ways that new hackers on those platforms can establish themselves, get some reputation points and get invited to some of the actual cash paying bounty programs. Right.
Chris Lindsey [00:08:48]:
And you got to start somewhere. And the nice thing about it is when you have somebody that's wanting to get into hacking, this is a great avenue for them to have a chance to legally do this without getting in trouble.
Michael Vance [00:08:59]:
Right.
Chris Lindsey [00:09:00]:
You've set the boundaries, you have a nice program, and what you're talking about is spot on. Perfect, because really creating that channel initially of, hey, we have a finding, how do I get it to you? Because in the past, a lot of companies would get upset if you come to them and say, look, I have a find. And their first thing is, wait, what do we do with this? And wait, why are you hacking us?
Michael Vance [00:09:23]:
Right. And let's be honest, how many of us actually have the technically required email address of security domain name going someplace where it's actually monitored all the time? That was, in theory, is supposed to be the way that we were supposed to find out about problems that other users found with our Internet presence. Right. But yeah, that theory hasn't scaled. Well, I think those email addresses definitely get more spam than they do anything else. Yes, but yeah. So we went with one of the big providers, did their managed vulnerability disclosure program. And for those who've never done it before, really, this isn't just, hey, you sign and all of a sudden people are reporting to the first, quite honestly, probably couple months.
Michael Vance [00:10:02]:
I think in most cases, in some cases even longer, is creating your policy and defining your scope. This is the rules that you want the hackers to follow, telling them which sites are the ones they should hack. What are the limitations on the types of attacks you want to go? How far do you want them to go? If they find an attack that actually lets them, for example, say an unlimited SQL injection attack, this is where you set the rules to say, okay, do not drop tables, do not download the entire database, download a few records, enough to prove your concept, to prove that it works. But do not suck down my entire database of names, addresses and Social Security numbers, please. Things like that. If you want them to use, if you have Internet exposed, non prod instances of your applications, do you want them to use those instead of the production versions? All those things that you can define in terms and which of your applications, you may not want some of your applications to be in scope for your bug bubbly program for whatever reason, whether that's on a permanent basis or whether that's on a cyclical rotating basis, maybe during some peak business season. You just don't want that going on. You don't want the risk of a hacker accidentally breaking your app during that particular important time of year, time of month time, a quarter, or whatever, right? All those rules you can define in your policy, in your scope, and that does take time.
Michael Vance [00:11:27]:
So when you sign the contract, it's going to be two, three months working with your account team at the platform company and getting that up, getting it published, then they've got to go and they've got to invite people. A vulnerability disclosure program will initially start private, so, meaning that it won't be there for anyone on the platform to see. They'll invite hackers according to a certain profile they may have, like I said, they may do a certain percentage of newbies. They may have a few experienced hackers who have said, yes, I will participate in new vdps in order to help get the program running. So they try to get a cross section and get those invited. And then you just sit back and wait and wait and wait. Well, and especially when you're just doing a VDP where you're not paying, because one of the things you don't need to worry about at that point is what are your bounty levels? And do you have the account funded properly and all of those things? And then really the next question on a managed vulnerability disclosure program is when do I take it public? Right? And there's two ways to take it public. One is kind of a semi public and that is that.
Michael Vance [00:12:37]:
And you can choose to do that. You can choose to do this with the initial rollout, most don't is you actually put a page on your public website that allows people to report a bug through your website. And really all that is, is either a frame or some other convergence over to the platform. But from the browser standpoint it's on your website and you make it easy to find from your homepage. Right. Generally we all have a security and privacy section somewhere. So you click that and there's a reporter bug, and then you get the page that lets people do that. What that'll do is if somebody who isn't already a member isn't already a participant in the platform, that will let them report something.
Michael Vance [00:13:19]:
And if they're not a member, given the option of joining. So it's a little bit of a recruiting tool also for the platform management companies, but it's a way of getting that public in the sense that it's on your website where everybody can see it. And then the next phase is actually going public within the platform. And that goes from that invited pool of hackers to now saying, hey, we've got this new disclosure program that is open to anyone on the platform. So there's a place where hackers can go and they can say, hey, let me look for people who are paying but who still have platforms and they'll get the list of companies and they might get prioritized by stack expertise or whatever algorithms the platform is using to get those in front of people and then they'll go. So we had that, we had just gone public, so we'd gone through developing our policy. We had been private for about eight months or so, seven, eight months, and we had just taken the VDP public when budget for it got cut. So here we've got the program for a year.
Michael Vance [00:14:19]:
We were having some success, we were moving forward and decision was made that we just weren't getting the juice out of it. So kind of let it go. Right. Very reluctantly cut the program.
Chris Lindsey [00:14:30]:
Now my question to you, which I know a lot of people are thinking, which one did you get the better results. Now when you look at the findings and from the criticality, when you look at the bounty program versus a paid penetration testing company. Which one gave you the best juice.
Michael Vance [00:14:47]:
For the squeeze at this point in the story with just doing the VDP without paying bounties, it was still the managed testing service, the managed pen test. And sometimes it's hard to tell when you're doing the same sort of tests repeatedly, year after year, using the same company, even if it is individuals doing the hacking or different, but using the same company. And you're doing that because you've got compliance reasons, right? Everybody familiar with PCI knows you've got to do your annual ethical hacks. When you're doing those and you start seeing a drop off in results, you've got to take a real hard look and ask, okay, is this for a good reason? That is that my developers are doing things better now. They've learned from the last two years of findings, two or three or however many years of findings for as long as we've been doing this, that they're actually getting better and they're not creating as many new bugs or as many new critical bugs or is this a little bit of that? We've kind of reached a plateau with this vendor and their testing methodology and we're just not going to get anything new.
Chris Lindsey [00:15:55]:
Hey guys, as you can see, Mike is in a different spot. We ran into a technical difficulty while we were recording, but we have now since resolved that and we're able to keep going.
Michael Vance [00:16:06]:
If you start seeing findings go down either in quantity or quality, looking for the root causes of that, right. Are your developers just getting better or is the testing firm a little stagnant? And that's when it's a good idea. If you haven't done it by then, that's a good idea to introduce the bug bounty program. And I think that it got to the point where we've gone through, we've done our vulnerability disclosure program and then it got canceled because of budget. And then as luck, maybe we'll see would have it. A couple months later, we got kind of a strange email from. It was me and our director of vulnerability management got an email out of the blue from a guy who says, hey, I just found a bug in one of your Internet facing apps. And I was able to download a whole bunch of information on people with their names, address, Social Security numbers.
Michael Vance [00:17:00]:
I found out about you guys because I'm currently working for your former director of network security. My regular job is working for him, but I also do some white hat hacking on the side. And so he eventually explained when we did get in touch with him that, yeah, he'd found us because there are a couple of websites unaffiliated with the bug bounty program management companies that will still kind of aggregate and publish names of companies that they think have bug bounty programs. So he'd found us through there. When we had our managed VDP, we'd gotten onto this one sites list. When he had started looking at a couple of our apps, he found something, but then he couldn't actually find any way to report it because, of course, our program had been discontinued and we didn't have anything on our public webpage anymore to show it. So he actually did. I mean, he was, in this sense, he was a really good white hat hacker.
Michael Vance [00:17:55]:
He's like, okay, do I have any connections at all with this company? So he, he discovered from LinkedIn. That has just so happened, by total coincidence, that his then current boss was a former employee of ours. And so he contacted him and through him, got a couple of our email addresses and was able to get our attention then with an email. Turns out the problem that he found was very real and took a little bit to fix. We actually had to take the app down for about a week. This was a B two B app, so it didn't completely cripple that line of business, thankfully. Otherwise we probably wouldn't have been able to take it down, but we were able to do that while we got it fixed. Between that and the fact that we had, in that intervening time, we had had a change in senior management.
Michael Vance [00:18:41]:
We've gotten a new CIO. Between this happening and the new CIO, there was now renewed interest in a bug bounty program, because I wonder why? What do we do if this happens again? Or why was it so hard for this guy to tell us about this? We were able to revive the bounty program, this time with backing for going full bounty.
Chris Lindsey [00:19:05]:
Good, good.
Michael Vance [00:19:06]:
Not just VDP, we've had some good success. Another thing that I think is a lot of value with using one of the bug bounty platforms and being able to specify what you want, is that applications that may be hard to find to the general public, you can put them out there and say, here, come at this. And so we were able to find, we got quite a bit of activity within this. We had another ramp up period. We had to go back in and revive our scope. Thankfully, the platform provider that we were working with, they don't actually, unless you explicitly tell them to, they don't actually delete your entire, irrevocably, your program until after, I think it's like two or three years. We'd only had a one year lapse, or actually not even quite a year. I'm remember how long it ended up being.
Michael Vance [00:19:51]:
I don't remember off the top of my head, but bottom line, they were able to simply revive the corpse. And so we were able to get a little of a head start. We were up and running with bounty program. Then in about a month and a half, because our scope was still pretty much intact, we just had to define what our bounty levels were, get the bounty pool, get some money transferred into the bounty pool, which is basically an escrow account, and then once again, sit back and wait for the reports to come in. They came fairly quickly, and they came on some applications that we hadn't necessarily expected, and they found some very different things. One of the things that has always been, one of the big differences in approach to application security testing is, I always kind of liked the terminology on this. The difference between bugs and flaws, bug being the simple bit of code that's wrong, that allows somebody to do, when you're just doing concatenated strings to do your SQL queries versus doing parameterized queries, that's a bug. But flaws get into things like design, things like business logic errors, and other things that you really need to kind of use the application to do.
Michael Vance [00:21:06]:
Now, a dynamic tester should be doing both, but depending on who's doing the testing, what methodology they're using, one or the other may be emphasized more. Right. And a lot of the time, some of the simple testing that a scanner can do, testing for SQL injection or cross site scripting or cookies that aren't encrypted, and things like that are going to be the first thing that somebody.
Chris Lindsey [00:21:29]:
Goes to the low hanging route.
Michael Vance [00:21:30]:
Whereas I think with the bug bounty crowd, they're not doing much of that. They're assuming that if you've got a bug bounty program that you've probably done most of the basic hygiene. Yeah, they'll still throw something out there if they find it. Don't get me wrong, because anything for another couple hundred bucks, what is the.
Chris Lindsey [00:21:47]:
Best security advice you've ever been given?
Michael Vance [00:21:50]:
Wow. Best actual security advice or the best advice as a security practitioner? Either way, because those are two different things.
Chris Lindsey [00:21:57]:
Either way, the best advice actually, I.
Michael Vance [00:21:59]:
Mean, let me just touch on that. I mean, those really are two different things. Because if you're talking about the best security advice, best advice to secure things, I don't even know where I'd start because the answer to that is going to vary depending on what the problem is, what the situation is. What you're being asked to secure. But I think that that kind of leads into the best advice as a security practitioner, and that's that always ask the questions to make sure that you completely understand the problem you're being asked to solve. Because non security folks are not always good at defining exactly what the security problem is or the right security problem. Right. So making sure you fully understand the issue.
Michael Vance [00:22:43]:
And that means not just asking the technical questions, that means understanding the business behind it, too. Right. Because the part of the CIA triad that everybody always forgets about availability.
Chris Lindsey [00:22:55]:
Yeah. Well, everybody learned with Crowdstrike.
Michael Vance [00:23:01]:
So you may come up with a great solution, but that great solution involves having to take down the core business app for some sort of testing at 05:00 and the business actually runs until 08:00 well, and that's an oversimple example of the things that can go wrong. But, no, I think that's probably it is. Don't be afraid to ask questions.
Chris Lindsey [00:23:20]:
No, that's great advice. That's great advice. Mike, I appreciate your time for coming on today, and thank you so much for being here.
Michael Vance [00:23:27]:
All right, thanks for having me, Chris. Bye.
Chris Lindsey [00:23:32]:
Thank you so much for joining me on this episode of Secrets of Appsec Champions. If you found this valuable, hit that subscribe button on Apple Podcasts, Spotify, or wherever you get your podcast. And hey, ratings and reviews are like gold Force. So if you're feeling generous, please leave a kind word. It helps others discover our show. Until next time, take care.
