Your First 90 Days in a New AppSec Role

📋 Show Notes
Secrets of AppSec Champions: Laying the Foundation of Application Security

In the inaugural episode of the multi-part series 'Decoding Application Security,' host Chris Lindsey and guest Anthony Israel-Davis, Product Security Manager at Fortra, dive into the fundamentals of building a successful application security program for large teams. They discuss essential first steps when starting at a new company, the importance of understanding the company culture, and the critical role of security champions. The conversation covers various aspects of application security, including the implementation of SCA, SAST, and DAST tools, the nuances of API and container security, and the importance of building strong relationships with developers and QA teams. Ultimately, the episode emphasizes the incremental and strategic approach necessary for managing and mitigating risks effectively in a complex software development environment.

❇️ Key Topics with Timestamps
00:00 Introduction to Software Building
 
00:59 Meet the Expert: Anthony Israel Davis
 
01:08 First Steps in a New Company
 
02:57 Understanding the Application Environment
 
04:54 Building a Solid Security Foundation
 
11:29 The Role of Static Analysis (SAST)
 
17:12 Empowering Teams with Security Mindset
 
22:07 Collaboration with QA for Security
 
24:47 Ensuring a Clean Build: Developer and QA Collaboration
 
26:17 Dynamic Scanning Explained
 
27:32 Regression Testing and DAST
 
28:05 Understanding DAST Results and Fuzzing
 
33:24 API Testing: A Critical Component
 
37:02 Containerization and Security
 
42:12 Building a Secure Development Process
 
46:39 Final Thoughts and Key Takeaways

Creators & Guests

Host

Chris Lindsey

Chris Lindsey is a seasoned speaker who has appeared at conferences, webinars, and private events. Currently building an online community and creating a podcast series, Chris draws on expertise from more than 15 years of direct security experience and over 35 years of experience leading teams in programming and software, solutions, and security architecture. For three years, Chris built and led an entire application security program that includes the implementation of mature AppSec programs, including oversight of security processes and procedures, SAST, DAST, CSA/OSA, compliance, training, developer communication, code reviews, application inventory gathering, and risk analysis.

What is Secrets of AppSec Champions?

Join host Chris Lindsey as he digs into the world of Application Security with experts from leading enterprises. Each episode is theme based, so it's more conversational and topic based instead of the general interview style. Our focus is growing your knowledge, providing useful tips and advice. With Chris' development background of 35 years, 15+ years of secure coding and 3+ years running an application security program for large enterprise, the conversations will be deep and provide a lot of good takeaway's that you can use almost immediately.

[00:00:00] Anthony Israel-Davis: How do we actually build software? Especially if you've got a large group. Now we talked about in this scenario that we, I walked in and there are 200 developers and it's the largest organization. I need security champions. For me, a security champion is critical to our success. The important thing is that you've got somebody there that wants to represent security and can do that effectively in that team.

[00:00:23] Chris Lindsey: Hello and welcome. I'm Chris Lindsey and you're watching the Secrets of AppSec Champions podcast. I would like to introduce our first multi part series, Decoding Application Security, where technology leaders will walk through what a successful application security program looks like. Today, we are talking with Anthony Israel Davis.

[00:00:40] Chris Lindsey: And we will be walking through part one, walking in cold. The premise CISO or application security manager has just started at a new company that has 200 plus developers. The previous manager quit and did not leave much information. We're going to cover what the first several months of what steps should be taken.

[00:00:59] Chris Lindsey: Anthony, [00:01:00] welcome to the program.

[00:01:01] Anthony Israel-Davis: Thank you.

[00:01:01] Chris Lindsey: Would you like to introduce yourself?

[00:01:03] Anthony Israel-Davis: Sure. I'm Anthony Israel Davis. I am the manager of product security at Fortra. Welcome.

[00:01:07] Chris Lindsey: Welcome. Anthony, let's assume that you're the one that just walked into the new company. Knowing what I just shared in the premise, Where would you start?

[00:01:17] Chris Lindsey: Day one. Let's assume that all the HR stuff's out of the way, and you have your desk and computer and the basics ready to go.

[00:01:24] Anthony Israel-Davis: Sure. Well, you know, it's interesting. When you walk in, there's a lot to do. And you think, here's all these tasks I need to do and all the things that need to get done. What really needs to happen out of the gate is, You need to understand what's going on there.

[00:01:40] Anthony Israel-Davis: You need to understand the culture. You need to understand the products, but most of all, you need to understand the people and so for me, the first thing that I want to do is get to know all the people I'm going to be working with because the last thing you need is to walk in as the security person and everybody immediately closes all their [00:02:00] laptops and doesn't want to talk to you.

[00:02:01] Anthony Israel-Davis: You want to be their friends because what you really are is you're there to enable their success and they need to know that. And so what I would want to do walking into there is start to meet with all the people, all the leaders of the teams. And you said, you know, this company has got. 200 developers, I don't know how many managers there are, how many VPs, but I'd want to start talking with those people.

[00:02:25] Anthony Israel-Davis: So I would set up meetings with them. So I would want to meet with all those people and I would just ask them, what keeps you up at night? What are the things you're concerned about? What are the things that will make you successful? And then one of the most critical questions. Who else should I be talking to?

[00:02:41] Anthony Israel-Davis: Absolutely. And listening to them. Absolutely. So, that's where I would start. Very cool.

[00:02:47] Chris Lindsey: So, you know, as you're talking to the different managers, software architects, finding out what keeps people up late at night, I guess, you know, one of the big things is, is you're right. Having a good inventory of the applications that [00:03:00] are within the environment and what's going on, what kind of things would you want to know about the applications running in your, your new

[00:03:06] Anthony Israel-Davis: environment?

[00:03:07] Anthony Israel-Davis: Oh, wow. Yeah. So the first thing I would want to know is am I dealing with a single application? Is this a company that's like a Netflix or something that that deliver essentially a single service? That's fairly, I don't want to say monolithic because I know that Netflix has Tons and tons of stuff going on behind the scenes, but something like that, or is it something that's multifaceted with lots of different types of products, something that maybe they've got a lot of acquisitions.

[00:03:32] Anthony Israel-Davis: And so there's, there's things that are on prem and things that are in the cloud. So I want to understand what the products looks like. I want to understand what the culture is, how do they develop? Is it a lot of smaller teams? Is it one big team? Are they using a DevOps principle? Are they agile? How do they work?

[00:03:50] Anthony Israel-Davis: And. Then what tools and processes do they have at their disposal? So are they using something now? Are they already in the [00:04:00] flow? Do they have a security first culture? Are they secured by design or secured by default? Or are they starting from scratch? I think the other big question too is, Is this a new company?

[00:04:10] Anthony Israel-Davis: Is this a new product? Are they building something that's a startup? Or do we walk into something that's a 20 year old product? And there's lots of different things to deal with in that situation.

[00:04:22] Chris Lindsey: No, absolutely. I mean, some of the things that you can do with the newer technologies, you can't really apply or use with the older technology.

[00:04:30] Chris Lindsey: I mean, you VB6 application that's running, you know, as part of the pipeline. Amen. You know, there's not much you can do there. Whereas you might also have something written in Java that's, you know, brand new Java or brand new NET Core. I mean, you know, the language is, you're spot on. And so, uh, having a good inventory and knowing what's out there is absolutely critical

[00:04:51] Anthony Israel-Davis: for success.

[00:04:53] Anthony Israel-Davis: Right. And what do they have in place? So when I walked in, there was a little bit in place, but there wasn't a lot. [00:05:00] And so, as we started to look, we thought, well, what do the teams need to be successful? What's the first thing they need? And for us, the first thing they needed was just visibility. You can't really work on what you can't see.

[00:05:13] Anthony Israel-Davis: And so, As a security focused person, like, what are the security things that these people need to see? And what's the quickest thing I can deliver out of the gate? There's a lot of security tools you can deploy, but what are the things that are going to make the team most successful more quickly and that you can build on from there?

[00:05:30] Anthony Israel-Davis: So you want to be able to. Start with a solid foundation and begin to build from there. It's really easy to see like, oh, there's a thousand things I can do and let's start doing them all. But that's a recipe for disaster. You really need to build incrementally. And so find out where your successes are and begin to target those.

[00:05:49] Chris Lindsey: No, absolutely. That's what I was just thinking as you're talking is. Where to start because you can't boil the ocean on day one and knowing what [00:06:00] applications you guys are using. I guess another big question about the applications that are running is, you know, where are they running? Are they running on prem in your own data center?

[00:06:09] Chris Lindsey: Are they running in the cloud? Are they externally exposed internal tools? Are you running microservices? I mean, there's so much that plays into this.

[00:06:18] Anthony Israel-Davis: Right. And not only that, as you're talking about all those different where they live, you also have what they house. What's their risk profile? Do they have critical information?

[00:06:29] Anthony Israel-Davis: What are you protecting? Because not only are you saying, what's the easy thing to deploy? But what's the most important thing to deploy because you want to find out where your greatest risk areas are and manage that risk appropriately because what we're really trying to do is manage risk when security comes in is like how do we help manage risk to your customers data to your data to your company so that we can effectively deliver your products at speed.

[00:06:55] Anthony Israel-Davis: Right, absolutely.

[00:06:57] Chris Lindsey: So, as you're looking at this, one of the [00:07:00] first things you're going to want to know is how are you building your application? Because depending on how you're building it depends on, you know, your workflows, right? You may have something that's Running in GitHub as an action build script.

[00:07:13] Chris Lindsey: You may have something that's running in a pipeline. Hey, I've committed code and it just picks up. You know, you have multiple tools out there. Team server is, you know, one. Jenkins build server. There's Circle. There's plenty of different build servers that are out there. And knowing where things run and build, would you take a look at those and see, Hey, can I.

[00:07:35] Chris Lindsey: What kind of scalability could my design or, you know, plan play into

[00:07:39] Anthony Israel-Davis: that? Sure. You know, absolutely. I have seen everything from we build this on a laptop by hand to it's part of a mature build pipeline and depending on how things are done. You have different opportunities in order to engage that.

[00:07:54] Anthony Israel-Davis: Sometimes those things that are not as mature or just have an older style of building, [00:08:00] you can still take advantage of that. But maybe what you're wanting to do is let's get this into a pipeline. Let's get this into something that's doing some CICD. Rather than trying to do it all on a laptop. On the other hand, you might walk into that exactly what you said.

[00:08:14] Anthony Israel-Davis: They've got a mature pipeline that's going from source to artifact or deployment. And now you can plug into that and that affords you all the automation opportunities that you want to take advantage of. So you can see what things look like from the code. You can see what things look like, what's going into those things, what's going out.

[00:08:33] Anthony Israel-Davis: And then. Evaluate risk at each stage of the way. And that pipeline, you could even stop that build and say, nope, this is too risky, let's go back and fix that risk before it gets to production, because we don't want to expose our customers to that. Absolutely.

[00:08:48] Chris Lindsey: So let's assume that. You've gone out and you've found out there just aren't any tools there.

[00:08:55] Chris Lindsey: What's a good tool to start with? I mean, cause on your security wheel, you have so many [00:09:00] different things and they each bring different values.

[00:09:02] Anthony Israel-Davis: Yeah. So that's a great question. The way I like to think about the software is. Almost like cooking. So when you're putting something together, when you're cooking, you've got a lot of ingredients that go in and your raw ingredients might be your source code.

[00:09:19] Anthony Israel-Davis: This is sort of, you want those to be really good. And then you've got maybe some other things that are raw, they're composed. So that would be like your third party libraries. And so in order to make good software, you need to have good ingredients, just like, if you're going to make a good cake, you need to have good ingredients.

[00:09:34] Anthony Israel-Davis: You don't want to use expired eggs. That would be bad. And so you want to check, is this good? Is it fresh? Is it going to make a good product at the end of the day? And so I would start building those foundations. Now, where I start might be how quickly I can get that done. Deployed and in use. So, for instance, for us, that was on the open source side software.

[00:09:57] Anthony Israel-Davis: So, software composition analysis and third party libraries [00:10:00] that was already in place. We already had some of that going. It just wasn't where it needed to be in terms of organization and maturity. And so for us, I could take what we'd already started and bring that into a place where a completion. So you move it from a sort of beginning stage to a sustained stage.

[00:10:16] Anthony Israel-Davis: And now not only do you see what's out there, you can do something about it and effectively manage it. And then I can move into like a static analysis, start taking a look at the source code. So that it was a little bit trickier for us. In fact, it's still something that we are working on. And the reason is for us, We've got a ton of different languages.

[00:10:36] Anthony Israel-Davis: So you've got lots of different languages, lots of different groups, different levels of maturity, and that's a lot harder to unify than we're all building in Python and we can deploy something this works for everybody and they're off to the races. And so that's just a matter of efficiency. You're always going to look at.

[00:10:55] Anthony Israel-Davis: How much is this going to cost? What's the benefit? And then how quickly can I derive that [00:11:00] benefit for our developers? And again, it's all about the developers. So we talked about security at the end, but if the developers aren't using it, if they're not getting value out of it, if they're not able to do the things they need to do, upgrade the libraries, make their code better.

[00:11:14] Anthony Israel-Davis: Then you've got tools that tell you a lot of information, but they don't help you make any progress.

[00:11:21] Chris Lindsey: You brought SAST up. SAST is absolutely a great second thing to have in your book of tools. Let's talk about SAST. For those who may not know SAST or the benefits of SAST, let's talk about what kind of value and benefits it brings.

[00:11:35] Anthony Israel-Davis: Sure, that's great. So Static Analysis, or SAST, is a way to look at that source code and find problems, and a lot of SAST tools use a pattern matching, so it's looking for all these patterns which are dangerous, or unsafe, or maybe they're looking for secrets that you've put inside your code, and pointing those out to you, and saying, We found all these [00:12:00] problems in your code.

[00:12:01] Anthony Israel-Davis: Here's some recommendations to fix it. It's a little bit noisier than SCA because code is freeform. You can write what you want, right? So you can write a function, you can use, there are built in things, but every developer does it differently, potentially, or every team might do it a little bit differently.

[00:12:19] Anthony Israel-Davis: And so the SAS tools have to find the things that are most important to address. And Point those out. And so it's a definitely work, but it really helps developers see things in their code that they might not have seen. The other advantage of that is a lot of developers inherit code. Like I didn't write this.

[00:12:40] Anthony Israel-Davis: I was just hired a week ago and I've got this pile of source code. I'm working on this module. I'm not even touching this old modules. I don't know what's in there. So your SAS is going to say, well, I know it wasn't you, but the previous developer wrote this bad code, you might want to take a look at it.

[00:12:56] Anthony Israel-Davis: So, it's a very valuable tool to point out things that [00:13:00] are potential issues. Yes.

[00:13:02] Chris Lindsey: Well, and the other thing you run into with software development, I call it notepad inheritance, where I need something, this other application has it, I'm just going to copy it from there, plug it into my application, and go forward.

[00:13:15] Chris Lindsey: Again, same mentality as the open source. Hey, it works, if it's the need. Yeah, I'm moving forward, and the things that you run into, for those who aren't familiar with SAS, there's several key Areas that it focuses on and helps you, for example, you have SQL injection, command injection, cross site script injection, multiple, you know, both really, really bad things that you really want to address immediately versus is not.

[00:13:43] Chris Lindsey: And when you're a large enterprise, and you were alluding to this earlier, the big thing that you run into is you have technology that are, you know, that was written back in the 90s. Take care. Or even earlier all the way up to last week today even for that matter, you know [00:14:00] and so a common development practice back in javascript was using eval and Eval today equals evil because you know whenever you run into those you're gonna find Very very bad things and especially if they're not handled and so The nice thing about the SAS tools of today is it's going to point out you're using eval and here's why it's bad and here's the information and you should be concerned about it.

[00:14:28] Chris Lindsey: And if you're not looking at a tool or using a tool, you're not really, you're just running day to day and you're not thinking about the different landmines that your applications have.

[00:14:38] Anthony Israel-Davis: Yeah, absolutely. So what I would think of the way I think of SAST and this SCA is like this as well is it's a diagnostic.

[00:14:45] Anthony Israel-Davis: So it's going to run something. It's going to tell you, Hey, we found this stuff. It looks like it might be a problem. And here's how this code is executed. So we can trace the path and maybe like something with eval is like, well, maybe that's just something that we don't allow at all. We want to make [00:15:00] sure you get that out of the code and use a different pattern.

[00:15:02] Anthony Israel-Davis: But it also might find things that, well, in this situation it's okay, so we're going to accept that, but in this situation it's not, so it really It helps developers see areas that they need to pay attention to, versus just trusting what's out there. We can do peer review. We can do all these other processes, which are really great, but a SAS tool is one of those areas that says, I'm going to light up all these things on this board and I want you to go look at those things and you can turn that light off if you want.

[00:15:30] Anthony Israel-Davis: You see something, let's get that fixed and let's also prioritize it. So that's another thing that you'll see both in SCA and SAS. It's web scores, right? This is a low thing. This could be executed, but it probably won't have a high impact. This is a critical thing. If this gets impacted, it's gonna be really bad.

[00:15:47] Anthony Israel-Davis: So let's deal with that. And that also gives you a way to prioritize that list and say, okay, I'm gonna do this stuff because it looks really bad. And if it doesn't need to be fixed because it's not [00:16:00] dangerous, great. What I've got done is the developers looked at it and they said, I accept this. This is fine.

[00:16:06] Anthony Israel-Davis: This is a false positive and they've done their due diligence. So now we're affirmatively and proactively managing risk rather than just ignoring it. And that's really critical. You've got to manage risk rather than ignore it. Right.

[00:16:20] Chris Lindsey: I mean, one of my, my old little taglines was enable innovation. And you can't really stump your developers from being active.

[00:16:29] Chris Lindsey: And one of the things, and you were just mentioning it, you know, a SAS scan will give you a lot of information. And one thing I want to bring up is some tools will give you a lot more than others. One of the things that you want to be concerned about when you're looking at the results is where does this run?

[00:16:46] Chris Lindsey: You know, I have a command execute, you know, a command injection. But if it's running at a console application, does it matter that you can get to the command prompt when you're already at the command prompt to run it? You know? Right. So environment really plays a [00:17:00] big part into this. And that kind of goes back to the inventory thing you were talking about before.

[00:17:05] Chris Lindsey: is knowing what applications do I have and where do they run? Because knowing the scope really is a benefit for any of the

[00:17:12] Anthony Israel-Davis: tools. As we start talking about tools and processes and standards, and we've talked about some tools, but even just what you're talking about with a medical device, or any type of issue, people come to us and they're like, well, let's do a threat model.

[00:17:24] Anthony Israel-Davis: So now we start to enable our teams to think in a secure mindset. Like, well, what's a threat model look like and how do we do that? Mm hmm. What are the things that could go wrong with that? What kind of data does it have? And now we've empowered those teams to make their own decisions as well. They can help evaluate risk.

[00:17:42] Anthony Israel-Davis: And then we can come in and support that and maybe check off. Yep, we agree. Nice job. At part of that enablement is now we've got a security mindset. Now we're able to look at this and go, Oh, okay. Yeah, I can see where that problem is. And I think this will be good enough. And here's why. [00:18:00] And that's great because at some point we can't just be answering questions all the time.

[00:18:05] Anthony Israel-Davis: How do I do this? How do I do that? Right? We are happy to do that. We like to do that. But once teams are answering their own questions again, now we know we're being successful.

[00:18:14] Chris Lindsey: One of the things that was very successful for me when I was running, my program was creating a central location for documentation.

[00:18:23] Chris Lindsey: And so, as I've best practices, as I've helped them on the mobile application questions and things, we had an internal confluence page. That I simply put all the information up there so that then next time when the question is being asked, now you can point them there or ask the team, say, now that you've done this, would you mind doing a quick blurb and uploading it there?

[00:18:45] Chris Lindsey: That way you have a nice common location for this kind of stuff.

[00:18:48] Anthony Israel-Davis: Yeah, no, that's great. I think that's a fantastic idea. And you can make it work. It's good. However, that works for your company or your group, because maybe it is a central location for everybody, or [00:19:00] maybe it's various teams have their own sets of standards, because one's doing mobile and one's doing SAS and one's doing something else.

[00:19:07] Anthony Israel-Davis: But overall, what you need is this is the way we do things. We've solved that problem. We don't need to solve that problem again, because we've got a good pattern for that. And now we can follow the pattern. So once you've done it the first time, now everybody knows how to do it. And you don't have to reinvent that over and over again, which is absolutely the way to go.

[00:19:26] Anthony Israel-Davis: It's like centralized documentation is key and having a centralized standard. So for instance, something like the OASP ASVS, which is just an application security verification standard. So you can say, here's all these practices, 400 practices or whatever they are, that are, how do you handle sessions? How do you handle encryption?

[00:19:44] Anthony Israel-Davis: And then. Now you can audit and say, are we doing these things? And if we're not okay, why not? Maybe there's a reason for it or, Oh, we should be doing that. Let's get that in the next version or in the next iteration. So those standards and having that is critical, whether you choose [00:20:00] that version or not, but having something that this is how we build things, and this is what security means to us.

[00:20:06] Chris Lindsey: One of the beautiful things about OWASP is when you're following OWASP and you're actually doing. A lot of their suggestions, it actually cleans up your SAS results and things that the OWASP will tell you about command injections and. The different things out there and the proper ways to fix it. Your tool, whatever tool you're using for sash, really should align with a lot of those recommendations and suggestions coming out,

[00:20:31] Anthony Israel-Davis: right?

[00:20:31] Anthony Israel-Davis: No, absolutely. That they look for those sorts of things as well. So the CWE, the common weakness enumeration, they say this thing has this particular problem, so this. What that pattern looks like, and there's a whole website that actually has code snippets for that. So tying those back in to say, this is what that looks like.

[00:20:52] Anthony Israel-Davis: This is what happens when that gets executed. That's why this is bad. Here's how to fix it. And now [00:21:00] your code is slowly improving over time. And that's key to that. So, understanding. What the problems are, why they're a problem and what you can do to fix it are critical. And that goes back to education as well.

[00:21:13] Anthony Israel-Davis: So being able to train your developers so they understand and recognize those patterns. So if you can get those patterns, if they don't show up and they don't, the scan doesn't see them, that means they've solved them out of the gate. And that's the other thing with SAS too, if you don't mind, just like thinking through, like.

[00:21:31] Anthony Israel-Davis: I keep trying to push that as far to the developer as possible so we can run the scans after the fact so we can look at the code repo and find all the stuff. That's great because there's lots of areas where the developer might not have been developing. It might be old code, but when you're developing new stuff, you want to solve that.

[00:21:48] Anthony Israel-Davis: Right away. Right. I want to see that in your IDE. You want to see that like, Oh, there's a problem. I'll fix that right now. It never even hits the merge request. It doesn't even hit the main branch because you fix it out of the gate. So if we can do that [00:22:00] again, that's a win for us. So I don't want to see it because it's not there.

[00:22:05] Anthony Israel-Davis: And I like that. Well,

[00:22:07] Chris Lindsey: and you know, one of the things, cause this is a path that I was actually talking with a couple of companies over at developer week is QA, the real cost of QA. And you talked about coming into the company and, you know, building those relationships. QA is one of those relationships because, you know, with QA on board, now they're a gatekeeper.

[00:22:29] Chris Lindsey: Now they're going to be there. They're going to be your partner. And they're going to look at the transparency. Hey, show me the number of vulnerabilities coming into this release. And those are questions and thoughts that you should probably, if you're not doing today, talk to your QA director and your QA staff and say, Hey, You should be asking the questions.

[00:22:48] Chris Lindsey: What kind of security footprint am I looking at with this release? Did you increase the, you know, the vulnerability risk of the application? Or the, you know, not vulnerability, but just did you increase the risk? [00:23:00] Did you reduce the risk? What does that look like? Because from a QA standpoint, QA can be the guard keeper and say, Look, I don't want this release to go out the door because you've just introduced all these new vulnerabilities.

[00:23:13] Chris Lindsey: That the security team or someone's going to have to put some kind of mitigating factors out there to help ensure that we're not going to be

[00:23:20] Anthony Israel-Davis: more at risk. Yeah. Well, and QA is a natural ally as well. So they're doing a lot of things that. We're doing just looking at different facets of the diamond. So they might be looking at optimization.

[00:23:34] Anthony Israel-Davis: They might be looking at that defects of functionality, but we're also looking at defects of weaknesses in the security code. And, and they're doing a lot of things that we can piggyback off of. So I really, uh, forming that relationship with the QA group is critical because. They're actually building software.

[00:23:51] Anthony Israel-Davis: They're standing up and it's running out there. So if I can piggyback on that infrastructure, now I can run my tests alongside theirs and that [00:24:00] build pipeline potentially. They've got running a software, so now I can do dynamic analysis. They can set up software for me to do pen testing on. So there's a lot of things that we can do that's already there that I don't have to invent or build myself.

[00:24:14] Anthony Israel-Davis: And my team can then just go in and say, Oh, You've got it there. Let's just plug right into that, whether that's a DevOps pipeline or a QA pipeline, being able to use that infrastructure is huge. And QA or test is a good place to do that.

[00:24:27] Chris Lindsey: No, absolutely. And you're right when you have your pipeline and it's all working, it can spin up when I've committed code, my developer integration branch, making sure I didn't break anybody else.

[00:24:37] Chris Lindsey: And. gives me a spot that maybe you're right, you can pin test against. And that same artifact that was built there can then move on to QA, and then QA can do their thing. And as things move up, then, you know, up the ladder, you do have those different touch points that you can work with to ensure a good, clean build.

[00:24:56] Chris Lindsey: One of the things that I ran into frequently when I first started [00:25:00] really working with QA and QA pushing back is a lot of people, you're right, shifting it left into the developer's hands is, is because Once you get to QA and QA does all their testing and then it lands in your, you know, on one of your reports, say, guess what?

[00:25:16] Chris Lindsey: There's a critical vulnerability out there that they're introducing. The risk passes your threshold. Now you're telling QA, yeah, you spent all this great time testing the application, but I'm going to force the developer to fix it, which means you're going to have to rework and redo everything you just spent those 1 to 2 weeks doing.

[00:25:36] Anthony Israel-Davis: Yeah, well, and that's why you want to make sure that that's known in your dev up front. The sooner you can find out about something, the sooner you can fix it. Not always possible, but again, we're trying to do that as soon as we possibly can. So if we can test in dev and we do this with our SCA, we say, here's everything that's in development.

[00:25:54] Anthony Israel-Davis: We want to see where things are, what libraries need to get updated. But then we're also monitoring our [00:26:00] production because libraries get old, new stuff happens every day, you get a new CVE pops out. So you've got, okay, well, the stuff in production needs to be fixed. That's also in dev, but they can fix it.

[00:26:11] Anthony Israel-Davis: And that should incorporate that in the next release. And we're aware of what's out there. So, yeah, that's huge.

[00:26:17] Chris Lindsey: So, Anthony, you brought up dynamic scanning. Let's touch base on that real briefly.

[00:26:23] Anthony Israel-Davis: So for those listening, what is dynamic scanning? Sure, so I'll go back to my food analogy. So we've got all these ingredients, we've put them together, we've got high quality ingredients.

[00:26:32] Anthony Israel-Davis: But they're just ingredients. How those ingredients fit together affects the end result. So what did you do to them? Just because you have the ingredients doesn't mean you have a cake. So once the cake comes out, you want to make sure that cake is good. So you got to eat it. It's probably the best way to do that with a cake with software.

[00:26:50] Anthony Israel-Davis: You build the software. Now the software is running. So you've got your source code. You've got your libraries. How are they interacting? How are they put together? What's the logic? What are they running [00:27:00] on? Do they have a web server? How's the web server configured? How is the infrastructure configured? And so now your dynamic testing is going to test that build software.

[00:27:08] Anthony Israel-Davis: It's going to say, okay, now you're live. Let's run tests against that. And it can run, you know, Unauthenticated scans. Just what do I see on the surface? And we're authenticated scans. Let's log in and run through and test the application from an authenticated standpoint. And there's varying degrees of that.

[00:27:24] Anthony Israel-Davis: You can go very light or very deep, but that's essentially dynamic testing in a nutshell.

[00:27:29] Chris Lindsey: That's a good way to put it for those listening. One of the things that as you're going through the pipeline, you want to kick off regression testing. At the same time you're doing regression testing, that's a great time to kick off your DAST.

[00:27:41] Chris Lindsey: Actually. As soon as it's built, you kick off your DAST, because typically DAST runs for a long time. Some vendors, it runs really quick, you're talking within hours. Some vendors, it could be days before you get a good result. Just depending on the size of your application and to the complexity and the depth that it goes.

[00:27:59] Chris Lindsey: And [00:28:00] so, one of the nice things about DAST is typically the results are very actionable. And for those listening who aren't familiar with DAST, and this is part of the reason for the podcast series, is to give you this in depth information. But with DAST, the beautiful thing is, is it records what it did.

[00:28:16] Chris Lindsey: The finding that it has, it tells you, this is what I found, how I got there. And even some software gives you all the steps and allows you to actually try to fix it and then rerun that, that one specific test. so much. So you're not spending all the time trying to, you know, fix it.

[00:28:34] Anthony Israel-Davis: Yeah, and just for those listening, too, a lot of things, times what a DAST will do is it will just throw all kinds of stuff, garbage, in fact, at the, the The application, just to see how it behaves.

[00:28:47] Anthony Israel-Davis: So it's going to put all kinds of stuff and that's usually referred to as fuzzing. So that's one of the things that we'll want to do is we want to like throw a whole bunch of stuff at the web app. What are we going to put in the [00:29:00] URL? Things that people would never put in, but. Bad actors will just to see how the application responds.

[00:29:06] Anthony Israel-Davis: And so now you see, oh, if somebody did a thing this way, and a lot of those patterns are things that those bad actors like to do, path traversal, cross site scripting. So it will try to do all those things. And if it gets a success, now, you know, you've got a problem. And it'll, like you said, we found it in this string, this URL, here's the request.

[00:29:27] Anthony Israel-Davis: Here's the response. This is what you eat. Found and here's what you can do about it to fix it. So, yeah, that's it's critical because those pieces aren't always obvious when you're putting them together. But now when you actually have running software, suddenly their surface because something's interacting in a way that you weren't expecting,

[00:29:44] Chris Lindsey: right?

[00:29:45] Chris Lindsey: And the nice thing, and you alluded to it, you know, the thing that's really nice about Dask 2 is your UI may be, hey, I accept 20 characters. But it's going to go through 4, 000 characters at it behind the scenes, you know, and see what blows up and it's always kind [00:30:00] of fun when you run the dash results, then you go back to the development team.

[00:30:03] Chris Lindsey: And you're like, hey, look what I found. And their 1st answer is, but the UI doesn't accept it. And then you're like, yeah, but let me show you the.

[00:30:10] Anthony Israel-Davis: You know what happens. Mm hmm. Steps to reproduce. That's critical too. So DAST usually has that, so you want to be able to say, I did these things, here's the request that I put in, here's the response, and it's irrefutable.

[00:30:24] Anthony Israel-Davis: Yeah. Like, oh, yeah, I can repeat this and this is what happens. And it's just another step in that visibility that we're trying to get. We want visibility in the code, we want visibility in the source, and we want visibility into the running software to say. Here are all the things that we found. Here's the risk that it has exposed, and here's what we can do about that risk.

[00:30:45] Chris Lindsey: Right. So one of the things for those listening, the nice benefit of DAST is it actually gives you actual findings. And so when you look at it from a false positive standpoint, there still can be a false positive, but typically it's pretty low. [00:31:00] When you look at the SCA, SCA is really what it is. Again, it's rare to have a false positive with SCA as well as DAST.

[00:31:07] Chris Lindsey: Typically some software and just kind of helpful for you. I harp on SQL injection all the time. You know, one of the things that you want to make sure is, is, you know, whatever environment this is running on, don't run it under powered because it's going to take longer to run. But the other thing, too, is some of the commands that it runs for identifying a SQL injection is, you know, telling the SQL server, go wait 10 seconds and then come back with the result and most tools, if it sends something there and it takes 10 seconds for it to just process, it's not going to know the difference of, was it at the database or was it somewhere else?

[00:31:43] Chris Lindsey: And so that's something to consider. And then your SAS results. Some of those are knowing which ones are more, uh, false positive prone versus others finding that, you know, Hey, I have a hard coded password, but it's in a dot config file [00:32:00] that's on a dev environment. That's never going to go to production.

[00:32:04] Chris Lindsey: Can definitely be minimized. And so, yeah, you look at the noise and you brought it up a little bit noisier. It's just something that you can go through and say, yeah, we shouldn't be checking configs or things into the repositories.

[00:32:15] Anthony Israel-Davis: Yeah. And even with SAST, it's not that it's a false positive because it's doing pattern matching.

[00:32:21] Anthony Israel-Davis: So this is a known bad pattern. We found it in your code now, just because you have that pattern doesn't mean that it's exploitable. But it's a known bad pattern and you should take a look at it. Right. It might be acceptable. I think that's the same thing for open source software. We see that all the time is, Oh, this is a CVE, but only if I'm using this particular function in this particular context, we've got mitigating controls in place.

[00:32:45] Anthony Israel-Davis: Still update your libraries, always be updating, but it, all it's doing is saying here's a known risk that we know about. We're letting you know about it. What are you going to do about it? And now we can't claim ignorance. I didn't know about it. We know about it. [00:33:00] Now we can choose what to do about it. And it's all about choice because at the end, I'll just continue to return to that.

[00:33:06] Anthony Israel-Davis: It's all about risk and managing risk. And what is your risk tolerance and what's your need, you know about it, and now you are able to act responsibly about that risk. Absolutely. Absolutely.

[00:33:18] Chris Lindsey: So we've hit three items on the security wheel. We've hit, you know, SDA and SAST and DAST.

[00:33:24] Anthony Israel-Davis: Mm hmm.

[00:33:24] Chris Lindsey: One thing that goes really well with DAST is looking at API testing.

[00:33:29] Chris Lindsey: Because API testing in a lot of ways is really kind of DAST in a way, but now hitting endpoints versus URL endpoints, now you're hitting an endpoint testing for APIs. And so, uh, where does API testing, not for where you're at, but, you know, just kind of in your book, right? Where does API testing sit?

[00:33:49] Anthony Israel-Davis: No, API testing is critical.

[00:33:50] Anthony Israel-Davis: I don't really distinguish between DAST and API testing. That's just all part of the same test, because whether it's a user interface or an application [00:34:00] interface, it's still an interface. It's still an endpoint that something is interacting with, whether that's a program or whether that's an end user.

[00:34:07] Anthony Israel-Davis: And so API is just another doorway into that application that can be used, abused, misused. It can leak data. It can cause problems. So you absolutely need to check what that API is doing, how it's interacting. It is not even, I would say, separate from that DAS. It's just part of it at this point because APS are so critical now for everything that right software does if there's an API.

[00:34:35] Anthony Israel-Davis: It's unusual. Let's put it that highly unusual.

[00:34:38] Chris Lindsey: Yeah. And for those listening, you know, most people think of APIs for, you know, Hey, I got a website. It's hitting it. I can have a web farm, whatever. But the other thing too, where APIs come in, think mobile, right? You got a mobile phone. You're not going to store the database on the mobile phone.

[00:34:54] Chris Lindsey: You shouldn't. You have it up in the cloud. And so by having APIs, you store it up there. And then that way you [00:35:00] can go from an application, you know, mobile to web. And now you've shared it. But it even goes beyond that. You also have B2B APIs, right? Hey, I want to be able to pull data out of your system. And you've written APIs is user roles, right?

[00:35:17] Chris Lindsey: They may have access to certain data, but not all the data. So make sure your API endpoints are really only pulling the right

[00:35:23] Anthony Israel-Davis: thing. Yeah, it's easy to have to leak data if it's just uncontrolled in that way. And if we go back to the OWASP ASVS and other patterns like that, there's lots of guidance out there of how to build an API.

[00:35:36] Anthony Israel-Davis: What are the best security practices for APIs? And those can all be tested either through something like DAST or a fuzzer where it's doing it automatically. Or even through just a code review and talking through the developers and how did you secure this? And let's walk through this from a just an architectural diagram or data flow diagram.

[00:35:56] Anthony Israel-Davis: What role is this user have when it's [00:36:00] accessing this particular data? And is it limited to only that data that user should pull and. I know that we've done that internally, and it's very helpful to think through as you're designing your APIs, running your APIs, and as you're testing your APIs, what can we do with those?

[00:36:15] Anthony Israel-Davis: And often data, those APIs are extremely powerful. So you've got to secure those in. The best way possible, right? And not expose any ones that you don't want to expose to the say, the Internet. Like, if this is an internal API, let's keep that endpoint hidden. Let's not expose that to the Internet. And so that's another way we start looking like, is, should this be accessible?

[00:36:37] Anthony Israel-Davis: Because we accessed it. So let's talk about that. Right. And help lock that down. Yeah. Yeah. APIs are absolutely a critical element now. Particularly anything that's web facing. Once you're internal, obviously those are important, but your attack surface is now reduced to your network and you have different ways of managing that risk.

[00:36:58] Chris Lindsey: Well, [00:37:00] let's talk images, right? So

[00:37:02] Anthony Israel-Davis: containerization. Oh, wow. Okay. Yeah. There's a lot in containerization. Oh my gosh. So now it's another trail to walk down. So like SCA, and in fact, uh, sort of really along the same path of the SCA, There's a lot of open source that goes into a container. A lot of times they're using an open source, like a Linux kernel, and a Linux image, and that's using a lot of open source software as well.

[00:37:28] Anthony Israel-Davis: And then we're putting our own software on top of that, that's running. So understanding what's going in and what the exposure is there is really important. So being able to scan those container images, just like you're scanning, you Open source software, just like you're scanning your built software, scanning those images to know what's in there and what potential risks there are is critical because then you can update those images if they need to be updated.

[00:37:52] Anthony Israel-Davis: Maybe you're using an old operating system. Maybe you're using old software. And so. That's just another layer [00:38:00] to the onion of that we continue to go down from a security standpoint.

[00:38:04] Chris Lindsey: No, absolutely. What is your advice? Should I just pull directly from the registry for every build? Or should I store something on prem and pull from there?

[00:38:13] Anthony Israel-Davis: Oh, so if I were to build it from sort of my dream, I would say that we put together an image that is got a container registry that's controlled by you. So, wherever that is, I'm not going to tell you where that should be or how that should be done, but you have a known good container registry that's gone through its scans.

[00:38:34] Anthony Israel-Davis: It's gone through its approvals. We say this is a good. Registry. Now that can continue to get stank and, and maybe it needs updates, but when it gets updated, then it gets swapped out and use the new known good container for your software. So yeah, I would say having that control is better than just pulling from just the open internet and maybe that's where you start, but that shouldn't be where you end.

[00:38:58] Anthony Israel-Davis: So you've got to start [00:39:00] somewhere, but once you've got that, make sure that it's well built, well architected and safe, and that everybody is pulling from the same one. If suddenly you've got. Different containers across your different images across your network, and they're all supposed to be running the same thing.

[00:39:15] Anthony Israel-Davis: You've got a big problem on your hand because now, uh, now you've got all these different things. Some might be good, some might be bad, and that's just a harder problem. It just adds complexity. So pulling from the same one. This is the known good one. Everybody use this one. Then, you know, you're off to the races.

[00:39:33] Anthony Israel-Davis: No. So. If I were to start, I would say we're doing it that way. That isn't always where you start. Sometimes you need to get there, but at least have a known good set that this is where we pull them from. It would, it's critical. It's key.

[00:39:47] Chris Lindsey: No,

[00:39:47] Anthony Israel-Davis: I,

[00:39:47] Chris Lindsey: I fully agree with you. I was hoping you'd get down this path because this is the right path.

[00:39:52] Chris Lindsey: That is the right answer. If you're pulling from the internet all the time, make sure you're pulling from a legitimate registry, not one that [00:40:00] you found on Stack Overflow that says, Hey, I've got this image over here. You know, make sure you're pulling legitimate ones, but even those from time to time get compromised.

[00:40:08] Chris Lindsey: And that's the benefit of making sure that you pull it down. You go through your testing, treat it like software, right? Run the scans against the image, make sure the image is nice and clean, put it over to the side, give it a, you know, don't. Just put it straight to production, give it some time to bake, give it a week, because within a week, you're going to know, hey, the image that I pulled wasn't compromised.

[00:40:30] Chris Lindsey: Because if it was, they're going to pull it down and fix it within a couple hours. Typically, on average, I've seen if something's out there compromise, it gets fixed really quick and notice goes, you know, it goes out. So then you can say, hey. It's sat here for a week. I know we're good. I know it's in it.

[00:40:45] Chris Lindsey: We're clean to go.

[00:40:46] Anthony Israel-Davis: Yeah, and it's a little bit of a side quest here, but that a lot of people, you're just pull down this image and run it because it's a software. It's an open source software I'm using. I've got the internal utilities that I'm using. [00:41:00] It's a container. Just pull it down, run it in your container, whether it's a doctor or a cast or something like that.

[00:41:05] Anthony Israel-Davis: And if you haven't done your due diligence to check that you pull that down, run your scans. Now you've introduced potentially risk into your own internal tooling. If that tooling is part of your security infrastructure. Now you might have even a bigger problem depending on what that has access to. So that whole ecosystem.

[00:41:25] Anthony Israel-Davis: We just need to continue to be check, check, check, check. That's it's always checking and validating. And if there's a hash or whatever, we can check. So we can compare that. So this is the known good one they match. Okay. The good. That doesn't mean we shouldn't scan them, but at least we can determine that it's not been tampered with.

[00:41:44] Anthony Israel-Davis: So a lot of stuff to do there.

[00:41:46] Chris Lindsey: No, that's a great point. So download it, play with it in the sandbox. Make sure because it could be sitting ticking bomb that, you know, you don't realize, and now you've just, you know, Hey, I just pulled something down and [00:42:00] poof. So that's a great point. So we've hit SAST and SCA we've talked APIs and we've talked containerization.

[00:42:09] Chris Lindsey: Is that the end of the road or is there more to

[00:42:11] Anthony Israel-Davis: be thinking about? You know, we've talked a lot about technology and I've sort of mentioned a couple of different things that are not necessarily technology related, but process related. And those are critical as well. There's things that are maybe not as sexy, but still need to get done.

[00:42:28] Anthony Israel-Davis: Things like policies and standards. And I talked about ASVS or like, how do we build our software and then training? So something like code training. So now how do I actually. Right, secure code. I don't know how many developers have this. Some really good at it. Some brand new and maybe don't know all of the stuff about SQL injection and parameterization and all the good patterns that you need to do.

[00:42:55] Anthony Israel-Davis: They need to learn about that somewhere. And if they haven't learned about it prior, part of our [00:43:00] responsibility is let's help educate you so that. Those things don't show up in the first place, and so an education program, that's really important. We've talked about sort of standards and the SDLs, that's a software development life cycle.

[00:43:15] Anthony Israel-Davis: How do we actually build software? And then another program that is critical for us, and especially if you've got a large group, and we talked about in this scenario that we, I walked in and there are 200 developers, and it's a large, you know, a large group. Largest organization. I need security champions, right?

[00:43:32] Anthony Israel-Davis: We haven't talked about security champions yet. But for me, a security champion is critical to our success. The important thing is that you've got somebody there that wants to represent security and can do that effectively in that team. So, you know, you've just

[00:43:47] Chris Lindsey: walked in the door, you're looking at what your tooling is, you're looking at your inventory, you're looking at your.

[00:43:52] Chris Lindsey: Direct reports that are up to you seeing what do they know and where they are and how things are going during the first three months you [00:44:00] were talking about it. We were talking about how effective are the tools and are they integrated? Well, and yeah, the shift, right? And the shift left type concepts of getting it in the developers hands versus.

[00:44:10] Chris Lindsey: Making sure that we're also looking at it from a higher level and across the board. The Security Champions is absolutely integral for it. Can you think of any other things that would probably be something that we would want to make sure that we're focused on?

[00:44:24] Anthony Israel-Davis: Oh, wow.

[00:44:24] Chris Lindsey: We've talked about a lot. We have

[00:44:26] Anthony Israel-Davis: talked about a lot.

[00:44:27] Anthony Israel-Davis: I would like to think through that. So I think if I go back just the highest level of what we're trying to do and return to some of those themes, Because there'll always be new tools, there'll always be new ways of looking at things. There's different categories of stuff that show up in application security posture management.

[00:44:44] Anthony Israel-Davis: Now you see ASPM out there and you see all these other types of security tools. There's a lot to absorb. But what we're really trying to do is manage the risk. So is the thing that we deliver for our customers safe? Does it protect their data? Does it protect the [00:45:00] company? And what are we doing to make it better?

[00:45:03] Anthony Israel-Davis: And You're always going to start a crawl, walk, run. You can't be running until you can walk, and you can't walk until you can crawl. And so finding the ways to build incrementally toward that end goal of securing the software. Getting the visibility. Now, I've got the visibility. What do I do about it? Now, what am I doing about it?

[00:45:25] Anthony Israel-Davis: Is it going in the right direction? Is it going in the right direction? Great. How do I sustain it? Right. How do we make it faster? How do we automate it? So, you're slowly building that program from maybe nothing to something that's, you know, a fully automated machine where you just get to see what's going on and then address the new things or address the anomalies.

[00:45:46] Anthony Israel-Davis: I've got a red blip over here. I got to go check out what's going on, but at some point you've got to get there. So again, I would say start small, build incrementally, find the areas that are going to be most effective out of the gate, provide the [00:46:00] most value quickly. And then build on those.

[00:46:03] Chris Lindsey: Absolutely.

[00:46:04] Chris Lindsey: That's a perfect starting point. And I would definitely, I'll take 20 of them, please.

[00:46:13] Anthony Israel-Davis: Oh, and be an ally. That's the number one thing. It's like, you've got to be an enabling ally, go in and make friends with all the developers and the development team and the product team and the product management team, because you're all. Driving toward the same goal and make sure you're aligned. That's the first thing I said, this conversation was, we've got to build those relationships and then get on the same page, get aligned, get going in the same direction and then build from there,

[00:46:39] Chris Lindsey: every customer that I've worked with in the past, The failed teams, the failed security groups were the ones that did not have the relationships with the developers.

[00:46:49] Chris Lindsey: And when you don't have that, you're never going to be successful. Yeah, Anthony, I really appreciate your time today. I appreciate all the conversation. We've had a lot to talk about. There's a [00:47:00] lot more. I mean, honestly, we're really just scratching the surface and there's just so much and I appreciate your time.

[00:47:08] Chris Lindsey: I feel like we've hit a lot. But on the other hand. With what we have here, this is a great starting point to be looking at and to, you know, Hey, you've got to have certain things in place and why you have to have certain things in place and then where you can go from there. So,

[00:47:21] Anthony Israel-Davis: yeah. Well, thank you. It's been a great conversation and I really look forward to seeing the other parts as well.

[00:47:27] Anthony Israel-Davis: So this has been fun. Absolutely.

[00:47:29] Chris Lindsey: Thank

[00:47:29] Anthony Israel-Davis: you.

[00:47:34] Chris Lindsey: Thank you so much for joining me on this episode of Secrets of AppSec Champions. If you found this valuable, hit that subscribe button on Apple Podcasts, Spotify, or wherever you get your podcasts. And hey, ratings and reviews are like gold for us. So if you're feeling generous, please leave a kind word. It helps others discover our show.

[00:47:52] Chris Lindsey: Until next time, take

[00:47:54] [00:48:00] care.