Compromised and Reactive to Proactive Approaches

In Episode 03 of Secrets of AppSec Champions podcast titled "Compromised: Proactive to Reactive," hosts Chris Lindsey and guest Phil Guimond tackle the critical distinctions between proactive and reactive security strategies. They emphasize the importance of access logging and visibility in detecting compromises early, pointing out how changes in access logs can signal potential threats. They stress the necessity of implementing secure, tamper-proof log storage and discuss automation solutions like the "Have I Been Pwned" API and CAPTCHA to mitigate risks such as account takeovers.
 
The discussion extends to network security, highlighting the dangers of rushed setups that overlook essential measures like network segmentation and client isolation. They examine the risks associated with flat networks in office environments and how external threats can penetrate poorly segmented Wi-Fi networks. Additionally, the episode covers the significance of managing software dependencies, advocating for regular updates to dependencies and leveraging multiple sources to detect vulnerabilities beyond the National Vulnerability Database (NVD). The utilization of container technologies like Kubernetes and Docker is highlighted for their ability to seamlessly update images and pods, thereby enhancing security.
 
Finally, Chris and Phil underscore the importance of proper repository management, focusing on active projects and addressing outdated or unused code that poses security risks. Training developers in security practices and involving security professionals who can write code are presented as key strategies for proactive security. Chris and Phil also acknowledge the challenges of finding and retaining skilled security personnel while encouraging the audience to engage with the podcast and provide feedback. Together, they advocate for a balanced approach to security—automating where possible, prioritizing proactive measures, and continuously improving the organization's overall security posture.

Key Topics with Time Stamps 
00:00 Password Reuse Across Websites: Detection Methods
 
 06:06 Managing Security Challenges and Password Reuse
 
 08:30 Challenges of Unused Code in Development Projects
 
 10:19 Managing Data Overload with GitHub API
 
 15:33 The Risks of Network Interconnected Cloud Access
 
 17:32 Security Risks of IP Whitelisting in Cloud Hadoop Clusters
 
 20:23 Securing Network Logs from Tampering
 
 24:12 The Impact of NVD Pausing on Vulnerability Detection
 
 26:23 Efficiently Addressing Container Image Vulnerabilities
 
 31:17 The Importance of Developer Training Over Tools
 
 35:43 Tools for High-Level Security Posture Overview
 
 38:13 The Vital Importance of App Security Leaders

Chris Lindsey LinkedIn: https://www.linkedin.com/in/chris-lindsey-39b3915/
AppSec Hive LinkedIn: https://www.linkedin.com/company/appsec-hive/

Creators & Guests

Host

Chris Lindsey

Chris Lindsey is a seasoned speaker who has appeared at conferences, webinars, and private events. Currently building an online community and creating a podcast series, Chris draws on expertise from more than 15 years of direct security experience and over 35 years of experience leading teams in programming and software, solutions, and security architecture. For three years, Chris built and led an entire application security program that includes the implementation of mature AppSec programs, including oversight of security processes and procedures, SAST, DAST, CSA/OSA, compliance, training, developer communication, code reviews, application inventory gathering, and risk analysis.

What is Secrets of AppSec Champions?

Join host Chris Lindsey as he digs into the world of Application Security with experts from leading enterprises. Each episode is theme based, so it's more conversational and topic based instead of the general interview style. Our focus is growing your knowledge, providing useful tips and advice. With Chris' development background of 35 years, 15+ years of secure coding and 3+ years running an application security program for large enterprise, the conversations will be deep and provide a lot of good takeaway's that you can use almost immediately.

Phil Guimond [00:00:06]:
Access log and actually is a really, really good way of detecting when you have a compromise. You could have a whole bunch of, you know, 400 that suddenly become 200. That shows that these people are eventually able to log in, right. And it shows that there was probably a brute force attempt and you can find all kinds of problems that way. But you need logging, you need visibility before you can have a proactive information security program. Without visibility, you're going to be constantly fighting fire is you're not going to be able to see where it's coming from. You might be able to treat this cause or that cause, but you won't be able to treat the root cause. It could be that somebody's computer was hacked, the developer was hacked.

Phil Guimond [00:00:43]:
It could be that you included a third party library that is pushing out your environment variables to a third party server from an Ajax post request or something like that. There's lots of different ways. If you have no login, you cannot find the root of the problem and you're just gonna be reacting, reacting, reacting, and you're gonna be stuck.

Chris Lindsey [00:01:06]:
Hello and welcome to Secrets of AppSec Champions. My name is Chris Lindsey and today we're talking with Phil Guimond. Today's conversation is going to be around being compromised and moving from a reactive to proactive application security program. Phil is the principal information security architect at Paramount. Phil, please introduce yourself.

Phil Guimond [00:01:26]:
Hey, thanks. I'm Phil Guimorm, I'm a principal information security architect at Paramount. Like you just said, that's the fancy way of seeing I do a little bit of everything. So I work on like application security, cloud security, penetration testing, digital forensics and incident response. And lately I've started dabbling in artificial intelligence, machine learning and stuff like that. They also do a lot of programming.

Chris Lindsey [00:01:49]:
That's exciting. AI is definitely something that's big in today's world and it's only going to get bigger.

Phil Guimond [00:01:55]:
Yeah, that's true.

Chris Lindsey [00:01:57]:
So a reactive program, right. It really feels like you're just fighting fires, just one fire after another.

Phil Guimond [00:02:03]:
You know, that's actually pretty common in a lot of organizations. It's pretty common across the industry. You're basically just constantly fighting fires. You have all of these problems coming in and you're dealing with them in a very manual way. One of the worst examples of a reactive program that I've seen was where we spent the entire day fighting a current takeovers. So we would go in and use some SQL queries and then we would look for evidence that the user had their account taken over. Very recently, however, as you know, account takeovers are very complicated. They're complicated because, one, the problem is on the end user side.

Phil Guimond [00:02:44]:
So the users are reusing passwords across all kinds of different websites. One of the ways we were detecting this was we had an overview in Grafana, and it will show us how many logins we had. If we have a spike of logins, if we have a spike of registration attempts, if we have a spike of checks against the active email endpoint. So let's say you want to create a user account on a website or for a program, some website, some product. If you try to create an account for which there is already an account, you go to one stage and you hit that API endpoint where you log in. I mean, when you create the registration and when you're registering your account, it will say this account is already taken. So attackers can actually abuse that to see which accounts are actually in use. And then they can attack all of the different accounts and attempt a password, you know, login, attempt password spring and stuff like that.

Phil Guimond [00:03:40]:
So this is actually very common in the industry. People go to, you can just go to Google and you can download a bridge compilation. You search for bridge compilation, brace dump, and you can get people's usernames and passwords that they have been reusing for a long period of time. Yes, yes, absolutely. So these people are reusing passwords. And we would spend all day just finding the account takeovers, which were happening every day. So the users, you know, the hackers, the attackers would just log in to and take over these accounts, change the passwords, and drain the funds in the accounts. Right.

Phil Guimond [00:04:18]:
And they would just keep doing this over and over and over again. It was pointless. So we were going through this phase where we had to manually look everything up in the SQL server so we would open, like, data. I forgot the name of that product. It's a Jetbrains product that allows you to query databases very easily. So we would look in there, we would run all these SQL queries on the database, and we would get a list of people who were potentially, and it wasn't necessarily 100% correct, but they were potentially taken over. So we would spend all day, sometimes from morning to night, solving this problem. And it would happen again the next day and the next day and the next day, never ending for three months.

Chris Lindsey [00:04:59]:
Wow.

Phil Guimond [00:05:01]:
And there were times when it was quiet and we were just fighting this. And after the first few times when this happened, I'm like, we need to find a way to automate some solution to avoid having this happen. Constantly. So a few of the ideas they come up with was using one using the have I been pwned API to see if they are known compromised credentials. Right. And another one, and this is the simplest one, right. We just put CapTCHA vcapture, you know, a Google capture on the login. So they would actually have to go in and select all these pictures.

Phil Guimond [00:05:35]:
Right. And this was actually before Google had came out with the really easy recapture where you just click I'm not a bar or I'm a human or whatever, and you had to select all these squares. Right, right. It was tiresome. So people were actually leaving the site when they tried implementing that. They were leaving the product, they were not using it because they were tired of having to mess with the capture stuff. So the business made a decision to keep fighting fast constantly. So at the time that was the best we could do.

Phil Guimond [00:06:06]:
But then shortly after that Google came up with a recapture that prevented that from being a problem. So all you had to do was check a button and they still wouldn't do it. So you were basically fighting fires 24/7 in the very manner way. So we tried building a lot of scripts and stuff to combat this. And I basically put my foot down and they said hey look, we are spending up to 8 hours a day trying to fix the customer's own problems themselves, but they are causing themselves by reusing passwords. So we're resetting the customer's password and we're sending them a reset password link in the mail, in the email. Right. But more than likely the customer's email was also compromised with the same password because people are reusing passwords across all services.

Phil Guimond [00:06:55]:
That was really annoying. But that's one of the biggest ways to know that you're in a very reactive instead of a proactive information security program. Other ways would be, let's say you have no way to find out what you have across your appsec stack. What third party libraries do you have? If you don't have a way to automatically find your third party libraries, then when something like log four j two comes out, you have to dig through all of your repositories in GitHub or whatever source control you use across everything. And you have to search for these specific files just to make sure that it's not being patched. So you've got to focus on that and it's very reactive. So in some ways even some proactive information security programs can be a little bit reactive and that's okay. But if everything is reactive, you've got problems.

Chris Lindsey [00:07:48]:
You got big problems. Well, and I want to dig just a few minutes on your repository thing. If I go in and I look at log four j, I could have the files there. It just may not actually be tied to my application. Now when I do a search on GitHub or BitBucket or whatever source code management system you're using and I'm going through and I'm looking for log four j, it's possible that it may be showing up because somebody just uploaded it, but it's not actually being utilized. Now all of a sudden you're wasting time trying to open up the project. And again the whole thing about being reactive is you're spinning your feet, you're wasting cycles doing no real benefit to your program.

Phil Guimond [00:08:30]:
I agree. It could just be including it like you said in some Java file or something and it's not being utilized, it's not being called. But that doesn't mean later down the road, a few minutes later, a few hours later that the developer can't actually enable that because they might actually just start using it because they would do that when they're coding in Java they'll be like import package name Comdatz, you know, something. And they may actually end up importing log for j two. So that's definitely a problem. I would definitely like to make sure that that is either excluded or upgraded. But like you said, you'd be wasting a lot of time chasing your tails on a lot of different projects. So it's important to focus on things that are actually running in production, things that are being utilized because there's a lot of test projects in there that's only running locally on your own system and it's not really accessible to the outside world.

Phil Guimond [00:09:20]:
So you don't really need to worry about that too much unless you being attacked on some hotel network or something.

Chris Lindsey [00:09:28]:
You're right, because with source code management systems people upload all the stuff in it. It could be a proof of concept something program, it could be something I just want to try. It could be something production, it could be something that has been deprecated and no longer being used for years. It's one of those things where people in source code management systems really don't do a pretty good job at cleaning up after themselves. I mean think about applications where people are writing things and they're progressing forward, but you may have removed the usage or the need for a method or some aspect of an application, but you still left the code in there. And so with that code sitting there that the code's really being used or not, that goes against your security tech debt and it just makes it harder for your reactive program to become proactive because you just have all this lingering stuff.

Phil Guimond [00:10:19]:
Yeah, there's a lot of lingering stuff at pretty much every organization out there. And I think it's a big problem when you're using those source code management tools that you're talking about, you import pretty much everything. You import every single repository you have out there. But maybe like only 10% of that or 15% is actually going to be in news and running in production. The rest is just stuff people built a long time ago. I found an interesting way to deal with that is by building a tagging system. So the GitHub GraphQl API actually allows you to search for, you can search for a list of organizations within a list of repositories within an organization. And if you get a list of all the repositories, then you want to take a look at the commit history.

Phil Guimond [00:11:02]:
So one of the things you can do is check to see if anybody has actually committed to that repository the last x number of days. So for example, it could be 90, it could be 120, it could be 360 days, 365, however you want to do it. Right. But in general, things that have not been committed to in a year or more are probably not active. So you can ignore that stuff. So what I did was basically I worked on building a tagging system that would basically mark a project as an inactive, if it hasn't been updated in a long time, it would still be there on the source control system. Right. But you would not have to worry about the vulnerabilities right away.

Phil Guimond [00:11:43]:
You would focus on the stuff that's being actively developed. Have you committed to this stuff in the last 60 or 90 days? Great. Well mark, that as active has it been x number of days since it's been committed to, well mark, that isn't active. That way you don't have to focus on all the vulnerabilities that all of these packages that are like 90%, maybe thousands of times larger than your actual list of running projects in production. So that helps a lot too.

Chris Lindsey [00:12:10]:
It does, because knowing what's active is important, because again, if you have a breach or you have something going on or you're spinning your wheels, where do I focus? Where's my priorities going to go? I have a spring for shell show up a vulnerability. Hey, we need to go identify these. Part of the problem is if when you're looking, as you were mentioning through GitHub, or your repository system, they're going to show up and knowing what's important, what's active, what's not active, is absolutely key to knowing prioritization of your problems.

Phil Guimond [00:12:42]:
Exactly, that's a good point. In fact I think it's really important to focus only on the stuff that's being actively ran. So let's say something like log four j two happens again and systems are being exploited everywhere. Well if you have this tagging system to focus only on the active project, all you have to do is like export that list and work with the developers to try to fix that. Or you know, depending on what kind of product you're using you may be able to fix that automatically. However, sometimes there is no fix available but there's something being actively exploited. So in that case it's important to have something like a web application firewall in place. I don't really trust web application firewalls to be honest with you.

Phil Guimond [00:13:23]:
They're like a Linux dish effort I day events, especially you know, active exploitation campaign. They can help a little bit against, you know, some of the more script kid I would say newbie script kiddies. But in some cases there have been a lot of cases actually where even though you have a web application firewall in place, stuff is getting through randomly. Even simple SQL map scans are getting through. So it's important to stay as proactive as possible. Sometimes though you have to be reactive. It just shouldn't be everything, right?

Chris Lindsey [00:13:58]:
Yeah, we got to choose your battle and figure out where to focus. I was thinking about your account takeover story and the one thing that just kept reverberating in my mind, when you're using the same password over and over and over for all your systems, the problem becomes if your passwords compromised on your email as well. Guess what? I go in and I can't log in. Okay, well I'll go change my password. Guess what, that password is being sent to an account that is not you. The problem becomes the attacker or the hacker is hey guess what? They're on to me. They're trying to reset their passwords but they'll never be able to get through again. It's kind of a cyclical thing.

Chris Lindsey [00:14:38]:
The other thing that you brought up too that I really liked is your labeling idea of the repository. Because one of the things that could be extremely helpful is not just active or inactive applications, but you could also tag it as is this an internal application or an external facing application. You could also tag it as is this a console type application where something like a command injection or command execution, doesn't matter. You're already on the box.

Phil Guimond [00:15:05]:
Exactly. Exactly.

Chris Lindsey [00:15:06]:
Having those labels are extremely valuable and helpful, and I think that's an excellent idea.

Phil Guimond [00:15:12]:
Awesome. When it comes to reactive programs, it's not just the web application. It could be reactive. Internal network security. If you have an office, there's a lot of companies that have a flat network, correct? So if you're in the office, there's a flat network on there and everybody's sitting on the same network and everybody is reachable.

Chris Lindsey [00:15:33]:
Everybody's reachable. And the reality is once you're in that network, you can go east and west. And not only can you go east or west, but there's a lot of companies in today's world that's using the cloud. They're using an AWS, they're using a Google cloud, and that's tied directly into the network pretty flat. It's still outside your four walls, but it's still actually on your network. And so once you've compromised a box, or once you've compromised a password, or once you've actually looked at, if you do for any reason, have or gain access, this did happen to a couple of companies. I don't want to share names, where their credentials were compromised, and the GitHub repository, all the source code was active and they were able to pull it down. And guess what was part of that data and hard coded in the files, the access credentials to get into their cloud environment.

Phil Guimond [00:16:27]:
Indeed. I've actually seen some pretty interesting incidents where sometimes you have some network engineers who, they were under deadlines, right? They were under tight deadlines to get the network set up as soon as possible. So they were just doing everything as quickly as they could. They weren't hearing about segmentation, vlans, client isolation and stuff like that. So the problems in your network can actually affect your company, especially when you have an office. Let me give you an example. Let's say you have a guest Wifi network, right? So you set up this guest Wi Fi and you give the guest a password to login, but that guest network happens to be part of your corporate network. You've got a problem.

Chris Lindsey [00:17:10]:
That's a big problem.

Phil Guimond [00:17:11]:
That means somebody could log in through WiFi and gain access to your networks, right? They could probably gain access to your AWS accounts, depending on whether they knew about the resources you have in the cloud or not. Because in a lot of these places, you're going to be whitelisting the IP address used to access those resources.

Chris Lindsey [00:17:31]:
Absolutely.

Phil Guimond [00:17:32]:
I have seen some pretty interesting things where attackers were able to actually gain access to some hadoop clusters in the cloud simply by having their IP address whitelisted. Because even if there was clarity and isolation on that network right there, and of course you have to know where that stuff is, but all you have to do is an NMAP, right? And if it's bridged onto your network, good game. So even if it's clearly in isolation, the end IP address is still the same as the whitelisted IP address for that particular company. You have the company users and you have the Wi Fi users, all with the same IP address facing externally. So they're whitelisted. These folks were able to hit some hadroot clusters, right? So way back in the day there was an unauthenticated RCE endpoint, RCE remote code execution. So you could actually do commanding injection on an undocumented API endpoint for hadoop clusters. And then you can just spin off a bunch of nodes with crypto miners on them and all kinds of stuff.

Phil Guimond [00:18:37]:
And it was a mess. And we found ourselves constantly responding to stuff like that. Nobody knew where it was coming from, so there was no login. That's another thing. If you don't have an, as to a login, you probably have a very reactive information security program if you have nowhere to tell where these attacks are coming from. As a pen tester, you know, I can, I can tell you a million ways to get in through what I know about that network, but we don't have any proof of it because there's no login. So it's really important to make sure that you log stuff. You log like access attempts, you know, access logs and stuff like that.

Phil Guimond [00:19:16]:
And access login actually is a really, really good way of detecting when you have a compromise. You could have a whole bunch of, you know, 400 that suddenly become 200, and that shows that these people are eventually able to log in. There was probably a brute force attempt, and you can find all kinds of problems that way, but you need login, you need visibility before you can have a reactive, I mean, a proactive information security program. Without visibility, you're going to be constantly fighting fires. You're not going to be able to see where it's coming from. You might be able to treat this cause or that cause, but you won't be able to treat the root cause. It could be that somebody's computer was hacked, a developer was hacked. It could be that you included a third party library that is pushing out your environment variables to a third party server.

Phil Guimond [00:20:06]:
From an Ajax post request or something like that. There's lots of different ways. So if you have no login, you cannot find the root of the problem and you're just going to be reacting, reacting, reacting, and you're going to be stuck.

Chris Lindsey [00:20:20]:
Right.

Phil Guimond [00:20:20]:
What's important to try to become as proactive as possible.

Chris Lindsey [00:20:23]:
Well, and something I want to throw in there too about the logs is it's absolutely vital that your logs are in a position where they can't be tampered with. Because if I have access to your network, the first thing I'm going to do is go have a little fun, do a little sightseeing, see what's going on on your network, and then I'm going to clear my path. Hey look, I found the logs. I found where these are being stored. I'm either just going to wipe them all together and if you're just doing basic authentication and basic stuff that tells me as an attacker or a hacker that everything else probably within your environment is probably set to defaults or basic. The knowledge level of whoever set it up was just basic. And when you have someone that's just basic, most likely the logs can be tampered with, they can be edited, they can be modified, they can be just simply deleted.

Phil Guimond [00:21:11]:
Yeah, you know, I'm happy you brought that up. I've actually done that many times. Yeah, I've done it and I've seen it happen. So one of the more common ways you see that is in some of the old virtual machines and servers where they had all their access logs stored in var logs and stuff like that, www, HTTP, whatever, and they had their access logs stored in there. But if you had the right permission, if you were able to escalate to root and in many cases it was very easy back in the day because everybody would just put their sudo on the system and you can just do Sudo echo to see if you actually have the permissions without entering a password. Passwordless sudo. Yeah. And then you can actually just go in and edit the logs.

Phil Guimond [00:21:55]:
So you can, let's say all of the logs are going to show your internal IP address that's hitting the server and doing all this stuff. Well, you can actually just omit those entries in their entirety through grep or something and then redirect that and pipe it out and replace the log, as long as it's not locked.

Chris Lindsey [00:22:15]:
A lot of people will run these as local admin on a Windows box or run them with escalated privileges on a Linux box. And people don't think about least common privilege needs. It's one of those things going back, it's default. I don't know how to get this to run because it fails. I'll just escalate it to a local admin or escalated privileges on a Linux. Hey, guess what, it's working. I'm running as root, but guess what, I'm in. It's working.

Phil Guimond [00:22:43]:
Yeah, that's good. That happens all the time. People like to run things. As administrator. I've seen some pretty crazy things out there. Windows has like address space layout randomization data execution prevention and stuff like that. Now if you've ever played certain video games on Steam, there have been occasions where the games just keep crashing, right? They just keep crashing over and over again. It's just a problem with the games code.

Phil Guimond [00:23:09]:
So on the phone people are telling you to disable data execution prevention and ASLR and stuff like that. They tell you to go in, disable all this stuff and people do that and they forget that they've left that on, right? I mean that they've left that off. So over the years the machines gets increasingly more vulnerable as new vulnerabilities are discovered. Maybe there's a vulnerability in the game itself, but they're basically telling people to disable security protection just so they can play a game. And that sounds kind of scary.

Chris Lindsey [00:23:40]:
Think about it this way. What if somebody actually compromises or becomes a developer within that game and puts malicious work or malicious code in there? And what if we know that this is a game that a lot of network admins love to play? Guess what? They may be playing it on their work laptop, sitting in the office on the IP address that can access their clusters.

Phil Guimond [00:24:02]:
Yeah, that's fun.

Chris Lindsey [00:24:03]:
Just saying. I've seen it happen and I've seen it done.

Phil Guimond [00:24:09]:
Yeah, stuff like that happens all the time.

Chris Lindsey [00:24:12]:
So the other thing too is you had brought up dependency management, and with dependency management, think about what's going on right now with the NVD. The NVD is kind of in a paused state and because it's in a paused state, vulnerabilities are still being detected, they're still being sent into the NVD. Hey, there's this vulnerability. Due diligence is still working. And so when people are identifying vulnerabilities in third party tools or libraries, they're doing the due diligence, they're sharing the information with the developers, the developers are still fixing it. The problem is some of your free tools that you're using for vulnerability detection is just looking at the NVD and so you're going to have what's called a false negative where you actually have the problem, but because of the data. The nice thing about some of the tools out there is the tools, some of the better vendors use multiple sources. So the NVD problem is no longer an issue, but there's actually a better way to address it.

Chris Lindsey [00:25:12]:
And that is just update your dependencies. Because if you're sitting there and you're focused only on hey, I've got this dependency that needs to be updated because of something, you're already in a reactive program. If you're proactive, you're actually focused on keeping everything up to date. If you're always staying current on your dependencies, what's going to happen when the NVD does say, guess what, there's a vulnerability. It's not a version that you've already passed on, you're already running the fix. Got any thoughts on this?

Phil Guimond [00:25:43]:
Yeah, that's a good question. One of the things that I have found that works for vulnerability management, it works very well. Like you said, you know, we should always be focused on keeping the dependencies as up to date as possible, as often as we can. And if we can't update them because the project has been abandoned years ago, there's a lot of projects like that, then maybe we need to find a different one. But that may not be feasible for a lot of different, you know, projects out there. So what you would do is basically fork one of those dependencies and, you know, make your own changes to it. That's something you have to do occasionally. I don't see it happening quite too often, but you brought up a good point about the vulnerabilities.

Phil Guimond [00:26:23]:
Right now let's imagine that you have a project that has 4000 vulnerabilities that can quickly overwhelm you if you have to fix all of them, right? Yeah, but if you take a look at the classes of vulnerabilities that exist there, our data libraries, our data container images, or you have some problems with your infrastructure as code, terraform modules or whatever, but in particular those container images in the third party libraries are where you're going to find almost all of the vulnerabilities across your infrastructure. And if you've got 4000 vulnerabilities in one container image, how do you fix that? Well, there's probably a container image where if you upgrade it to either the latest version or you know, use a slim version like auto and stuff like that, you basically eliminate several thousands of vulnerabilities in one go. You eliminate all of them for a lot of these products, the open source libraries, let's say you are open source libraries are maybe one or two years out of date. You haven't updated them in quite a while. That's very common. You might find that there's maybe two or three different packages that you've included that are vulnerable and they have hundreds of vulnerabilities on their own. So simply upgrading that to the latest version gets rid of all of those and you don't have to focus on them. Unfortunately, sometimes you can't upgrade right away because some of those libraries have changes that will break your build.

Phil Guimond [00:27:50]:
So you have to do like regression testing, QA testing and all that, just to see if it works. Sometimes those projects have to be rolled back. That's when things get a little bit more reactive. But in general, these days, a lot of third party library developers are good at not having that. Now there might be some functions that are no longer being used and you have to upgrade to the new set of functions that are being included in the library, and that takes a lot of time away from the developers. But in general, when you upgrade those libraries, you knock off thousands of vulnerabilities. Same with the container images, you knock off thousands of them. So that makes it a lot more easy to manage those vulnerabilities.

Phil Guimond [00:28:32]:
And it gets even easier with the tagging system that I was mentioning earlier, where if a project hasn't been updated in like a year, it's probably not something we're running, right. And that's a long running piece of code that is basically considered perfect. I've never seen perfect code, so let me know if you find that.

Chris Lindsey [00:28:52]:
I do have stored procs and some code that I wrote back in the early two thousands still running in production today. I would be afraid to see what it looks like. I have grown just a little bit in my ability since those days. You bring up a really good point. When you were talking about the images, one of the things that we had mentioned earlier was kubernetes and pods and the ability to spin up when you have a good build process within your environment where you're downloading a good image, where you're actually being able to upgrade. One of the beautiful things about running kubernetes or pods or docker or any of those is when you've created a good foundation to work from, a good structure. You can just say, look, those pods, those images are old. We can switch it out real time with updated versions without any outage, any downtime.

Chris Lindsey [00:29:49]:
It's almost like changing a flat tire going down the highway, you spin up new pods, new images, new versions, and you just destroy the old. And if you put a time to live that's reasonable, then all of a sudden you're always rotating your images. So if you do have an image that does get compromised for any reason, it's short lived. And depending on what you're doing, maybe that's half an hour, an hour, it could just be a day. Trying to have pods or environments that are running for days, weeks, months is just a bad design.

Phil Guimond [00:30:22]:
That's a good point.

Chris Lindsey [00:30:23]:
The other thing that you're also dealing with in reactive is gaps. Because one of the things that when I think about different leaders that I've talked to over the years, their programs, if you're reactive, you don't know what your gaps are. I don't know, good luck. But as you start putting it together and you start putting your program in a position to be successful, now all of a sudden your gaps are becoming more and more known. We have this environment, we're not doing anything on it, we're not monitoring it, we're not doing anything. We should probably put some effort into that. And from a development standpoint, as we're looking at our dependency management, as we're starting to roll more into a proactive approach where we're keeping things current, we're looking at the images, we're using tools. Tools are great.

Chris Lindsey [00:31:17]:
As an assistant to the developer, a tool is not going to solve your problems. A tool will bring your problems to light and it's the developers and training on the developers that really make the difference. If you have developers who are not security focused or minded, then the problem becomes those developers are going to just continue writing insecure code. The tools are going to show, hey, you went from twelve issues to 20 issues to 30 issues to 40 and you may be focused on, hey, fix this one or fix that one. But if they're duplicating the code on their side, they're only just duplicating and extending the problem. And going back to the compromise that you were talking about earlier where you may have a library that's compromising you, you may have removed the code to use that library, but somebody grabbed old dead code that's still in the system and brought it back to life.

Phil Guimond [00:32:14]:
That's a good point. I like the mention of gaps, you know, because when you're in a reactive program, you will find gaps everywhere. You'll be finding gaps constantly. It's important to close as many gaps as you can. Within the budget you have. Of course, you know, sometimes you'll have to make some sacrifices because no information security program is a perfect budget. So you have to make do with what you've got. But I have found that when you focus on eliminating as many gaps as you can and automating things and tying them into either a single dashboard, if you can, you know, that single pane of glass that everybody likes to talk about is actually a pain in the ass.

Phil Guimond [00:32:52]:
But as long as the security products that you're using have a good API, you will be able to get all of the information that you see in the dashboard from the API as well. And then you can just build your own dashboards around it and dump that into one place so that everybody can see that. And there are some tours coming out now that are doing that for you. I won't discuss which ones they are, but they seem to be doing a pretty good job.

Chris Lindsey [00:33:18]:
So I think they're definitely getting better.

Phil Guimond [00:33:21]:
They're definitely getting better.

Chris Lindsey [00:33:22]:
One thing that you may have in an environment where you've got tons of gaps, where you may have tons of technical debt, tons of security debt, you're going to have high turnover for your staff, and there's a lot of leaders. Here's the most amazing thing that's really just kind of hit home this week for me, and that is a lot of leadership have talked about only having two or three security guys running the whole program. And when you're in a reactive fight, you can only go so long if you're sitting there stuck on your compromise story is earlier, spending one, two, three, the whole week, two weeks, three weeks, four weeks on the same problem and you're on that treadmill that never ends. The problem becomes at some point you're just like, I'm done, I'm leaving. Somebody else can come clean up this mess.

Phil Guimond [00:34:17]:
Yeah. And if you have a smaller team and you have a team of go getters, people who want to solve problems, it's much easier to develop automated solution to get around that kind of thing. And that could help you develop a less reactive program. But of course you have to have a balance. I have seen information security teams that just get too big. Nobody knows what the other person is doing. You know, around and around. Everybody gets to run around and nothing gets done.

Phil Guimond [00:34:46]:
I have seen that happen in the past and I have also seen much smaller teams contribute to a lot and get a lot more done.

Chris Lindsey [00:34:55]:
The interesting thing, and you and I, our programs that I ran, that you run as a developer, the beautiful thing is, you're okay. I have a gap, I have a need, I'll just code it, I'll make it work. Because knowing API, knowing development, you can actually just tie things together and just make it work. Security vendors know what they think customers want and they go after that. They feel like they've got a pulse on what's going on. But again, there's a lot of edge cases that you're going to run into in your program. And with those edge cases being able to code around it, to hit APIs, to pull it together. For me, one of the things that I did was I sent an email to every developer on commit and they knew the security for what they just worked on right in their inbox.

Chris Lindsey [00:35:43]:
There's no excuse whatsoever. Some of the tools out there do that, some don't. And for bringing tools or bringing, gosh, I hate to say the word single pane of glass, it's one of those things where it's nice to be able to see on a dashboard high level, what does my security posture look like? Because if you can pinpoint and say, I have all these criticalities over on this application, this application is external facing, this may be an API that your mobile device is talking to all the time. You know, those are things that you got to work on. But if it's something that is internal and even though it's critical, you can use some of these tools to kind of help minimize your chasing after by saying, this is an internal tool, this is something that's only used on the command line. Therefore, a lot of these vulnerabilities that are being detected are, yes, they're vulnerable, but they're not going to matter in my context.

Phil Guimond [00:36:47]:
Yeah, you know, I'm glad you brought that up. I think it's really important for information security teams to have assured programming on the team. It's like, you know, security software engineers, not everybody will work in software, but when you have a smaller team and a smaller budget, hiring security professionals who can actually write code and fix those solutions is game changing, because otherwise you have these tools that you buy and once the tool can't do what they need, they get stuck, or they're probably in a bunch of meetings with the vendor about, oh, we don't have this feature, we don't need this feature, we need that feature. But if the vendor has a good API, you can just code that yourself.

Chris Lindsey [00:37:26]:
Exactly. I mean, I ran into that multiple times with my program and having the API endpoints, I mean, it's life changing because to your point, it doesn't exist within the tool. That's fine. It does in the API. I can do something about it.

Phil Guimond [00:37:43]:
That's a good point. You know, I'm still doing that quite often. I'm running into a bunch of use cases I just not covered by some of the security products that I use. And it's funny because the security product is supposed to make everybody's lives easier, but in some cases that makes it harder, right? A lot harder. Not just for the developers, but for the information security team because it doesn't have the features that you need. So you find yourself constantly building those features yourself. It's just the way it works in this industry most of the time.

Chris Lindsey [00:38:13]:
Yeah, it's true. And you had to find a good application security director leader who understands application security is hard to do and it's vital. And so companies that have the right people, the good people on their staff, they need to do what they can to keep them on board. Because if the board of directors is just constantly pushing back, or I, your directors that you are underneath, are pushing you all the time. Hey, this and this, there's not many good people out there. And when you find somebody that's good, you got to hold on to them. I'd like to go ahead and conclude us here simply because you and I could probably talk for hours. And our listeners, as much as I bet they're enjoying this conversation, they probably looking at the clock going, it's time for lunch, or I, I've got to hop off this thing because I need to go eat.

Chris Lindsey [00:39:10]:
So, Phil, I appreciate your time today. I appreciate you talking about this.

Phil Guimond [00:39:15]:
Thanks for having me.

Chris Lindsey [00:39:16]:
It's such a deep topic and there's so much to it, from the account takeovers to running in circles, trying to close the gaps, trying to get things to work well and to just have a good program. And again, good programs have developers on it. They have people who care, who's passionate. And you're one of those guys. Absolutely. I can see the passion and, you know, the fact that you care. And so, Phil, thank you for coming on today's show.

Phil Guimond [00:39:45]:
Thank you so much for having me. Had a great time.

Chris Lindsey [00:39:50]:
Thank you so much for joining me on this episode of Secrets of Appsec Champions. If you found this valuable, hit that subscribe button on Apple Podcasts, podcast, Spotify, or wherever you get your podcast. And hey, ratings and reviews are like gold force. So if you're feeling generous, please leave a kind word. It helps others discover our show. Until next time, take care.