Chris Lindsey [00:00:00]:
And then also when it all kind of hums together nicely, then, you know, as the security group up above the four, the five, the six people running the whole program, it gives you the ability to sleep better at night.
Shashank Balasubramanian [00:00:14]:
Yeah.
Chris Lindsey [00:00:14]:
And as a matter of fact, I call it security is boring at 05:00 on Friday because everything's going right. If security is not boring in your environment, you're either not doing something right or you're not visible to something very big that's happening that you're unaware of.
Shashank Balasubramanian [00:00:33]:
100%. I completely agree with you there.
Chris Lindsey [00:00:35]:
Very cool. Can you think of anything else that would make a program that's proactive even better?
Shashank Balasubramanian [00:00:42]:
This is something which, again, as you build your apps at program, there are different levels of maturity as you reach that. It's great if you start out by. Even if you have, like, tools offline, which are scanning offline, and then you're basically scanning production stuff, and then you sort of slowly shift left and you have CI CD tools.
Chris Lindsey [00:01:03]:
Hello, and welcome to Secrets of AppSec Champions. Today we are going to talk with Shashank, who runs the application security program over at Tripadvisor. We're going to have an amazing conversation. Shashank, please introduce yourself.
Shashank Balasubramanian [00:01:19]:
Hey, Chris. Thanks for inviting me. My name is Shashank. I have been running the application security program here at Tripadvisor for a good amount of, like, four, four and a half, five years. And what's really interesting is that, you know, I've had that opportunity to sort of build my own program, essentially start from really nothing, to build something which is, in my experience, proactive and able to sort of weather the different issues that we face. So, really excited to talk about application security. Very pumped up about application security. And so, yeah, looking forward to it.
Chris Lindsey [00:01:50]:
So let me get started. All right, so your environment is on fire. It does not matter what you're doing. Anything that you do, any progress you're trying to make is just not going anywhere. That's a reactive environment. That's a reactive program. Moving to a proactive environment is where you're actually seeing things get done. You're actually moving the needle.
Chris Lindsey [00:02:09]:
You're making things better for everybody. So in today's conversation, we're going to talk about what does a reactive program look like? But we're also going to talk about the successes for a proactive program. With that, let's get started. So, Shashank, let's talk. Have you been in a reactive environment before?
Shashank Balasubramanian [00:02:28]:
I would say yes. And typically what I see for most professionals is that when they're really building a program from ground up, that's where you start out with, because you don't have all the capabilities deployed, you just don't have it at that point of time. And so you're really working yourself backwards from having nothing at all to sort of then slowly building up the program. In my mind when I'm thinking reactive, it's really kind of playing catch up with whatever's going on in your environment. And so whether it's applications or scanning or defects and triaging these things, it's all happening after the fact. Maybe applications deploy to production and you're just scanning stuff in production. You just don't have any control over the software supply chain development cycle early on, like in the design phase or whatever. And so that's how you start out.
Shashank Balasubramanian [00:03:18]:
I started out similarly and then sort of went my way backwards and sort of added more capabilities along the way in my whole journey in this appsec journey. And so, yeah, I would definitely say if you compare and contrast reactive to proactive, one of the biggest changes you will find is that it's much less load on the security team to sort of stay on top of everything that's going on, because a lot of the things you want to actually track would either be done through some of the tools that you deploy, but more importantly through your relationships with the development organization at large. And so I basically am of the opinion that you will automatically see the difference between a reactive program and a productive program as you sort of make that transition where you're getting more bang for the buck for your program and sort of lessening the stress around security, because application security is vast and there's a lot of things to look at. And so, yeah, you just need to make sure that your team's not burnt out either.
Chris Lindsey [00:04:13]:
Right? Application security is not an easy area, to be honest with you. A lot of leaders that I've spoken with over the years, it's a struggle because there's so much to it. And you're right, when you're starting your program, everything is reactive because, you know a system will go down or something will happen and you just won't know why something's going on. I don't know why. And the networking guys come to you and they're explaining, hey, we're seeing weird traffic and you just have no visibility. And as you start putting the tooling in place, you start having more and more visibility, and all of a sudden you have that aha moment. Have you been in that spot where you turned a tool on and you had that big aha moment.
Shashank Balasubramanian [00:04:51]:
So we've deployed a lot of tools specific to appsec in the past. And so I think the one which I want to talk about is when we sort of, so when we look at developing a product right, in a reactive environment, looking at what developers are doing day in and day out is not necessarily part of the equation, right, very far removed from that process. But one of the initial things that we did was that we installed a tool which basically scans for third party vulnerabilities like a software composition analysis tool. And we shifted it left in the sense that it wasn't in the CI CD pipeline. And so for me the AHA moment was really having the devs who are working in that environment see the vulnerability, know about the vulnerability and take action on it at that point of time. And I was like, oh, I don't have to do all this work. Later on in this, in the game we can actually proactively fix all of this where you have a good relationship with the devs. So I think that scale that you get not just by deploying any random tool, but by deploying to the right place, to the right audience, is something which will definitely help scale out your program in a very nice way.
Chris Lindsey [00:06:01]:
Right. Well, in the benefit of doing an sea as a tool first is it's not only taking care of security debt that you're unaware of, but it also looks at the developer technology debt. So from a development standpoint, the developers will feel better knowing that we're running or they're running updated libraries and knowing that they're current. So they're getting all the features and all the benefits and any bugs that may have happened with any of the versions that they're running. In my background, I was a developer myself and there were times where the library had bugs and you just worked around it, think, hey, let's go grab the latest and greatest version because you didn't know. Can I just drop this in? And is it going to work or is it going to break a lot of things? And so with a good tool that it gives you a lot more visibility into that.
Shashank Balasubramanian [00:06:48]:
Yep, that definitely makes sense. I'm just going to go back to just, if you're thinking about return on investment rate, if I install this tool, what return am I going to get? Installing tools in the CI CD pipeline is going to really reap good dividends for you long term because bear in mind, one aspect of it is exposing to devs and sort of making sure that they're fixing it. But another sort of hidden benefit of this, which you won't see at the surface, is that devs are learning as they're fixing these issues so they're not repeating these mistakes again and again. And so what you eventually see with successful appsec programs is that initially you might see a bunch of stuff surface up, but then over a period of time that whole thing matures and you start seeing lesser bugs and you start seeing developers being more security aware and coding more securely, which in turn is exactly what you want from your appsite program.
Chris Lindsey [00:07:40]:
Right. When developers know that you're actually on their side and you're trying to work along with them and help them be successful, then your program really, it goes a lot smoother just simply because there's not that friction of, yeah, here's something, you know, starting with an sea is an excellent starting point. It's low hanging fruit, but it also gives you the ability to go in there and work in their area. To say, this is something that should be upgraded because you're running 13 versions behind or two versions behind or whatnot. And so it's not like you're coming to them and saying you've got all these bugs and these problems and these bad security posture and you're going to hurt the company. You're saying, hey, let's do this right and keep a good, clean, tidy house on the development side.
Shashank Balasubramanian [00:08:26]:
And I'll add one more point to this, right? So when you're coming in new, right. When you're coming in from a reactive environment, you might not necessarily have that relationship with the dev organization at large, right? Like maybe your app set team doesn't know who the dev organization is, right. And so these are the ways where, you know, you can actually, once you have the findings of maybe an SCA result, you go ahead and approach the teams and then strike up that conversation because that's what's going to lead to building up that relationship where, you know, they trust the security team coming. It's not like we're always coming to complain about the code or anything of that sort. It should be a meaningful conversation. Right. And so that's also one thing I will highly stress, is that, you know, even in your outreach to the development community or whoever it is, right. Whoever it reaches towards, right.
Shashank Balasubramanian [00:09:13]:
Try to keep it specific and sort of make sure that you are. So we are the subject matter experts from a security perspective, so we should be aware of what the ramifications are, how to fix it and what the risk is and stuff like that. And then sort of have that conversation with the developers and then win that trust. Once you win that trust, trust me, they will be ambassadors of security, not just for their project or whatever, but developers talk to each other, right? And so they will be like, hey, you know what, this tool is pretty cool and this is a new thing I learned and that will grow. Right. The culture, it's more about security first, culture within the larger developer.
Chris Lindsey [00:09:50]:
Right. You hit a couple of key things that I want to kind of surface back up to. You talked about the developer and security relationships. If you have a good relationship, you're going to be successful, just like you were mentioning. The other thing you mentioned too is developer training. As they grow in their knowledge of security, they're going to be learning, they're going to create good best practices internally. They're going to say, hey, we've got a new version coming out of this application we should go look at. Are there any updates to the libraries or dependencies? The tools will do that for you.
Chris Lindsey [00:10:23]:
And some tools will even create the pull requests for you to kind of help you keep current. So that's an excellent start. Where would you look for next in your program?
Shashank Balasubramanian [00:10:33]:
The tool can get you so far. It can sort of scan everything and create the pull request. Right. The other side to that is that where securities work is not over is basically, I always stress that, you know, we need to sort of have, even though the devs are watching everything that's happening and it's visible, it's bubbling up to them, we need to have an overall visibility in terms of, you know, how many defects are they actually fixing, are we reducing our risk profile and stuff like that. Right. So the follow up aspect of that is constant. Right. It never ends where, you know, we're constantly looking at what kind of criticals and highs and whatever, we want to track how that's trending because it's not like we don't trust the development team to fix it.
Shashank Balasubramanian [00:11:14]:
But keep in mind, the developer's primary job is to ship features and write code to ship features which are new products or innovations in that area. And so security might take a backseat in that kind of a situation in terms of priorities. So it's equally important that, yes, it's very important to surface those to the developers, make them aware, train them and everything, but again, have that guiding hand or whatever, advising them, constantly following up with them and making sure that, yeah, you know about this, what is the plan to fix it? Again, we don't necessarily barge in and say fix it.
Chris Lindsey [00:11:49]:
Right?
Shashank Balasubramanian [00:11:49]:
Now. But then we try to sort of what your risk appetite is and you know, try to reason with them in terms of if you don't fix this, this is the side effect of that. And more often than not developers are very reasonable. They're not very adamant in the sense that I won't fix something because I don't want to fix it. It's like if you have a decent enough conversation with them and if you explain it to them reasonably, in my experience, I haven't even seen a single developer who will say, yeah, I don't want to fix it because I don't want to fix it. It's like maybe I have other priorities or whatever, but eventually everyone fixes these things, right?
Chris Lindsey [00:12:20]:
Well, and having a good foundation like that is excellent for building a good successful program because the programs where the security team is really pushing the development team to do stupid things. And what I mean by stupid things is I have a list of 40 vulnerabilities. They're high, they're critical. You're going to go fix this right now the problem becomes, in our pre call we talked about this, so I'll bring it up here, but it's a matter of your environment. And where does that vulnerability exist? If it's a vulnerability that is sitting internally, that is only accessible internally, that's sitting on the command line or the terminal window inside Linux, it doesn't matter if you have a high critical command execution or injection issue because you're already there. And so having a good level head on your shoulders when you're looking at your security program is key because going to the developers with something like that, they're just going to look at you and just shake their head. And now you're going to build up a wall, whereas you come to them with something that's actionable, something that's actually relevant and something that's critical. If it's really super critical, then you're going to go to them immediately, you're going to explain it, you'll even show them, why is this critical? Even show them, possibly record it with a pen testing tool like burp suite or something and play it back to them.
Chris Lindsey [00:13:38]:
This is why it's critical. This is why we need to do something versus, hey, guess what? I have a list of 300 of them. Can you get them done by next week?
Shashank Balasubramanian [00:13:47]:
No, I completely agree with what you're saying. I'm reminded of the time when like for instance, the zero day comes along. Let's take for instance lock four j, right? These are the kinds of times when these kinds of decisions need to be made. Right. Because I remember, right. I think Lockforger came over kind of like the Christmas weekend. Last part of the year.
Chris Lindsey [00:14:04]:
It was late November.
Shashank Balasubramanian [00:14:05]:
A lot of the folks would be out on vacation. And, you know, you don't necessarily want to call everyone up and say, come online and sort of fix this issue for me because it's a zero day. If it's public facing. Okay, maybe that bumps up the criticality. But again, as you mentioned, if it's sitting behind a VPN and if it's guarded relatively well from that perspective, then maybe you can come back, Jan or something. You can come back and take a look at it. So it's those kinds of decisions where it's not knee jerk reactions to just hear some news, something crazy is happening, and then you just rush to the dev desk and say, hey, we need to get this fixed. Yeah, I completely agree with what you're saying.
Shashank Balasubramanian [00:14:39]:
It's a trust relationship. It's just not sort of asking for something to get done for you.
Chris Lindsey [00:14:44]:
Yes. So when you are in a proactive program and you're actually working very well with the devs, one of the nice side effects that I saw when I ran my program was the development team would ask me questions about security. They would come to me and they would say, I'm working on a mobile app. I think we should probably have a lock screen. Is a thumbprint enough to unlock it? What do you think?
Shashank Balasubramanian [00:15:07]:
I think that's a great sign that your program is actually maturing. And I'll pre face it with this. Right. So usually, and I'll compare a reactive program with this proactive right here. Usually in a reactive program, what you find is that the security team is actually reaching out to the developers. A lot of the outreach is security reaches out to engineering. What happens in a proactive program, and it's great when it happens, is that the dev team will automatically start reaching out to security for various matters. And that's what happens when you mature a program.
Shashank Balasubramanian [00:15:36]:
And so to that point. Right. They will come to you with questions and recommendations on, like, something that you post. Right. And so I think it's a great sign because in certain areas specifically related to security, we have expertise in that particular area. And I think it's very important that, you know, devs take in that feedback when they're sort of designing various applications. Right, sure. I mean, you can build everything on your own, but then us coming and doing a security review at the end, once it's in production once it's published, the cost to make those changes, I mean, everyone's aware of the cost of fixing bugs along the production cycle.
Shashank Balasubramanian [00:16:11]:
It just gets more expensive. Right. And so if you sort of reach out, and again, this actually goes back for me when I think of like, so SDLC, we have SDLC, but when I think of secure SDLC, these kinds of things are what really sort of help you along the way. Right. Where if the devs are reaching out to you with recommendations and advice and for a consultant or whatever, I think that's perfect. I mean, that's how products should be built. And the truth is it's not happening nearly enough. But when it does happen, it's really nice.
Shashank Balasubramanian [00:16:42]:
Right. And I'm all for it. So, yeah, I would give two thumbs up if that's happening in your organization.
Chris Lindsey [00:16:47]:
Yes. And what do you think about security champion programs? Do you have any thoughts there?
Shashank Balasubramanian [00:16:51]:
Very important. Right. So again, this goes back to the relationship, right. The relationship between security and development or engineering. So for me, right, when I'm thinking of champions, right, I don't sort of cast a wide net and say all developers are champions or something of that sort, right. I want my champion to be shortly security minded. But really the way I look at champions is a way to scale out my operation, right? I don't have 100 appsec engineers on my team. And so most security teams are small.
Shashank Balasubramanian [00:17:21]:
We have like five or six people within a team. If you look at the application landscape, we're looking at hundreds of applications and you're looking like 1000 plus member dev organizations. And so champions are really the way you effectively scale out your operation because you've given them some sort of training. Usually they have a security bent of mind, which is why they're attracted to the champions program in general. And so you've given them the right training. You have frequent checkpoints with them. I mean, it's just not like, you know, you give them training and you forget about it, right? You need to have good assignments for them. Maybe.
Shashank Balasubramanian [00:17:54]:
We touch base very frequently. We discuss various topics. What are you hearing on the ground? They would have exposure to some new things coming up. And we are like, yeah, tell us about it. Because we cannot be at all places at all times. Basically, they're just ambassadors of security within various points within the dev organization. And so I think it's tremendously important to have or at least start thinking about having a good security champions program because that's the only way you're going to scale your program. That's really the only way you're going to get better visibility over what's really happening out there because you cannot really individually, you cannot have access to everything.
Shashank Balasubramanian [00:18:31]:
So I think it's hugely important.
Chris Lindsey [00:18:33]:
Right. Also with part of the having a champion program is also bringing developers into the security fold. And it's hard. There's not very many developers who are also very deep in appsec. And so to be able to still have that developer come in and as they grow and they learn in security, they're also starting to be able to think about the criticality and how it doesn't affect the environment, and they start thinking about and asking the questions. And the beautiful thing about having a champion program too, is they're sitting in the scrums for the development teams, possibly the senior tech reviews, and they're asking the questions to QA. Hey, this is great. It's passing.
Chris Lindsey [00:19:17]:
I have a couple of checks I want to do, or maybe I want to do some automation, like dast is part of the QA process. That way we have an idea from a security standpoint there kind of an automation regression from a security standpoint, which is great. And then also when it all comes together nicely, then, you know, as the security group up above the four, the five, the six people running the whole program, it gives you the ability to sleep better. Better at night.
Shashank Balasubramanian [00:19:45]:
Yeah.
Chris Lindsey [00:19:46]:
And as a matter of fact, I call it security's boring at 05:00 on Friday because everything's going right. If security is not boring in your environment, you're either not doing something right or you're not visible to something very big that's happening that you're unaware of.
Shashank Balasubramanian [00:20:04]:
100%. I completely agree with you there.
Chris Lindsey [00:20:06]:
Very cool. Can you think of anything else that would make a program that's proactive even better?
Shashank Balasubramanian [00:20:14]:
This is something which, again, as you build your apps at program, there are different levels of maturity as you reach there. It's great if you start out by, even if you have tools offline, which are scanning offline, and then you're basically scanning production stuff, and then you slowly shift left and you have CI CD tools. But I think real value where you can actually proactively push forward the agenda of security is if you have that mindset of product security as a whole in general. Right. And so that just doesn't mean, you know, scanning where devs are sort of developing. You need to move further left where you're talking about design. You may be working with products, say, for instance, and you're working with maybe things like threat modeling and sort of looking at architectures, I mean, very early on in the life cycle of an application. Right.
Shashank Balasubramanian [00:21:03]:
And so I think if you get that, and if you get that right where, you know, right. Talking with the right stakeholders, I'm assuming that, you know, you have a mechanism in place where, you know, maybe you have a seat at the table where you are getting that visibility into any of the new developments coming on. Right, right. So I think having a conversation at that point and having a talk about security at the design phase is ideal. And so I would definitely answer it that way where, you know, the earlier you can actually talk security with people who understand it, the better it is. And that's ideal for a proactive program.
Chris Lindsey [00:21:38]:
Awesome. And one thing I want to share that worked very well with my program is instead of going left, going right, you're actually going up, up to the executives and sharing and building reports and sharing the threat landscape with the executives, with your CISO, with their boss, and all the way up to potentially even the board and CEO, that visibility, when they have the visibility into your program, they know, hey, we are really trying to be secure. They're going to sleep better and they're going to feel better knowing that people aren't going to be able to steal the data, our data, without us realizing it. And it's one of those things where executives do not like surprises. A security incident is a surprise for a lot of people, and it's one that you try to do everything you can avoid, really. The thing with security is it's not a matter of if, it's a matter of when and when you're doing the right security and kicking the can down the road. Hopefully it's not going to happen under your watch. Hopefully it's not going to happen under the next person's watch.
Chris Lindsey [00:22:43]:
Hopefully it will never happen. But also, depending on what you are in the industry, size wise, you can become a target or not. You just always have to be prepared for the unexpected. And just all the different things that you've talked about and we've talked about today is really just kind of helping, ensuring that, you know, the basics, you know, starting with the low hanging fruit, the seas, and then kind of working up and looking at the different things, the relationships, when you have that relationship with the developers, things just go a lot smoother and it just kind of makes for a great program. So let me ask you one final question before we go. And really, I'm just going to make this wide open. I know I kind of threw it out a little bit, but is there anything else that you want to throw in there for advice to somebody that's small, somebody just getting started or somebody that's writing a program and think about kind of one of the biggest hurdles that you ran into and maybe some, a bit of advice that you might share to somebody and say, hey, welcome, Tapsec. Here's something that might help you.
Shashank Balasubramanian [00:23:43]:
Right. And that's equally important. Right. I mean, it's nice to sort of discuss about what mature programs do and what kind of tooling is available. Let's assume you have nothing on the plate and you're coming in brand new, you're starting the program from ground up. I think one thing you should look to implement some sort of tooling which would assist you in sort of making sense of what kind of things you're looking at. Right. And one of the first things you'll see is, right, you might not have the budget to buy the best tool out there and stuff like that.
Shashank Balasubramanian [00:24:11]:
So what I recommend is basically start looking at like open source stuff, free stuff, that a lot of that stuff is available and they have very good integrations with most of the modern frameworks. Start by leveraging that. It might be a bit more painful to implement day to day, but once you have the dials and controls in place, you've actually, as part of setting this up, you've built up the knowledge to assimilate a tool like this. Because when you go to a paid tool or whatever, it's just a matter of taking those feeds or taking those findings and running with it. So I would say start off with open source. The other thing I would say is, you know, tools are great, but, you know, I would want to emphasize pen testing is actually something which is, I don't want to say underrated, but it's very, very useful when you use it in the right way. Right. So you might have some critical applications in your organization, and if you don't have local pen testers on the team, I would definitely encourage you to sort of leverage some ways to either a third party vendor who can do pen testing for you.
Shashank Balasubramanian [00:25:16]:
But even better is that, you know, if you could participate in sort of a public bug bounty program or even a private bug bounty program where, you know, you're exposing an application to a bunch of researchers, that will give you tremendous bang for the buck. Right. I mean, that will actually almost compensate for your lack of having a tool out there, which is looking at everything, because you have a bunch of hackers who are looking at your website and sort of hitting against it. I mean, just two of the suggestions to sort of start out and then I'm sure that as you sort of start building the program, you will find your sweet spot in terms of what next to focus on and what to sort of put in place in order to have good visibility and control over that.
Chris Lindsey [00:25:55]:
I absolutely agree. I think to your point, pen testers are kind of one of those necessity items at some point within your program because the value they bring, they can. When you have a finding, before you go to the development team and say, hey, there's this critical finding, you can actually go pen test it. You can go attack it yourself and just validate it. And if you can do it yourself and you have a good pen tester, then you know this is something that's there. And if you go to them and your development team and say, look, we have a finding, it's critical. I need a hotfix. I need it today.
Chris Lindsey [00:26:27]:
Here's the rationale, here's the video or the details behind it. 99% of the time going to say thank you, we'll get right on it. Or that's when you know you're reactive, when they're saying, go away.
Shashank Balasubramanian [00:26:43]:
I hope not. But yeah, I agree with you.
Chris Lindsey [00:26:46]:
All right. So, Shashank, thank you so much for coming. The conversation we had was absolutely amazing. The details, the insight. Knowing what a good, proactive program looks like is absolutely critical for success. And I appreciate you coming today and I appreciate your insights.
Shashank Balasubramanian [00:27:03]:
Thank you. Thank you for having me.
Chris Lindsey [00:27:05]:
And for the rest of us, have a good rest of the day. Thank you so much for joining me on this episode of Secrets of Appsec Champions. If you found this valuable, hit that subscribe button on Apple Podcasts, Spotify, or wherever you get your podcast. And hey, ratings and reviews are like gold for so if you're feeling generous, please leave a kind word. It helps others discover our show. Until next time, take care.
