Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.
Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That’s S-Y-M-O-P-S.com/corey
Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. My guest today is Jess Dodson, who’s a Senior Cloud Solution Architect at Microsoft. Jess, thank you for joining me. We have been passing like ships in the night on social media for years now. It is so good to finally talk to you.
Jess: Lovely to talk to you in person. Thank you for inviting me on.
Corey: Well, to be clear, we’re talking remotely when we record this. You are presumably Australian, and I’m still operating from a somewhat American-centric viewpoint that more or less everything in Australia is deeply poisonous.
Jess: Yeah, that includes me. Yes. So, I am in Australia at the moment. I believe it is the ninth of March for you. It is the 10th of March for me [laugh].
Corey: Yes, some of us are living in the future.
Jess: The future. So yes, it’s about seven o’clock in the morning for me, which is fabulous. I’m awake, I’m awake.
Corey: So, let’s talk about security. It seems to be top-of-mind and everyone’s talking about it. Unfortunately, it seems that they’re usually talking about it in the form of an email that starts with, “Your security is extremely important to us,” and then transitions into, “Here’s how we dropped the ball on it.” I was once told by an analyst client of mine that I was the only analyst who ever told them that companies don’t care about security. Like, “No one says that. Why is that?” And my answer was, “Well, no one will say it out loud, but I ignore what people say, I pay attention to what they do, and where they spend the money, and it is clearly not a priority.”
And I would argue that in some ways, that’s okay, depending upon context and who you are, and what you do. And in other cases, it is the exact opposite of okay; it is an unmitigated disaster for everyone, just waiting to happen. How do you feel about security?
Jess: A very loaded question.
Corey: Isn’t it though?
Jess: [laugh]. [crosstalk 00:02:20]—
Corey: “I don’t care,” is probably not going to be your answer on this one, I’m just going to assume, but let’s go.
Jess: No. Don’t—well, considering my title, I would hope not, considering the work that I do.
Corey: You have words in your title, but none of them are, “Cares about,” so we’re good.
Jess: That’s true. That’s true. I do care about security and that’s my job. I do agree that organizations probably don’t care about security until they do care about security, and by that point, it’s probably too late. And that’s the issue that we’re facing. It’s that proactive versus reactive. Reactive is great. Reactive is wonderful. But unless you’re doing the proactive and spending the money on the proactive, the reactive is just constantly going to be fighting fires.
Corey: It’s two classes of problems. My world of cost optimization absolutely is in this world as well. Security is buying fire insurance in case the building burns down. These are all things you should probably do, but you can spend infinite money and effort and time and all of those things and it doesn’t get you one iota closer to your business goals, whereas speeding time-to-market, launching into new markets, and being able to ship faster, that is transformative that gets you to your next business milestone. So, companies will overwhelmingly bias towards investing in that and not the other stuff until they’re caught flat-footed.
Jess: And I’m sure that’s wonderful, but my issue when I come in as a security person to an organization is, what if something goes wrong and it’s no longer—and I hate using buzzwords, but—when we’re thinking about zero trust, one of the principles around zero trust is assuming breach, which is, it’s not a matter of if but when, and it’s not a matter of when but it’s happening right now. Because as blue teamers, which is what I think of myself am, we have to plug all of the holes, we’ve got to try and patch all of the defenses, we’ve got to be the ones who are blocking out every attempt. An attacker only has to succeed once. And so, the way that we see it is there is going to be something in your environment, so what are you doing to make sure that that is contained as much as possible? And that’s proactive work and that’s something I don’t think you can skimp on.
Corey: It is. And it’s defense-in-depth. You can also turn it into business value if you’re clever enough. For example, whenever a cloud provider releases a new service that I can’t figure out how to configure all I do is, I wind up scoping a security role to just that service and then leaking credentials online. I wait, you know, 20 minutes for someone to exploit it and then configure the thing, presumably to mine Bitcoin. Great. Then I turn off the Bitcoin stuff and I take the config that they’ve built and there we go. It’s how we outsource intelligently on a budget. Uh, professional advice: please don’t do that.
Jess: [laugh]. I was going to say, that’s certainly a unique way of configuring services in the cloud. I’m not sure I would recommend it for everyone, but for you, I can totally see that working [laugh].
Corey: Yeah, I learned it from my buddy. He works at a bank. What of it? Yeah, it doesn’t go well.
Jess: [laugh]. When it comes to all of the stuff—and I think that’s probably one of the big issues that we have with the cloud, and I love the fact that the title of this is Screaming in the Cloud because we’re looking at all of the stuff that keeps coming out, everything is changing so quickly, how do you secure it when you don’t know from one week to the next what new services are going to be included, what changes are going to be made to the services that you are already currently using? How do you keep up to date with that? And I think that is what leads to security being seen as ‘the no people,’ which I hate. Security shouldn’t be ‘no.’ Security should be ‘yes, but.’
It also leads to, hopefully, our operations teams being a little bit more on the ball when it comes to that security. Because if they’re the ones who are looking at putting these new services or new productivity features in place, they should be vetting them from a security perspective as well. I say should. Maybe not necessarily actually happening and that’s a bridge we kind of need to cross at the moment.
Corey: A couple of years ago, I looked around the ecosystem, trying to find a weekly AWS-centric security newsletter, and there are some great ones now, don’t get me wrong. And some of them might have existed at the time, but I didn’t trip over them for one reason or another. And they tended to all fall into two buckets. Either they were security people talking to security people with a bunch of acronyms that I wasn’t tracking because I don’t eat, sleep, and breathe security most days, and/or they were vendor-captured and everything was, “See how terrible it is. That’s why you should buy our widget.” And it almost doesn’t matter who the vendor is.
So, I started a Thursday issue of the newsletter that I write just for the news that is security-centric for people who don’t have the word security in their job title. So, all the DevOps people of the world, the folks who are building applications, the folks who do have to care about this, but they don’t have time to filter through all of the noise that everyone’s putting out, and what is the stuff I should actually be paying attention to this week? And that seems to have struck a nerve, on some level. The thing I’m continually testing and being pleasantly slash unpleasantly surprised about is that I’m rarely the only person who has a particular problem.
Jess: And you’re definitely not in that regard. And I think when we look at security and how security is marketed, it is very pushed towards CISOs and security analysts and security operations. I, most of my life, and I still regard myself as, a recovering systems administrator. I’m a sysadmin, at heart, so I come from an operations background. The work that I’ve done in operations is what feeds the work that I do in security because I knew how worked, I helped build those systems.
And I don’t see it purely through a security lens; I see it through an operations lens. Security is useless without some form of usability. And I constantly talk about the line that divides security and usability, and where that line gets drawn. And for each organization, it’s going to be different depending on their risk, depending on their business profile, what they’re actually doing, how big their teams are. And we’ve got to make sure that we get that line right.
And that means getting your operations teams involved because they know how their stuff has to work. They know what they need to be able to make the business run. So, security can’t just be constantly saying, “No, you can’t do that.” We have to be working with operations teams, and in some cases, taking our lead from operations teams because at the end of the day, they know this stuff better than we do. So, it’s us providing advice and then providing their expertise for us to come together to a joint agreement on how to secure the thing. And I don’t think that’s being done at the moment. I call it the operations sandpit. Everyone needs to play nicely in the sandpit. No one should be fighting over toys. Everyone should be playing with each other with no arguments. We’re not there yet.
Corey: When I read a lot of security researchers and the stuff that they’re focusing on in stupendous depth, or I walk around expo halls at security conferences, making fun of the various vendors there, it seems like so many of them are talking about these incredibly intricate in-depth defenses against attacks that seem relatively esoteric. And then I fly home from the security conference that I’m at and I happened, in the airport lounge, see their CEO who’s yelling into a phone, and I know that they’re using the same password for work that they are for their personal stuff, and it’s ‘kitty.’ And I know this because the Post-It note on their laptop says it. And it’s, it feels like, yeah, you’re selling and solving the problems at, like, level ten for complexity, but you need to effectively handle the problems at level two first and do the easy stuff and the basics before you start getting into the world of imagining that the Mossad is coming for you.
Jess: And it’s not even level two. I’d say we’re almost—we’re at level zero for a lot of this stuff. And you’re a hundred percent right. A lot of security people and security organizations will be talking up all of these heavy-duty, “This is what we can do to protect you,” without even recognizing the elephant in the room of you’re not even using MFA. You’re not even looking at securing your administrative accounts. You’re not using separate administrative accounts and user accounts, so they are the same person and login to everything and can access the internet while still performing full administrative work using that account, which is just terrifying.
Corey: I still get constant notifications of security alerts from various vendors when there’s a new TLS policy that should be applied to the load balancer that I’m using with them. And it’s, you really think that people decrypting TLS-encrypted traffic somewhere between a user and my web server for a website that has the actual term ‘shitposting’ in its domain is my number one priority on these things? It just feels like it’s such a—it’s this incredible cacophony of noise. And then you completely miss the next email that comes in, “Oh, by the way, you put your credentials up on GitHub. Maybe do something about that.” It just becomes this entire walking modern version of a Nessus Report with the 7000 things you need to fix, and number 3768 is, “Oh, by the way, the kitchen’s on fire.” It hides this stuff. And it’s awful.
Jess: The prioritization of what can be done to fix some of the security holes that we see. It’s out of whack, I think is the polite term to use. It’s absolutely awful because we are prioritizing things that, yes, they are considered high severity in terms of what could happen, but in probability and likelihood, really, really low. Whereas things like having your password stored on a Post-It note that you have taped to your laptop that you then carry around and everyone can see it and they know who you are, maybe that’s a little bit more concerning. So, it’s trying to, from a security basics perspective, which I’ve been, I want to say ‘talking about’ but it’s not; it’s ranting about and screaming about since about 2018.
It’s about making sure that we get those basics right because without the basics, all you’re doing is putting Band-Aids over giant holes and giant leaks that anyone can slip through. And it’s a waste of time. So, making sure that we are doing those really small basic things that lets be—we’ve been talking about them for years. I think I’ve been screaming about MFA for at least ten years now. We’ve had this available to us as an option. Why are we not doing it? One of the statistics that I saw recently, inside our cloud provider, only 33% of accounts are protected with MFA. 33%? What’s happening with the other 66 because that’s just, that’s a staggering number of accounts that I would consider unsecure.
Corey: On some level, you almost have to put this back on the companies themselves. Our timing is apt on this. Earlier today, GIF-hub—which is, yes, how I pronounce it—announced that starting in a couple of weeks, they’re going to require MFA for every contributor on GIF-hub—or GitHub if you want to use the old-school pronunciation—and I think that’s great. Everyone is used to MFA, on some level. Mandating it for accounts for which the blast radius is significant goes a long way.
And yes, down the road longer-term, passkeys might make this a lot easier or not as necessary and/or better methods of authenticating against an MFA because typing in six-digit codes is annoying and goes out-of-bounds on these things. But I don’t think it’s necessarily the end of the world to make the sensitive accounts just a little bit harder to access.
Jess: I understand that for some folks, MFA, it’s that next step, it’s that next barrier, it’s another thing that they then have to do. It sounds terrible finding what I call silver linings in some of the events that have been happening recently, but if anything, particularly here where I am in my very small, tiny neck of the woods in Australia, some of the recent breaches that we have seen over here have made not just operations and technical people, but end-users, consumers, more aware of the fact that it only takes one slip-up, it only takes one small thing for something bad to happen. And anything that they can do to protect themselves is a good thing. So, we’re starting to see a bit of a shift towards an acceptance for some of those trickier things like MFA.
It’s a pain in the bum. It’s not something that is what I would consider user-experience-friendly. We’re getting better at it, but it’s still an issue when it comes to signing in and having to find your phone, which most of the time you have no idea where it is; you’ve got to use whatever automation, you have to call your phone to find it to be able to find your code so you can actually log in. It is a pain. But anything that we can do to be able to protect even our personal accounts is something that I consider to be really important.
And it’s not just for organizations. I’ve got to admit, I’m still working on my parents. They are of that generation where maybe it’s still a little bit of a step too far. But just trying to get people that I know who aren’t technical to start looking at things like this because as we move forward, it’s going to become the norm and I’d much rather people become comfortable with it now than later when it is forced upon them.
Corey: One of the problems that I’ve got is, we just went through our annual security awareness training that we are contractually obligated to provide to everyone and all the vendors that do this, I have problems with it, in different ways, in different degrees. It’s all terrible, on some level. And it’s not that these vendors themselves are bad, but it’s the state of security for most folks. One thing that gets hammered home in all of these trainings is, “Don’t click on the wrong link because if you do that it might destroy the company.” And I can’t help but think if someone in the accounting department clicks the wrong link and it destroys the company, I can’t really get myself to a point where I blame the accountant for that. I don’t feel that’s the accountant’s job.
Jess: That is definitely not the accountants job. And there is no way that a single user or a single device should be able to take out an organization. If that is the case, something has gone [laugh] very badly wrong. And I would say the blame is definitely not on the person who clicked the link. There would be a quite a range of people who would probably be hauled over the coals in regards to that one.
Educating users on what to be aware of and what to look out for, and what to not click on versus click on, how to spot scams, all of that is helpful and beneficial, not just for an organization, but I also see it as very useful even in their own personal lives. Because we are seeing ransomware scams targeting individuals, we’re seeing some of those awful scams coming from tax departments or coming from anything that’s talking about, “You’ve got to fine. You need to pay this. Please go off and buy Target gift cards.” Those are the kinds of things that we want people to be aware of even in their personal lives.
But if an organization can be taken down by one of those emails, then there has been something that hasn’t been put in place. I would say ‘something.’ A lot of things that haven’t been put in place. A lot of the work that we do from a security perspective is to limit what we call a blast radius, to try and reduce the impact an incident will have on an organization. And clicking on an email like that should produce an alert. It should say, “Hey, you probably shouldn’t be accessing this website.”
Even if they put their username and password in, it’s the credentials of that account that have been compromised. That can be reset. If necessary, the account can be rebuilt, but it certainly shouldn’t be something that brings down an entire organization. And I think our messaging around that puts the burden on users when it should be on those of us who are technical to have the… not necessarily the accountability, but the responsibility for looking after that and ensuring that that is not the case. And again, that comes back to that, number one basics and making sure we’ve got the basics right. Because if you’re clicking on something like that and it’s going to be installing malware, you shouldn’t have admin rights on your machine. That’s just j—no. Bad. But number two, it means that [crosstalk 00:19:41]—
Corey: And as part of that, the software we need to do our jobs should not require admin rights to run.
Jess: Oh—
Corey: There are certain—a couple of vendors that are hearing their ears burn because I’m thinking about them very hard right now.
Jess: Oh, if I have one more piece of software, “I need domain admin rights to be able to run.” “I need global admin rights.” No you don’t [laugh]. You really, really don’t. You are being lazy, you’re taking the piss, you’re making it easier for yourself, and causing a massive security hole for your customer. Stop it.
Back to my point [laugh]. If our operations folks and our security folks can work well together, we can get around some of that burden that we put on our users to be able to work out what it is from a usability perspective they need while still giving them the security they require. And I think when we’re looking at putting security in place, it’s that putting security first. It’s not an afterthought. It’s built into whatever we are doing, whatever processes we have, whatever systems we’re bringing in, whatever solutions we’re looking at using.
It’s thought of at the beginning, rather than, “Ah, maybe we should talk to security about that.” That involves a little bit of a culture shift, I realize that’s going to take a bit of time [laugh]. I can do technology, people are someone else’s problem; that is definitely not me to try and fix, and each organization will have their own battle when it comes to that [laugh].
Corey: There’s also this idea that—you know, companies figure this out after the first time they get it hilariously wrong—that not every person should have access to everything. I appreciate the idea of transparent culture, don’t get me wrong, but if I’m not working with a particular customer, why should I have access to that customer’s data, just lying around? It should be something that takes work, that you have to affirmatively say, “Yes, this is what I should be looking at right now.” One of the early things I learned back when I was going through one of the innumerable compliance regimens that I have throughout the course of my career is, it’s a heck of a lot easier if you can constrain the regulated or sensitive information to as smallest surface area as possible.
Here’s the PCI environment where all that stuff lives, and yeah, you turn on all the obnoxious, difficult security things involving that environment, but what that means then is you don’t have customer data in staging and development environments to worry about and you can relax a lot of the other controls, just because you don’t need to have that high-friction process for people to do things that are completely unrelated to the sensitivity of that data. And that still seems like it’s a revelation for some folks.
Jess: For a lot of folks. So, the idea of role-based access control and giving people the rights they need to be able to do their job, no more and no less. And again, it’s a balancing act of trying to work out what that is. And when it comes to the people who are doing that job, I often find when looking at putting role-based access control in place, it’s never end-users that are the problem. Ever. It’s always technical people because they need all of the access all of the time. You don’t.
So, it’s working out what they actually need versus what they want [laugh]. And that’s an even harder discussion to have. So, I find it’s not what access do you need, it’s what are you trying to do, and let’s work backwards from that. And that’s something that I still think a lot of organizations haven’t managed to get right. And also from an access perspective, that governance—again, we’re getting into process and policy and people—ahh—but it’s that access governance lifecycle as well.
Just because someone had access doesn’t necessarily they need that access going forward. And making sure we’re reviewing those levels of access going forward and removing people who don’t need that anymore. Normally, when I’m talking about that, people are thinking like IT folks who have lots of privilege or finance people who have lots of privilege. I’ll be honest. The worst folks in any organization for access privileges are executive assistants.
Those people collect access rights like it’s going out of fashion because they never get revoked. And they will move from person to person, from organization to internal organization collecting all of those rights, and they’ll maintain them for the entire length of their stay with that company. It’s an extraordinary amount of rights.
Corey: They’re like the human equivalent of the CI/CD server because it has access into every environment, it’s generally configured by hand and evolves naturally as bespoke. “Infrastructure as Code for everything except the Jenkins box. We just take a disk snapshot of that and kind of hope for the best if we ever have to rebuild it somehow.” Yeah.
Jess: [laugh]. They are. And they want—like, don’t get me wrong, they are wonderful folks. If something happens to them, usually the organization is in a lot of trouble. But when they leave, looking at the sheer amount of access they have, the mail permissions they have to be able to send on behalf of so many folks in the organization.
And you’re like, this is a lot of trust and a lot of responsibility to give to one person. So, making sure that the rights that the folks have in your organization are what they need, they… are checked regularly. And I think that’s the part that gets skipped a lot. It’s easy to give rights; it’s harder to remove them. So, it’s making sure that those access reviews are being done to say, “Hey, you’re no longer the EA for that person. Maybe you don’t need sender’s permissions on their mailbox anymore. Maybe you don’t need access into their personal files and their calendar to be able to see exactly what’s going on and being able to look at that personal folder that contains all of their photos and files of what’s going on.” So, making sure that [laugh] you’re reducing the risk, not just on the organization, but on those people as well. Because that’s a lot of responsibility to have should something go wrong.
Corey: That’s part of the problem, too, I think, is that the surface areas gotten complex and the sheer number of services that we all use to go about our jobs, regardless of what those jobs might happen to be, is so overwhelmingly massive. I remember going through an acquisition at one point, and we had something like 40 people at the company, and we were well over 800 distinct SaaS products that we were using throughout the entire company for different things. And how many of these are important? How many things are going in other directions? And if we look across the entire surface area of these companies and the things that we’re using, no one knows what’s going on in their environment and where.
Okay, you get access to this? I don’t know, there’s this ridiculous little thing I used to caption funny pictures. Okay, it’s technically a SaaS product, but it’s probably not critical path. Oh, here’s the system we use to do payroll. Yeah, you probably want to double-check that one. And it all ones have lumped together in the big bucket of, “We have no idea what’s in this thing.”
Jess: Oh, my God. I need people to understand that documentation is important [laugh] and this is exactly why. Unless it is written down, unless someone has been able to take out of their head a decision that was made about this as a product we are using, this as what it does, this is how critical it is to our organization, unless that is somewhere, you end up in exactly the situation you’re talking about. You’ve got all of these products and you don’t know which ones are business-critical. You don’t know which ones are considered your primary goals, particularly in the case of a BCP, so a Business Continuity Plan or Disaster Recovery perspective.
They’re the ones that you want to protect and that should be somewhere. But a lot of people seem to think that documentation is boring and documentation is it’s not necessary. It’s my head. Why would I need to write any of that down? I know it all.
Please make sure you’re getting it out of your head, I don’t want to have to pilfer through your head to find the stuff. And I know that there’s a couple of folks, personally that I know who will feel very attacked by this. Just because it’s in your head, just because you know, doesn’t mean everybody else knows. And I’m very much of the opinion that if someone asks you a question twice, that is a piece of information that needs to be on a piece of paper somewhere that other people can see.
And we don’t do that enough. We don’t pass on information enough and share information enough. There is very much an information-hoarding mentality for a lot of folk of, “I need to protect this information to be able to keep my job.” And that is just not the case. Because in the case of a disaster, in the case of a merger, where you’re trying to work out what is actually needed versus what is not, that information is crucial and it can cost time and a lot of money if you don’t know that information, if you don’t have someone that has put that down and you can reference it in an easy and quick way.
Corey: I really want to thank you for taking the time to go through how you think about these things. If people want to learn more, where’s the best place for them to find you? And please don’t say Australia.
Jess: Well, yes, Australia, but no one wants to come down here because again, we have things that will kill you. I would say Twitter or Mastodon. You can find me as at @girlgerms, if you haven’t found me already. I am on InfoSec.exchange as @girlgerms.
I will also be speaking at a number of conferences coming up here in Australia. Again, if you want to come to Australia, please do, but I know that recordings of these will be available at some point, so I will be speaking. Luckily, I’ve been invited to go and speak in New Zealand, which is very exciting. So, I’m going to be speaking at DevNxt in Christchurch on the 21st of April.
And I will also be at [unintelligible 00:29:33] between the 9th and 12th of May this year. Again, that’s in the future, literally in the future. So, I will be talking about all sorts of things related to operations and security operations, my talk at [unintelligible 00:29:48] actually going to be really interesting. It’s talking about using your operations folks and how to get the best out of them from a security perspective, based on my history being an operations person and how I’ve moved into security and how that can be a real benefit and an asset to an organization.
Corey: And we will, of course, put links to that in the [show notes 00:30:08]. Thank you so much for being so generous with your time. I appreciate it.
Jess: [No dramas 00:30:12]. Thank you very much for having me.
Corey: Jess Dodson, Senior Cloud Solution Architect at Microsoft, I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that I’ll log in as you and delete later because you use the same password for everything you log into. It’s ‘poisonouskitty.’
Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.