Subscribe
Share
Share
Embed
Cybersecurity is complex. Its user experience doesn’t have to be. Heidi Trost interviews information security experts about how we can make it easier for people—and their organizations—to stay secure.
Well, welcome everyone. Welcome to Human-Centered Security. I'm your host, Heidi Trost, and I am joined by Ariana Mirian. Is that how you say your last name?
Yeah, that was great. Thank you.
Okay, perfect. So, it is a Friday afternoon, and Ariana and I are just, you know, just gonna wing it. We're just gonna wing it. We found our cool backgrounds, which are, are complementary. So if you're not, if, if you're not seeing us, if you're just listening, so I have a beautiful pink background, and Ariana has a beautiful purple background, and we felt like they were totally Friday vibes.
So, Ariana, thank you so much for joining me. Super excited to hear about some research that you did and unpack, you know, all things phishing simulations and cybersecurity awareness training, all that good stuff.
Yeah, Heidi, I'm super happy to be here and thanks for having me. The, the phrase this week when people have been asking like, oh, how are you doing? More often than not, I have responded with, I'm vibing and thriving, um, which is just how I've gotten through this week. So I am happy to be vibing and thriving here on Human-Centered Security with you. Thank you so much for having me.
Oh, awesome. I like it. Yes, yes. So, okay, so give me, okay. What do you do? Like, what, what, yeah, what does Ariana do?
Yeah, absolutely. So, a little bit of a background. I was introduced to security, oh boy, 12 or 13 years ago. Very serendipitously I kind of stumbled across it when I was doing some of my undergraduate studies. And I was really fascinated by security research, and specifically with this intersection of, of measurement. You know, not only, how can we better do security research, but how can we better quantify it specifically when it comes to the human and the person behind devices, screens, whatever. And, and this, this interest actually led me to pursue a PhD at UC San Diego.
And a lot of my thesis was spent on, you know, trying to figure out how can we use large scale data analysis to better drive security decisions, specifically with the person in mind. Because at the end of the day, I inherently believe—and I know you do, because we are here on this podcast together—that security is a person driven field. You know, it's so easy to get caught up in like, what are the hardware components? What are the software components? But I'm like, what are the person, what are they doing? I wanna know that. And so, you know, this initial fascination led me down this entire research path.
And now I currently work as a senior security researcher over at Censys, which is like the internet map of all devices and things on the world. And I still very much believe and advocate for the person behind it all.
So it's kind of a high level overview of how we ended up here and how we're chatting today.
Yeah, we were kind of serendipitously connected. And Ariana is in San Diego, as well. So we actually got to meet in person, which is just so delightful.
But as we were talking and we were, you know, having coffee, we were talking about her research, and she told me what the name of the research paper was, and I was like, wait, are you famous? Because I've read this, I seen this go around LinkedIn and folks talking about this. So that was, I was like, you have to be on, you have to be on the podcast.
We totally, you know, vibed over human-centered security.
So. All right. So let's dive into this research. By the way, it's called “Understanding the Efficacy of Phishing Training in Practice.”
Yes. Yeah. Okay.
Yes, correct. And this was the last project of my PhD. It was one of the projects I am most proud of and I think very much exemplifies the need and the benefit of combining this data driven security analysis with people.
It's like, how efficacious is security training? Let's find out, shall we?
Whoa, you get major points for saying that word out loud. I would never, I'm not even gonna try to say it now. That's amazing.
You'll notice it's, we, we use that phrase a lot in the entirety of the project. So don't worry, I might stumble on it in the next 45 minutes, however long we have. Don't worry.
Okay. So tell me, tell me about this research. Why, why did you conduct it? What, what drew you to it? You know, what, what were the research questions?
Yeah, so, um, just a little bit background. You know, when I was working at UCSD during my PhD, we were very fortunate and the folks who are still at UCSD continue this relationship to have a very close-knit relationship with like both the, the UCSD IT folks, and also the folks over at UCSD Health and in their security team.
And so this was kind of born out of a series of conversations where it was like, Hey, we have the, you know, security research and measurement expertise and desire to run some of these studies. And there's also a desire at the organizational level to understand, you know, how effective is security training, this thing that lots of people are spending lots of money on. And so it was a very, you know, as there are many things in life, a serendipitous moment where the right people at the right time came together to tackle this question.
And specifically, you know, I think there were a lot of folks at UC San Diego Health who wanted to understand, you know, how effective is the security training. We have these phishing simulations where, you know, we send fake fish to actual employees who will interact with it on their actual work laptops during their actual workday. And then when they fail, we give them a training. But like, is that effective? Is that doing anything? If not, what, what else should we be doing?
And so this was a, a beautiful collaboration. A huge shout out to all of my collaborators who are also on this paper, to which this could not have been done without, you know, everyone involved.
But yeah, that, that's kind of the backstory.
And so the premise, like I've said or alluded to, is how effective is the security training? You know, we know it, we love it. A lot of folks in various enterprises.
We love it. I dunno, who's the “we”? Yeah, this was, you were being facetious. I get it. I'm a little slow on Friday. Okay. I get, I get it. We know it, we love it. We being, uh, not me. That's for sure.
Well, I just, I wanna pause for a second. I feel like, like if we're we're being real, the folks, you know, who were kind of starting, you know, to think about this research, you said like a bunch of collaborators, you know, came together and like, you know, this beautiful research happened, but it sounded like kinda the undertones of, we're spending a lot of money on this s**t. Like, is it really working? And oh, we have a lot of smart people who know how to measure stuff. So like, can we, can we see if this is actually working? Right?
Yeah. Yeah. A hundred percent. A hundred percent. And you know, I will say, a little bit of a, I won't, I won't dive too much into a spoiler, but going into the paper, we were very optimistic. Like, our goal was to find the most effective modality for security training. And then I remember we were throwing a lot of ideas around, you know, we talked about in the paper, we examined like static training.
So you're just given a webpage. It's like, Hey, here's what phishing is and here's why it's bad. And we also, um, investigated this more like interactive training where you get a single page. It's like, Hey, let's talk about headers. And then it, like, it's pointing to what headers are. And, and you click next, and then it's like, okay, now let's talk about how often these phishing emails have urgency in the request and why that's a big red flag. And then you read, then you click next.
So we looked at static and interactive, but we were even throwing around these ideas of, you know, what if we tried other modalities? Like, what if we tried a just in time training where if you fail a phish, you get dropped into a Zoom where someone's like, hello, welcome, you failed. Now let's talk about why that happened and why that's bad.
Obviously we did not test that. There was a lot of, uh, logistical difficulties with that idea, but, you know, uh, the initial idea was
It's like the worst, the worst one-on-one meeting you've ever had in your life.
Can you imagine? You're like, you're working from home because, uh, I, I think at the time, a lot of UCSD health employees were still like able to work remote, or, or still do, I'm actually not sure what the policy is nowadays. But just imagine you're like, at home, you're not looking the best, you're like answering some emails.
You click on something and then all of a sudden someone's on a Zoom and they're like, so that was a phish. And it's just like, what is happening?
Oh my gosh. That's crazy.
Oh, so backing up, uh, wanna ask you some like, researcher nerdy questions?
Yeah.
What, what were your research questions?
Yeah, so the research questions, there were a couple, I'll actually break this down by data set because that might be a little bit easier.
So we looked at two different data sets. One was this annual security training.
So everyone who's an employee needs to do this annual security training. Where, you know, you sit through a module, I think it's like 35 minutes or something, and it walks you through very…
It's the best day of the year… You go through security training, you're like, Yes, It's on my calendar! I can't wait to spend 35 minutes learning about best practices for security.
Um, it is, yeah. And you know, I say this, I think a lot of folks on this podcast are probably familiar with this annual security training, no matter what form it takes, right? This is something a lot of enterprises are, are deploying. And this one was, um, so the, the dataset was, you know,
for folks who had completed this annual security training, was there any temporal relation with when and if they failed a phishing simulation, right? Um, so that was—
You're using too many big words on a Friday. What, what does “temporal” mean?
Yeah. So the idea, you know, it's like, okay, so how do we measure at the crux of this? And, and, you know, Heidi, this is one of my, this is maybe a complete additional hour aside, but, you know, metrics really matter. This is one of the things that I am consistently saying as a measurement scientist at my core, is that the metrics that you're using using matters.
And so when we say like, oh, how efficacious is this, these trainings, I know the word I had to drop it in. When we're, when we're saying how efficacious is it, there's an underlying metric that lets us answer that question.
And so when we were looking at annual security training, we were looking at like, okay, how do we understand the outcomes here? Well, one idea was if you have recently been trained, right? Like, if I had done my training yesterday, you, Heidi had done it two months ago, hypothetically, I should be less likely to click on a phishing email than you, because it's fresher in my mind. I mean, two months, that's a basically a year, right? It's, it's so long ago at this point.
And so we looked at, you know, that the time relation between when people did their training and when, or if they clicked on a phishing email. So that was, that was one data set.
And actually, should I, should I just dive into the outcome there before I dive onto, into the second data set? Or would you prefer an overview—
Dive into, dive into the second data set and then talk to us about like what, like…What did you overall find out?
Yeah, yeah, yeah. So then, then the second data set…so first one was annual training, right? One done every three under 65 days. The second data set was this series of simulated phishing emails that were sent to employees, which were already getting sent as part of their security regimen anyway.
So the, it was monthly phishing emails that was sent for eight months in a row. And the idea is that if someone clicked on this phishing simulation, then they effectively failed. And then they were dropped into a training. And we had come up with different modalities of training, which I had kind of alluded to before, right?
So we, we had like a static generic, which was just a one pager, like, here's why phishing is bad.
We had a static contextual, which is like, here is why that phishing email that you just clicked on was bad. And then the same two types for interactive training. So again, the more like, let's have moving arrows that show you where the header is and why this urgency is bad, and you click next.
So static interactive, contextual versus generic. And so we took this pool of employees, which was roughly about 20,000 employees, and broke them up into different groups, and then also had a control.
So we had a control group that was like, if you click on a phishing, a phishing simulation from us, you just get a 404.
And it's like, oh, the link didn't work, whatever. So we can compare, you know, how folks in different groups are performing to the control to each other with the end goal of saying, which of these modalities is most effective in preventing someone from clicking on phishing again?
And this is where the repetition really came in handy, is that we had eight months of data. And so we could see, okay, so you failed last month. Did you fail again? Or if you failed, or if you didn't fail, did you fail in the future?
And so there was just like a lot of
Who was writing, who was writing, was it based on the, the, the vendor? Like these were pre these, okay. These weren't things that you were like, okay, drafting our own phish email. Like, you didn't have any say in what they were.
So, uh, sorry, I jumped ahead and I assumed what your question was. So the, the phishing emails, and this is another interesting outcome of the study. We also sent different types of phishing lures. We sent anything from like, Hey, your password…please click on this to reset your password. All the way to, we crafted one that was like, your vacation policy is changing, please review the change in vacation policy. Click on this link.
And so the lures, we were looking at the vendor's library, and we ended up crafting a few of our own, while also trying to remain ethical, because, you know, this was back in 2023. And COVID, I mean, COVID is still around, but it felt more present at the time. And so we were like, okay, we're not gonna craft a lure about like COVID, because that, for us, seems a little too far over the line for a phishing email for a research study. Attackers may do that, but we, we were trying to toe the line between what do, what, what is a spread of different lures.
So we could also compare how the lures did—which spoiler they perform very differently.
Yeah, I thought that was really, really interesting. Yeah. Okay. Go ahead. I won't interrupt you.
And then, yeah, so and thanks to everyone for bearing with me, there's, like I said, a very rich data set here. The trainings were basically out of the box from the vendor. So the lures we like, took inspiration. We maybe tweaked them a little bit to be like, how would we craft this to be more realistic? But the trainings as much as we could, were, we're gonna take this out of the box because that's what was being deployed.
And that's ultimately what you were analyzing, like, should we be paying for this, this particular solution? Or are there other things that would be more, I'm not gonna say the word because I can't say it, but are gonna be more effective, right?
Yeah, exactly. Exactly.
Makes sense.
So I really wanna be clear, the lures had some crafting to it. I really do feel like I could have an alternate career on the other side of security…
Because I'm like, oh, wow. The way that you were smiling made me say, I was like, she had a lot of fun crafting these emails.
Honestly, we did, we were like, is, I think there was one, it was like a traffic violation, you know, like, you have a ticket, please click on this. And I was like, Ooh, that's mean, let's send it. Which
is like a real thing, like on a university campus.
I remember getting those and being like, oh, they got me again. I was only there for 10 minutes.
I know. They're so, they're actually so fast. Especially UCSD parking. Good on them for being so fast.
But yeah. But the training was completely out of the box. I wanna be clear for that. Because that's what we were trying to measure is like this industry standard vendor. How does that perform?
Yeah. Awesome. Oh gosh. Like, yeah. I mean, you read, so folks should definitely read the paper because there's, there's just a lot to unpack and it's really, really, really interesting research.
Okay. So we, we kind of set the stage on like the research questions, the dataset. The research participants were part of, of the university.
It was, uh, sorry, my mom, mom's calling me. Lemme just decline that for a second.
Does she have something to say? Does she wanna learn about…?
She probably actually, she, she would have would have a lot to say about this actually.
Oh my God. I know. Every time I tell people that this is a study that we perform, they're like, I have thoughts about training. And I'm like, I know you do. Everyone does.
Everyone, I feel it. Everyone I feel it. Yeah. Uh, yeah, I feel it. And I, and I'm sorry. I'm sorry for what, what you have gone through, right? Like, I feel like I need to apologize on behalf of the cybersecurity industry. Like I, I'm sorry. I'm really sorry.
Listen, we learned a lot. Yeah. So, so that's an overview of the data sets and the research questions. Let me go back to the first dataset, because that was simpler, more straightforward, right? It was, uh, we ran GLME which is a statistical model, um, that basically looked to see if you had been more recently trained where you less likely to click on a phishing email.
And the short answer is no. We actually found very little to no benefit from this annual security training, because it turns out your likelihood to click on phishing emails, did not vary based on the time relationship to when you did the training. And so—
I feel like you should revise the title to be like, short answer, like TLDR. Like no, it does not. No. Like what a snappy title.
Yeah. Well, and so, and so this is, you know, this is the difference between, because the paper was submitted to an academic conference, IEEE …And so there, you know, there's some formality that goes into that. I mean, every conference has their own structure, their own thing. But yeah, then when I'm talking about it in, in conversations, I be like, actually, no. Uh, long story, no. So, okay, so this—
Yeah, yeah, go ahead. Sorry.
No, no. So I was gonna say, so that was, that was the first outcome. And, and that was pretty straightforward, right? Because we didn't, we didn't have a lot of metrics. We didn't have the same riches of data. It was like, of the people who completed, when did they complete it? When did they click on a fish? And this is where like the measurement science comes in. So it's like, that was really the extent of what we could push on that data set.
Second data set, Heidi. Oh boy. Oh boy. Where do I even—
Get your popcorn folks.
I know. Where do I even begin? So again, uh, so this was, this was the embedded phishing training. This was, we sent you a simulated phish for eight months. If you clicked on it, we put you in one of the different trainings. And, uh, the short answer there is also not as useful—very limited benefit statistically.
And we suspect in part because folks are not actually engaging with the training. And so, let me just start from the top down. We ran, and I should say a couple of the co-authors on this paper are from the, and I wanna get this right, Biostatistics, Epidemiology, and Research Design Center, um, at UCSD. So we hired folks with backgrounds in statistics to help us with this, because we were like, we wanna get this right. Like, we wanna do this paper, we wanna do this.
Yeah. This was the parts of the paper that I was like, wow. Like, these people know what they're doing, and I don't understand half of this stuff, but this, this is very interesting.
Yeah. And this was the part where we were like, we have the capacity to do this, but if we include folks from BERD, this will be done, you know, just at a higher quality. And faster. And it allowed us to work on various other things as well. So I, I really do wanna cut out, shout out that the folks at BERD are, are really fantastic.
So, okay. Uh, I'm trying to figure out do. So what, so there were a lot of different statistical comparisons that were done. The first was just like, is there a reduction in the average failure rate from people who failed and then got a training versus people who like never failed?
And the answer is not really, I think it was very modest. So the folks who ended up getting a training, there was about a 2% difference in average failure rate than folks who didn't get training. And so, yes, there is a difference, but it is so small, it is so tiny that it really being brings into this question of, is all of this effort, all of this energy, all of this money worth a 2% reduction in average failure rate?
So that was the first big thing. The, the second big thing, and this is where, you know, the depth of the data really enabled, this research question is like, why is this phishing simulation, why is there only a 2% difference in average failure rate?
And so we were able to embed some metrics into the training that allowed us to see how long people were spending on the training.
This is not gonna be surprising. The answer is not much. It is not much. Most people, when they see the training spent, and I have to get the exact number, but, I think the majority spent less than like 10 seconds or 30 seconds on it. I mean, yeah. A lot of folks—
Yeah, I feel like you, the, it was like they just, the majority of folks just were like, boop, like goodbye, and just clicked out of it immediately.
Yep. You see, you see a phishing, you click on an email, you see a thing that's like, Hey, you were phish, let's train you. And you close out of it because of course you are, you are a person trying to get through their daily routine. You're trying to get through work, get to your family, get to your friends, whatever's happening in the world. I mean, I can't even blame people for doing that because of course, you're just existing in this world and you have this blocker to your day, so you're like, I don't need this.
So we found that, you know, the majority of people spent little to no time on this training, which was actually, um, not necessarily surprising, but was a very useful finding for us.
Because then the, you know, one of the questions is, okay, well, is it worth trying to get people to engage more with the training? How can we do that more effectively if the answer is that they're not even spending time?
So of course there's no difference because they're effectively not getting trained. Right?
And then, you know, the, the third big outcome is that for the people who spent time on the training, we looked at the value of the training type. So remember we had like this static versus interactive. And when we look at, you know, the performance of these different groups, we actually found that interactive training, right, can reduce the likelihood of clicking on subsequent phishing emails by something like 20%.
I think the actual number is 19, but we'll say 20 to, to, to give it a little boost.
The issue with that is that the absolute numbers of people who spent time on training is so small that it is hard to say whether that is because of a selection bias or if this is a generalizable outcome. You know, all the other models, they had enough people in the various groups for us to say, we think that this is…we are confident in this outcome.
And, you know, with this, this 20% increase, we're like, well, we don't wanna sweep it under the rug, but also we're actually not sure if it generalizes because so few people are interacting with the training for more than a couple seconds. Um, yeah. I'll stop there. I’m just on a roll.
Yeah, that's so interesting. One thing I wanted to ask you about, and I didn't see it until like, the very, very end of the research paper. But you, one of the things that you did not track was how many folks, reported the phishing email, right?
So they, they received it, they saw it, and then they said, I know this is phishing, and now I'm gonna report it to my university. Can you just talk a little bit about why you decided not to capture those?
Yeah, yeah, yeah. Um, we were already tracking so much that it, it was an additional, it was an add-on. That would've complicated things a lot for us, just because of the way that the setup was the way that we were collecting data.
And so we decided for the sake of being able to get the experiment moving, we would focus on getting those phishing metrics, getting the training, et cetera. I will say, you know, and this is where I think there's a lot of beauty in academic research. There are other research studies that have also been able to deploy real world phishing simulation. How efficacious is training. Like Lane et. al., was one of the ones that came before our paper. And they found, you know, very similar results, which is that phishing training has little to limited effect, but if I remember correctly, their paper also found that the act that people were more, were better at reporting the phish.
And so there are potential side benefits, but if you look at just is the training, uh, yeah. Making it less likely that you'll click in the future, the answer is not really. And so I, and you know, I really don't wanna discount that there can be some additional side effects that others have studied that I think would be great to examine in different settings. Um, yeah, because that's the beauty of people, right? It's like maybe they don't grok that, what a phishing, you know, like they shouldn't click on it, but they can report it.
Or there's a larger people, a percentage of people who are reporting it.
And, and, and this is like the beauty and the complexity of people.
Yes. And I, I totally understand. Like there's, you know, you're, you're, you have a set of research questions, you have a, you know, a data set. It sounds like in the research paper you talked about how like, so like, it was, it was actually gonna be complicated to track. There are multiple ways to, you know, report phishing and that sort of thing. Okay. Just wanna get that outta the way.
Now I wanna talk about—
Like, yeah, great question.
What, knowing what you know now doing the research hat you did, like, what sort of recommendations do you have for folks? Or, um, you know, even if, if it's just like, you know, there, there are opportunities for future research in, in these areas.
Yeah, completely. You know, and, and Heidi, this is where I would love to hear your opinions as well. I kind of—
No one wants to hear my opinions.No one pays—
Oh my God.
No one pays to hear. Just kidding.
Please. I do. I would love one, me would love to hear your opinions. Oh, man, I gotta say this is, this is, this conversation is giving me life on this Friday afternoon.
Um, so, you know, there, there's, there's two different approaches, right?
So like I said, we, and this is the scientists and me coming out. We measured, uh, two, uh, a couple different modalities, static, interactive with out of the box training. It is very possible that there are other modalities, that there are other ways to deliver the training that are actually more effective and specifically are more effective at engaging someone that is incredibly possible.
The one on one Zoom call, that's what Ariana's like.
I, I'm on, I am gonna be on the Zoom teaching everyone how to, honestly, that's what I feel sometimes with my mom. She'll call me and I'm like, mom, are you okay? And she's like, I got this email. And I'm like, oh my God. Which to her credit, her expertise is not security. So I can't even be mad because her expertise is something completely different.
She did, she's totally the right, she
I know. Yeah. She's asking me—
She's phoning, she's phoning the smartest person she knows, so she can phone a friend. She's phoning a friend, right? It's like the equivalent of phoning a friend. So she's doing what she's supposed to be doing. You, you must have taught her well, I'm sure she's so proud.
She's getting just-in-time trained that is right by me, only me. Um, right. And so, so there is this discourse of, there there is limit. There seems to be, um, a difference with the interactive training group, even though the absolute difference is small, the absolute numbers are small.
So maybe the deeper question is, how do we get people to more meaningfully engage in the first place? And then if you can get the engagement then the next question is, how do you convey this information in a way that someone can remember and, you know—
A TikTok video. Right? That's what you recommend.
Yeah. Yeah. That's it. The training is just gonna be a TikTok link.
That's it. That's it. Yep.
Well, you say engagement, I think a TikTok video.
Yeah.
Yeah. You heard it…You heard it from the researcher yourself.
Now, now you know, I'm gonna have to come up with an anti-phishing dance, Heidi. Now I, that's what I'll be doing this weekend.
Oh my gosh. That is the best idea I've heard in a really long time. Alright, I love it.
So you and I are gonna get together.
Uh, Whoa, I didn't volunteer to do this, by the way.
Like I said, great research is not, I don’t…great research is done as a team.
So true. Yeah.
And so, so I'm, I'm of two, I'm of two minds here, right? One is that there's a lot of great psychology and cognitive science research behind learning.And it is possible that it just has not really made its way into the security realm as deeply. We're not, you know, applying those lessons as well.
The other side of me is like, people are not security experts. Like when we think about, you know, driving lessons, for example, the reason, one of the reasons many of the reasons that's effective is it's something you do every day. It's something that doesn't require specialized knowledge. And it's, and oh, I guess already says, but it's repetitive, right? Unless we're deploying training every day—which I don't think we should do—I just find it, my hypothesis is that it would be very hard, again, for someone who is not a security expert, who is trying to go about their day, who has so many other things going on in their mind, worries, stresses, love, interests, whatever it is, they have so much else going on.
How can we reasonably expect that person to also be an anti phishing expert?
So I, I'm up two minds of that. Yeah. You know, it's like, I acknowledge that there could be a different modality, there could be different ways. And also how much more effective would it actually be to go down that entire route?
So, so that, that's one area, right? The other area, which is my personal philosophy, is remove the struggle from the user and put it on the system as much as possible, right?
So, get better. There's so many startups that are trying to do anti- phishing tracking and removing it from the inbox before it even gets in front of the user. So like, how do we catch phish better? And earlier, what if we deployed things like two factor authentication, more ubiquitously—
I came up with a startup name as you were talking. I call it the Phish Hoover. There's probably some trademark issues, but I feel like, like as you were talking about, like sucking up all the phishing emails, the Phishing Hoover, you're welcome.
That is going in the TikTok.
That's my one con contribution to the podcast. Okay.Continue.
That's, that's what we're doing this weekend, Heidi. We're coming up with a dance to the phishing hoover. Oh, no, this is all I'm gonna think about all weekend.
Oh, no.
Um, yeah. So, so then the other question is, you know, the other mentality, so there's the Phishing Hoover idea in general, but then there's, or sorry, there's the user has to figure it out, or there's like, how do we “phish hoove” for the user, right? And so, two-factor authentication is still phishable, but it is still an additional layer of protection than just a password, right?
Just, just a link would… it is an additional layer of protection. This is my Friday afternoon brain failing me a little bit. Um, I was like, I have nothing else to say. That's the end of the sentence.
And, you know, if we even wanted to take it a step further, we could even say, you know, everyone has to use a YubiKey key, right? And so, like those hardware tokens that you put in your computer. Those are actually not phishable because they're doing a cryptographic protocol with the computer that it's plugged into, at least not fishable to our current understanding.
Maybe in three years there's gonna be a BlackHat talk that's like jk, it's all broken, at which case I don't even know what to do.
I feel like that is all cybersecurity is. It's like, okay, here's what you gotta do, right? And like six months later, it's like, throw everything out the window. That's not effective anymore.
But those, that's great advice.
I'm, okay, so we talked about, I'm, loving the let's put the burden on the system, right? Or even even putting more of a burden on, you know, the, the system designers, right?
Yeah. And being like, can you come up with more, better creative ways to not put the burden on end users? But yes, like the system design is, is more robust. So, you know, people don't have to worry about phishing, right? Like, that is the ideal.
Right. Right. Right. Exactly. And, you know, um, I think there's a lot of instances where this has played out well in security. Like, for example, 10 years ago, uh, HTTPS was not as ubiquitous on the internet. And it wasn't that people, browsers, service providers were going to every website owner and saying, Hey, you gotta move to HTTPS. I mean, there was some of that, but I think what actually moved the needle is that organizations took this on themselves.
Right? They were like, we're just gonna include HTTPS in your web hosting package.
Like, we're just including that.
Well, I was gonna say, I think yes, and at some point, I don't know when this was, Google basically said like, if it's not HTTPS like your search results are gonna—
Exactly. And that was the other example that I was gonna give. Yeah. It's the—
Sorry.
No, no, no. We we're, we're on the TikTok dance level. We're good. Yeah. Um, but no, that's exactly it, right? Yeah. And then the browser stepped in and the browser was like, for the good of the internet, we are just deciding that HTTPS needs to be ubiquitous, and here's how we're gonna do it technically.
And so that's an example in a slightly different context, you know, HTTPS is different than like phishing, where the systems, the enterprises, the organizations took on that burden. So that at the end of the day, my mom isn't calling me saying, wait, so if the lock isn't there—which I know there's not a lock anymore—but she's not like, so can you imagine…
Well, can we talk about the lock being gone now? Like, this is, this was like a, I was like, wait a second, hold the phone, someone hold my drink. There's no lock.
There is no lock.
We don't have to go down that rabbit hole.
Yeah. It was a big change. I remember seeing that too. And I was like, wow, that is how the internet evolves, is that it is now. So it is just the default. We don't even have to say it's secure. It is just the default. Yeah. Um, which is a beautiful and sad thing because… Yeah, I really love the lock. There's so many great studies behind the lock.
Yeah. There's a lot of great studies.
But yeah, so, so, you know, I think it really boils down to like, do we somehow make the training better? Because what is coming out of the box right now is not effective, or do we try and remove that struggle as much as we can from the user in these other ways? Two factor, um, you know, email monitoring for enterprises, it doesn't really solve the like, personal issue.
But I, I think it really boils down to those two, and I think they both have merits, but it's a, it's a question of which one will be most effective for the time and energy that you're gonna put into it.
And if you ask me personally, I'm like, let's go towards 2FA, let's try and see if that is a more effective route towards protecting users. And maybe there needs to be training that goes into that.
But that seems a lot more promising to me than like, okay, so what if we tweak the current training to be like a little different or have a different modality or something.
Right. Um, so yeah, that's that…
I love that.
My, my Friday afternoon rambling, so…
I love it. Well, no, it's like, it's, and I've, I've thought about this too. I'm like, if we, if we pooled all of the money that we've spent/wasted on, on these, these trainings, could we have just built a better system? Could we have just made email more secure from the ground up?
I don't know. Like, it's complicated. I get it. You know, people think it's an oversimplification, but like, think about all that money that … It's really this like equation of like the time and energy and money that we're putting into this.
What is the outcome that is most effective?
You know, Heidi, if we've still got time, there's two things I wanna call out before I forget, um, that I didn't talk about when we were—
Follow Ariana on TikTok and—
I’m @ … you can find me at Phish Hoover. Uh, I'm actually gonna gonna see if that, if that handle is available, it's probably trademarked.
You know, and, and again this is like the measurement scientists in me geeking out. The two really beautiful things about this study is that, like some other security studies and basically all of medicine, it was a randomized control trial, right?
And so there was a control group that we could compare to baseline, and that was incredibly useful because it wasn't just like, oh, yeah, well, like, they didn't click less over time. It was like, no, compared to the baseline of people who are not getting trained, what is the difference here?
And this isn't impossible in security, and we were not the first to do it, but I will say, at least in the, the studies and papers I've seen, that's rare.
And then you look at something like medicine, and that is the standard, right? You cannot do any sort of trial without a control. And so this, this is why at the beginning I was like, this paper was like, my dream is because it was person, it was people oriented. It was a real life problem.
And we were able to say, okay, to do this right, we need a control, and that's what we're gonna do.
The second thing, and I don't think you…and I, uh, we just like, there was so much else to talk about, is that your phishing lure matters. And I kind of alluded to this, the failure rates were anywhere from like 1% to 30%. The vacation policy trade change really freaked people out.
Don't mess with people's vacation, right?
I know. Same.
No, no, no.
Same. But I, I do think, you know, it calls into question, when folks are making claims, having the data to also show the behavior behind these claims, right?
Because you could imagine that somewhere in the world there's a security organization where they're told, Hey, you need to get phishing rates down to 5%. And so they just deploy a phishing lure that will behave at 5%. When really, if you send something about taxes during tax season, or you send something about vacation policy, all of a sudden there's way more people who are falling for it.
And so, so I think, you know, this, this really boils down to this, this philosophy of like, the metrics matter. What you're measuring matters. What's the data you have? How did you craft this?
Well, it seems like you're saying also like the incentives matter, right?
Yeah.
Yeah. Because who, who was it that, I mean, I'm gonna totally mess this up, but like, you can make the data say whatever you want it to say, right?
Exactly. You can a hundred percent. And again, as a measurement scientist, I'm like, I know this is possible. We have a graph that shows that is possible. And so I think it's really important, you know, that a lot, you can claim a lot of things on the internet about like efficacy and whether this works or that works, but at the end of the day, showing the data behind it, how you craft… how you collected that data, what it's saying…I think is really imperative for us to just move the security field forward.
And, and this is something that I'm fairly passionate about.
Yeah. I feel like that's a…cybersecurity marketing is just like, well, you know what I mean? Like, it's one of like their worst vices, right? Like, we'll say anything like we can come up with a number for anything, and it's gonna be a hundred percent…Like 90, 99.9%.
Yeah. And I'm like 92 of what? How did you calculate that? What was your denominator? I don't know.
What’s a denominator? No.
Yeah. Yeah. Was there a denominator? I know, I, so Heidi, sometimes I see things and I'm like, are we certain about how we calculated this metric? Uh, I have questions. I always have questions. This is the curse of working in data, is I, I see a graph and I'm like, wait, but like, did you, how did you calculate this?
And did you measure this? And did you account for…? I'm just, it's a curse. Yeah.
See, when I look at a graph, I'm just like, what font did you use?
Also important. Also important.
Why is this so hard for my old eyes to read? No, Arianna, this has been so great.
Oh, that's so real. And I'm so, I'm so, so happy that we had this conversation.
The research paper is, is very, very interesting and so, so excited that you had the opportunity to, you know, kind of blend all of your, your different interests and, and just like, come out with these amazing findings.
Yeah. Yeah. Me too. Honestly, like I said, this is, this study is like the pinnacle of really what I was working towards, towards my PhD. I was like, I know this is possible. I know we can do this, but like I said, it's about being in the right time in the right place, having the incentives, right. And this beautiful research study was born with a lot of interesting outcomes.
So I'm, I'm very…it's been interesting to see the responses. I'm very curious to see where things go from here. And it's been a delight. It's been a delight to be on this, this podcast, Heidi. Thank you so much. Yeah.
Well, okay. So, so very, very last question. What are you, like, what are you thinking about next? Like what, what would you love to do research on what's worrying you, you know, kind of what's on your mind right now?
Yeah. You know, um, uh, there's a lot of things on my mind right now. Wow. In the human centered realm. I think there is a lot of really open questions. And you and I have already chatted about this, of like, what is the most bang for our buck, you know, what is going to be the most effective of all these different avenues to, at the end of the day, protect users?
Is it like we go hire Phish Hoover? Is it that we just de say, Hey, we don't care about the usability concerns. Everyone needs a YubiKey. And then see, you know, like how productivity changes over the next couple months, which is like its own messy metric to handle. Is it that, you know, training?
Are you getting sponsored by YubiKey?
Oh my God.
By the way, please sponsor the show.
I love YubiKeys, and I, uh, I wish, no, really not. I just, I love YubiKeys
'm like, this is the—I can't get my mom to use it yet, but I'm working on it.
You know, that you've made some progress if you get your mom to use it.
Yeah, I know.
So report back. That is, that's your mission. And we'll check in in a few months and, and see how you did.
I will, I will. But yeah, those are, those are the things on my mind. You know, it's, um, at the end of the day, how do we improve the end users' life?
How do we make sure that they're not getting phished?
Because I mean, even whether it's work or personal, it has repercussions. It is incredibly stressful. There's so, so many bad things that could go wrong. And like, at the end of the day, you know, removing the science and the data and everything aside, I'm like, how do we protect people? Like what is the, the most effective way to do that? So, so that's what I think about.
Yeah, I love that. How to protect people. That's fantastic. All right, well, thank you, Ariana. This has been so much fun. Thank you so much for being on the show and for sharing your research.
Absolutely. Thanks so much, Heidi. Uh, stay tuned for our TikTok dance coming out soon.