As somebody who was a researcher, I built it for something that
would make my life better and makes me happier with the experience.
I need more feedback from active researchers and
community of what you want to see in the program.
Welcome to Screaming in the Cloud.
I'm Corey Quinn.
Here today, we're going to talk about something near and dear to my heart
that I've tried desperately to get away from and somehow can't quite seem to.
Ryan Nolette is a Senior Security Engineer at AWS Outreach.
Ryan, thank you for joining me.
Thank you very much for having me.
I'm super excited to be here and chat with you today.
Backblaze B2 Cloud Storage helps you scale
applications and deliver services globally.
Egress is free up to 3X of data stored, and completely free
between S3-compatible Backblaze and leading CDN and compute
providers like Fastly, Cloudflare, Vultr, and CoreWeave.
Visit backblaze.com to learn more.
Backblaze—cloud storage built better.
So AWS having an outreach group in its security team on some
level just seems like a bit of an aberration compared to most
interactions I've had with large company security groups.
The only type of outreach they tend to do is with
cease and desist or preferably a baseball bat.
What does AWS outreach look like?
Well, second of all, our bats are very high
quality and they are Amazon Basics brand.
Oh, absolutely.
Amazon essentials has made the shirt that I'm wearing today.
I hear you.
Everything is an Amazon in the front of it.
It's how it works.
Well, excellent, because we'll actually be talking
about my shirt today as well as part of the talk.
Because one of the topics that, uh, that I really want to go into today
is the recent expansion of the AWS Vulnerability Disclosure Program.
So, VDP is what it's commonly called.
It's one of the methods of starting the coordinated
vulnerability disclosure process, which is incredibly
important in the software world, or really any developer.
How it kind of interacts today with this wonderful
security world that you're talking about.
is it gives a point of engagement between the large companies such
as, you know, AWS and the security community at large, whether it's
an individual, a company, even some governments and nation states.
Got it.
So a Vulnerability Disclosure Program to my somewhat limited
understanding, I usually find out about these things only after I've
asked a, "That's funny question on Twitter," and then, "Oh no, what
have I just unleashed," is the way of responsibly and collaboratively
reporting security issues or suspected security issues to a vendor.
Is that directionally correct or is there
nuance here that I'm failing to capture?
There's a little bit of nuance.
One of the things that I picked out from the
question is the term "responsible disclosure."
So that has become more of an industry
term now, but it's actually a branded term.
So it's, it's a specific terminology that's meant to describe the CVD process.
And CVD, once again, is the Coordinated Vulnerability Disclosure process.
And I'm using it in a more colloquial sense, as opposed to, instead of tweeting
about it, I would go ahead and send in an email to a, or now, apparently, go
on HackerOne, which you folks are recently making a significant expansion with.
Exactly.
You know, the key difference is that term
coordination, and really what comes into that is scale.
Responsible disclosure is usually more
between a single entity and a single entity.
Coordinated vulnerability disclosure could work at any scale.
And that is something that AWS very much requires as part
of our VDP program, is the ability to respond at scale.
We are incredibly large, and a lot of the technologies and issues that
we deal with are very different once you get to a scale of our size.
So, kind of talking about how we take in these new reports, how we end
up actually doing the remediation and mitigation process, those are all
super important things to talk about at scale and how it's different.
Yeah, I've, I tend to be much earlier in the process because again,
all the stuff that I tend to find is of the form, instead of the
form of, "Haha, I found a security issue," it's a, "that's funny."
And one of the things that I've learned from a lot of folks over
the years is even when you think you've found a security issue, I
always find it best to say, "It appears," and "I think," as opposed
to speaking declaratively, because the first time you tell someone,
"You have a problem here," and it turns out that no, no, you're the
one with a problem instead, no one will ever listen to you again.
Because you're just someone who's going to sound off
and sound confident when you really shouldn't be.
That was a lesson I think some people could really stand to benefit
from, but my experience historically, when I found things that
of the form of "That's interesting," is I would email into the
AWS Security Questions email address, and I would get a human
being responding to it, and it would sort of take off from there.
Is this replacing that?
Expanding that?
How is this manifesting?
Yeah, it's expanding on it.
So AWS has a VDP program, the Vulnerability Disclosure Program.
It has had one in a very formalized sense for at least the last
five or six years, and ten-plus years in a less formalized version.
All we're doing with this HackerOne onboarding is
really giving another input source for our VDP program.
So whether you're submitting through the AWS Security Inbox,
which is aws-security@amazon.com, or if you're going to HackerOne,
you can either search for "AWS" or you can go directly to it.
I believe it's hackerone.com/aws_vdp.
Either of those places, you can submit the same type of information.
We wanted feature parity, and really what we're trying to do is make
it easier and quicker for the community to do submissions to AWS.
And there's some additional benefits.
So, the traditional way of having an inbox on the internet is it's
the front door to the internet, so there's a lot of information
that goes into it and a lot of filtering that has to occur.
With a more traditional portal or console experience, we're able to give a very
structured, safe, consistent process and experience for researchers to submit.
We're able to more clearly communicate to them what we believe should be in
reports by including certain required fields, and we're able to communicate
backwards to them what types of information is and is not in scope.
While it's an expansion of the existing program,
it brings along an additional set of benefits.
I think probably for the community, what they
care about most is the user experience of it.
Currently, right now, when you use an email to do any kind of
communications, you're restricted by a response from either side.
There isn't a way to check current status
unless you ask or somebody sends you an update.
There isn't a way to easily collaborate with multiple people unless
you have a CC line going and then that expands into all kinds of
thread problems with people responding out of sequence in the emails.
Top-posting versus bottom-posting.
I remember the mailing list and Usenet wars over this.
Oh yes, those were the days.
Give me that command line back so I can just grep it.
But one of the nice experiences about, you know, having that portal
console experience is that it gives you a lot of information.
You know, all of your submissions are linked to a single
user account, so you can check your entire history.
You can go into a single report and see all the
comms between two people or 20 people if be.
You're able to search and filter your own
reports based on what you submitted on.
Do you want to be interested in how many reports you sent about service 1, 2, 3?
You can filter on that field.
So it gives a lot better experience for people to pull
information about what they're already working on.
Additionally, it allowed us to adopt a new safe harbor.
So we've traditionally used a modified version of Disclose.io's Safe Harbor.
And now we're using Gold Standard Safe Harbor, which was
written by, I believe, one of the execs at HackerOne, but
he's been very prominent in the security realm for a while.
And this is what he wrote and has been
pretty popular adoption across the community.
A lot of people have asked about it and whether or not we could adopt something
similar to it, so as part of this launch, we adopted it across the entire VDP.
So whether you submit security inbox or through HackerOne, you're under the
same Safe Harbor, the ability to publicly and transparently describe the scope.
So we have two scopes, right?
The scope of what can be submitted into the VDP, and then a tighter
scope of what qualifies for a security advisory like a CVE or a GHSA.
And a lot of times when you're dealing with companies, it's a black box really.
You know, what can I submit?
Does this count?
Can I get a CVE for this?
Does it even qualify?
All those types of information, how to score the thing
that you're submitting, all those details, we are posting.
So there's no longer a question.
You can control+F on that policy and find that information.
So we've kind of built it off of what are the common questions
that we get, and we're rolling that back in to answer.
That all kind of leans toward increased transparency.
Alright, I, as a researcher myself, I've done a lot of submissions
to different programs, the transparency aspect was always a point of
frustration for me, because I want to know what's going on to see if
I need to do work to help and provide guidance, or if I'm way off base
and I should move on, and this increased transparency really allows
me as a researcher to be more informed about the overall engagement.
And finally, one of the really cool things about this is the approved statement.
So traditionally, how do you show recognition for
something that was submitted into a VDP program?
In many cases, the answer historically has been, "Eh, companies
don't," and then they're surprised when the researcher posts a big blog
post, and then they get very upset about the phrasing and the choice.
It's like, for a vulnerability security researcher,
this is the sort of thing that you build a career out
of if you can show a track record of doing these things.
Whereas companies would love to be able to fix the problem on some level
and then never tell a soul in their idealized version of these things.
Because honestly, who wants to trumpet the things that they get wrong?
Few of us, if any.
I'm just weird like that.
There's a certain sen- but again, I'm not saying
that AWS is trying to sweep things under the rug.
Far from it.
Whenever I've engaged with the team, the first and overriding concern
has always been, is there an issue here that affects customers?
And if so, how do we fix that immediately?
At a few calls where I've had conversations before things
went out, there's been someone from PR on the call.
And I was always a little, "Hmm, is this about to
be where they're going to attempt to spin things?"
And never once did the PR person say anything until I dug into.
"Why are you here exactly?"
"Oh, I want to see how the context is and how you perceive this
to be, because you're not going to be the only person that has
these opinions, and I want to make sure that we're prepared.
I'm here to learn.
I'm not here to tell you what to say."
Which I found incredibly gratifying.
Honestly, that's one of the things that I think we
try to push a lot is the emphasis on communication.
We make everyone on our team go through like writing classes and different
forms of additional education, getting trained on the internal services and
terminology and everything to make sure we have a consistent and clear response.
And what I believe you're describing is kind of the difference
between responsible disclosure and coordinated disclosure, right?
Companies that will traditionally accept a report and
then be kind of quiet about it and nothing happens.
Well, they're not doing any coordination.
They are just accepting a report that somebody gave out of the
goodness of their heart responsibly and getting nothing back out of it.
Coordinated disclosure allows us not only to work
with you as an individual, but once we analyze.
And validate your findings and your submission.
We're going to do an additional scope analysis.
So what internally is impacted?
Is it a single application?
Is it 100 service teams?
Is it, you know, what is the scope of impact?
And then if we find that we've submitted is actually in, say, an open
source dependency that isn't us, but our service or application uses.
Well, we're not going to stop there.
We are going to reach back out to you and say, you know, we found the issue.
We understand why it looks like us, but it's a dependency.
This is the dependency.
Would you like assistance in submitting to them and coordinating with them?
And then we're going to provide guidance and
assistance for the researcher to do that.
Sometimes it's through, you know, direct email.
We find a contact point and we just do an introduction.
Sometimes we'll open a Vince case.
Vince is a wonderful, wonderful tool that I don't think a lot of the
community knows about where they specialize in coordination at mass scale.
I first learned that this existed with the
recent CUPS vulnerability and disclosure.
Yeah, that definitely used Vince, but I can see
why that came out and why it's not commonly known.
I, myself, who have been practicing security for 20 plus
years, both at the endpoint side, you know, done the IT work,
did cloud security, was a CISO, did all the fun stuff, right?
I didn't know about it until I started doing, uh, the
vulnerability disclosure portion of the security realm.
Because it was never needed for what I did previously,
even when I did vulnerability management for my companies.
So it is a very cool and useful thing.
People that aren't aware of organizations like First or
Cert should definitely give them a Google or whatever
search engine you feel like using will DuckDuckGo them.
And what you're going to find out.
is that there are these different entities that will do
coordination for you and help you find the proper contact
points for submission of vulnerabilities, and it's free.
Anybody can submit to Vince.
If you want to open a Vince case, you can open a Vince case and be
like, "I found a vulnerability and I believe it impacts these vendors,"
and request their assistance at coordinating and disclosing that
vulnerability with those vendors and getting fixes, getting CVEs generated.
All that kind of information.
So yeah, so kind of going back to how you said responsible disclosure,
and then just never hearing anything back, and then blogging about it.
That's not the experience we want people to have.
We want people to be engaged.
When you submit to our program, you're going
to get A human response in 24 hours, right?
That's, that's one of our SLAs.
After that, the actual submission is going to
get reproduction done and validation, right?
We want to triage it and make sure that it is an AWS security issue.
If not, we help you find the contact point and assist there.
If it is an AWS security issue, we are then going
to take it and ask, "Are customers impacted?
Is it service side?
Where is it?"
etcetera, because that's going to give us an idea of starting.
Do we need to do security advisories?
Do we need to do customer notification?
Do we need to post publicly in mass about it?
Things like that.
And, um, as part of my talk at fwdcloudsec: Europe, we went over.
the actual high level coordinated vulnerability disclosure process.
To be clear, I want to, I want to call something out here because
you folks have been great at the, at the piece that a number of
companies miss, which is continuing to remain in communication with
the researcher while you're doing all this stuff behind the scenes.
Because you can't, you can't necessarily tell them,
blow-by-blow what's happening behind the scenes.
And you can't make this comparison, but I absolutely can, if we look
at the last few years of Azure security issues, and the timelines
that are reported, that the vulnerability researcher reports it,
and like six weeks later, they're like, "Hello, is this thing on?
Did I forget to hit send on the report?"
And you have to goad them into action.
I have never had that experience with AWS.
Not once.
I'm super happy to hear that about AWS.
And I've never heard of anyone having that reaction.
I love that because we pride ourselves on the communication aspect.
On the being ignored.
I have to, this is a full disclosure, in the early days, when it
was an email with a ticket, it's like, "Okay, it's been four days.
Should I not have received an email other than just an autoresponder of,
'Yup, we got it?'" And yeah, there was some, there was some, I think that
at the time the response did not clarify timeline expectations and the rest.
Just because you have an internal SLA of X days to have a human talk
to the person does not necessarily mean that the human knows that.
So there's a, there was a lot of early days teething issues that,
to my understanding, have been smoothed over remarkably well.
I'd say that comes down to two points.
One, uh, we have an internal email rule that just
filters all your stuff to feed the delete bin.
That's generally what I tend to expect happens with my emails.
Cause that's me.
You know, it's standard practice across the industry at
this point, but what's kind of nice is that when it comes
into the inbox, like we have the human triage team guy.
You get a response saying that we received it, so you get that
instant gratification and notification that yes, the email that
you sent did make it across, but then we do have that 24 hour SLA,
which is posted publicly on our Vulnerability Disclosure page.
So that's the AWS Vulnerability Disclosure page
and the policy page on our HackerOne VDP program.
Both places publicly state our SLAs for response.
The commitment that we're always going to make for you.
This episode is brought to you by Gitpod.
Ever feel like you spend more time fighting
your dev environment than actually coding?
“Works on my machine issues” are too familiar and
the VDI setup in your organization drives you mad?
Gitpod brings automated, standardized development
environments to your laptop—and the cloud.
Describe your dev-environment-as-code and
streamline development workflows with Automations.
At the click of a button, you get a perfectly configured
environment and you can automate tasks and services like
seeding your database, provisioning infrastructure, running
security or scanning tools or any other development workflows.
You can self-host Gitpod in your cloud account for free in under
three minutes or run Gitpod Desktop locally on your computer.
Gitpods automated, standardized development environments
are the fastest and most secure way to develop software.
Gitpod is trusted by over 1.5 million developers including
some of the largest financial institutions in the world.
Visit gitpod.io and try it for free with your whole team.
And also, let me know what you think about it.
I've honestly been looking for this for a while,
and it's been on my list of things to try.
I'll be trying it this week.
Please, reach out, let me know what you think.
Something else I didn't realize that you have to deal with, that I, because
until I get to a point where I had to deal with it myself, is the sheer
number of "security researchers," and you should hear the sarcastic quotes
around that when I say it that way, who spam out these emails where they run
some scanner on a website and say, "I found vulnerabilities in your system."
And I followed up with one once the first time I saw it,
and it was, "You aren't hard failing on SPF, email delivery.
It means people could spoof your email domain."
It's- okay, I come from this world.
I would take a 3 percent drop in deliverability of my
email newsletter, because people use forwarding services.
Sure, they're not configured the right way in some cases,
but I'd rather people read my words than be technically
correct against an issue I really don't concern myself with.
People are like, well, I still found it.
Pay me.
It's, you know, That is not how this works.
They're basically shakedown artists.
So you have to sort out the wheat from the chaff when it comes to
figuring out, is this a real threat or is someone with an opinion or
a naive security tool just trying to basically get people to pay them?
Yeah, I think the industry terms for it is
like "bounty beggars" and things like that.
The people that are not really focused on improving
security, but they're looking for the lowest hanging
fruit to try and get some kind of monetary reward for it.
I'm glad you mentioned the word "bounty" because this
gets a little closer to where I am at this point.
I used to work in security and then I decided no, if this makes
me too happy I want something even more depressing than InfoSec.
So I went into AWS Billing.
So the economic incentives behind these things are fascinating
to me, but as I've learned somewhat recently, a vulnerability
disclosure program and a bug bounty program are two distinct things.
Correct.
There's actually three tiers that are really commonly issued out.
The first tier is kind of what we have launched right now.
The AWS, the VDP.
So a VDP is really meant to be a open door to the internet.
There's no reputation limits in order to report to it.
There's a clearly defined scope, but it's incredibly broad scope
and involves, in ours, particularly, we have close to 400 services,
which is pretty much everything AWS has, whether it's open source,
a managed service, one of our client apps, anything like that.
So it's a huge, huge scope, and it's really
meant to have that engagement with the community.
The second tier is usually called a VRP, a Vulnerability Reporting Platform.
And that's where people start kind of playing around with the idea of
moving to monetary, and it comes with a lot of issues of just implementation
to begin with legal stuff, trying to figure out if somebody is really
a contractor for you or not, if they submit a certain amount of things.
There's a lot of questions about that people want to deal with.
And then the third tier is what people are
most commonly called, "the bug bounty."
And that's usually a private invite-only program.
So you go public-wide, public-really small, and then private-small.
And you kind of ramp up that way.
It's a progression of maturity into the programs.
And in one of your slides at fwdcloudsec: Europe this year, you mentioned
that there is an invite only AWS Bug Bounty Program, if I'm not mistaken.
You are correct.
It's actually one of the new reward systems that were publicizing for the VDP.
So this was always an aspect that happened is that people that
report to the security inbox, our VDP program, we would recommend and
nominate different researchers to be invited into the private program.
But that's only one of the benefits of it.
You know, we also have a leadership board.
So with your reputation points, you can see if you're
the top researcher of the day, week, month, year, ever.
That kind of fun stuff.
We do swag vouchers and stuff.
We have promo codes.
So this week alone, I've given out a ton.
Please tell me that this is not the epitome of the Bug Bounty Program, "Wow!
You found a zero day remote code execution, hypervisor escape on EC2.
We got you a t-shirt in return, but you get to pay the shipping on it."
No, sir.
Seems like it'd be better not to have that kind of program.
100 percent agree.
As part of building this program, we got an official merch store.
It's an AWS merch store.
The shirt I'm wearing right now is actually from it.
It's, it's actually really nice quality,
which is why I chose to go that direction.
But no, your rewards are
incre- are entirely tied to severity.
Right?
One of the things that we are doing right now, the program is
new, people are starting to use it instead of the security inbox.
So pretty much anyone that submits a valid security issue or concern
to AWS through VDP, when we resolve it, we're issuing you some swag.
And the swag promo code covers a certain allotment
from the merch store, and covers shipping.
Yeah, I made sure because the first round
didn't, and I had to ask questions about that.
Like, "Cool.
Here's a present.
Pay for shipping."
No.
No, no.
That's not going to happen.
But out of that, we also, a couple other like non-tactile things, like
it's really hard to reward people across the world with physical items.
So we're looking at other ways to do so.
So skill badges is an attribute that the HackerOne platform has, and right
now, the same thing as when you, how we issue swag, we're also issuing out a
badge for the service that you have successfully submitted a valid report for.
This will do two things.
One, you can use badges wherever you want, LinkedIn, your resume, etc.
And you're able to count from them how many
successful valid reports per service you have done.
But also, we're going to be tiering these out, and after
you do a certain number of submissions, you get a new gold
badge, and so on and so forth to show you're upscaling.
And we use those as part of the nomination process to figure out who's
great at what services and would get special advice to special things.
I'm going to find one on Route 53, and I want that gold
badge to actually be a physical trophy that you send me.
If you find something nice and shiny, I will commit to
getting you a trophy, but I get to choose what's on it.
[laughs]
I think that's a fair approach because, oh yeah, good point.
Someone will try and out snark me.
Oh god, that always ends well.
I don't know, uh, for folks who are looking at me on the video, behind me
I have a Minecraft pickaxe, foam thing, that people always ask me about.
Like, where did that come from?
I wound up writing an article once saying that AWS's approach to machine
learning was like selling digital pickaxes into a gold rush, and the
head of machine learning analyst relations sent me that in retaliation.
It was genius.
It's where is this creativity in all of the
conference talks you give about this stuff?
It sounds like Star Trek Technobabble.
This is amazing.
I get more commentary on that than I do the custom commissioned
painting of Billy the Platypus fighting an AWS Billing dragon.
I love the imagery, and quite honestly,
like, that is a great, great marketing thing.
I've been trying to get somebody to build inflatable
bandhammers for swag giveaways at, like, conferences.
You know, big inflatable one, has squeakers on
the end, because one, it's easy to travel with.
Two, your kids are going to love it.
And three, they can't hurt themselves with it.
So it's like the ultimate giant bandhammer.
And, uh, anybody who is listening and does that and brings it to a
conference I'm at, I will hook you up with some swag because that is awesome.
That's a terrific idea.
We have a booth at reInvent for the first time this year.
I'm going to run that past my team and see what they say.
The problem is I'm not quite sure, "We fix AWS bills here," have a ban hammer.
Like that, that doesn't quite align in the same way,
but I love the idea so much I almost don't care.
Ban large charges with the ban hammer.
I don't know.
We'll work on marketing stuff later.
We can definitely torture a metaphor to death to make this work.
Yeah.
I appreciate that.
And that's what I appreciate about you is that commitment.
Exactly.
It's the, uh, what is it?
Uh, which has at least two or three leadership principles there.
Bias for action.
Dive deep.
Yeah.
Definitely
Dive deep.
Yes.
Uh, are right a lot.
Oh, I tell people I'm right a lot, which is kind of the same thing, right?
I mean, you got to fake it until you make it.
Disagree and commits, et cetera.
Oh yeah.
We have all the fun ones.
I like that.
So what's next?
You've, you effectively have fought what I can only assume
were political headwinds to get this out the door, even if the
only headwind that really existed was simply corporate inertia.
At AWS's size, that is no small thing.
It's Incredible that this saw the light of day.
It gives me hope that, okay, maybe there's a glimmer of old Amazon still
shining through sometimes, and maybe there's going to be a different future
than the last couple of years have been, but what are you aiming at next?
So I am incredibly proud that we got this out, and it's
a natural maturity progression to the program, right?
As you grow and scale and work on the efficiency
internally for how your mechanisms work, you're able to
bring on more external sources and more input sources.
So a couple of things I'd love to see us grow with is through
the feedback mechanisms we have, whether it's us asking surveys
on things like the cloud security forums or at conferences.
Oh, you do love your surveys.
You carpet bomb the world with them.
I love my surveys.
Ours are a little different though, because we
don't have an automated mechanism to do them.
So I manually count stuff.
But what is great is we have a few different
ways to get feedback now than we didn't before.
Mainly, on back one platform after every report gets
resolved, you have the ability to give some scoring on your
experience, but you also have an open box to give feedback.
And something that I constantly am asking people is, you know,
as somebody who was a researcher, I built it for something that
would make my life better and makes me happier with the experience.
I need more feedback from active researchers and
community of what you want to see in the program.
You know, you ask for more transparency.
We have a lot of public documentation now.
We gave that transparency.
You wanted to make sure that we were meeting certain SLAs for responding.
We're publicly committing to those SLAs.
So really, what's next?
Well, people asked for like testimonials.
Things they can put on their resume.
Things that they can show their work for, more recognition methods.
So besides badges, which are kind of cool on the platform,
we're also doing public testimonials and personalized accolades.
So this is something that will get customized for your report
as well as the experience of working with you as a researcher.
So not specific to your report, but you as a researcher, what the
engagement was like between us, um, and those are connected to your profile.
So you're able to pull them as basically like accolades for
you to show you had a great experience with a company like AWS.
How do you maintain that?
Because I, again, I haven't seen this from AWS, but I've heard stories that are
legion for a number of others, where the entire company's perspective is damage
control and trying to convince people not to say a word ever about any of it.
How do you fight back that, that natural force of, we have screwed
up and we preferred not to see it on a billboard somewhere?
It goes back to, you know, security is kind of job zero.
I understand how people will be afraid to talk about their
flaws and stuff, but security by obscurity has never been
an effective method of defense for a long period of time.
No matter what in the digital age, information will always find
a way to see the day of light, or, see the light of day, and,
really, what we're focusing on isn't trying to sweep things under
the rug, but make them valuable for the rest of the community.
You know, we're able to do more disclosure now as well.
So part of the VDP is an option to disclose your report.
Yes, there are some redactions in it, but it's mostly like
PII and stuff that you probably don't want to get doxxed for.
So we'll scan for that and make sure that gets
redacted, but your report can get publicized.
We give public statements now about all of your submissions.
So if you are writing a blog, not only will we collaborate with you, get
internal subject matter experts to give a technical review of your blog and give
feedback to make sure you're stating things in a technically accurate way, but
we're also going to give a statement on AWS's stance so that way you can use
it in your blog or your presentation or whatever you feel like doing for that.
You've become much more consistent about repor- about in the
security updates that you put out on your security informational feed
thanking whatever security researcher brought it to your attention.
You didn't used to do that.
You do now.
That is correct.
We have a group called Security Communications, our, our lead
for that has been excellent with showcasing improvements based on
community response and submissions from all across how AWS Security
communicates So outreach, my team that I work on, I work as tech lead.
We're just one of the teams in that org, and, but we also have all the
messaging as our sister team, and we work heavily with them on getting
out not only internal customer messaging, and the types of messages that
you get as like your dashboard update or things like that, but also the
mass communication emails to give guidance to possibly impacted users
and companies, but also things like security bulletins, GHSA posts, CVEs,
all those security advisories.
We write all of that stuff as well, so it's just a huge boon.
So we're constantly improving the process and assessment for those types
of messaging has been greatly improved over the last couple of years.
Messaging stuff is important.
I was getting tired of, "Security is job zero," as a repeated talking point
was on a bunch of conference slides, and great, I didn't believe it or
internalize it or- until one of the most impressive things I've ever seen
in a security disclosure was when, actually, AWS had an issue with AWS Glue.
And the the analysis on it on the website was, included the
phrase, "We have looked back at the previous six years of audit
logs going back to the launch of the service and the only time
that this was ever executed was when the researcher found it."
Which is- that makes me feel better than I was even
before learning that there was an issue at all.
It demonstrates an attention to detail, a defense in
depth, the response wasn't, "What could my logs be?"
It was one of the most reassuring things I'd seen, and it
It really did, it really showed this was something that was
internalized and believed rather than just dead words on a slide.
If we ever do find impact on something that has been submitted, the
the depth that we go through for analysis to find scoping and do risk
assessment to AWS and its customers is incredible in my opinion, right?
Not only like how far back we go, it's like,
great, you know, years and years of data.
Well, think of the scale of that data.
You know, if, if one customer can create petabytes of data in a day for logging.
Now multiply that by all of our customers that use that
service, then crawl through all of that, and still do that
in a reasonably fast manner that has accurate results, and is
able to give us confidence to say it is only executed once.
I've been part of multiple engagements in the last few years, and another
one that was very similar for that kind of look back was one that, I
believe it was Datadog, it was Nick Frechette with an Amplify issue.
And what ended up occurring is that we went back, and we looked
to see when it got fixed, when it got unfixed, all this, all the
way back to the start of the service entirely, which is years
and years old, and we're able to find some variants of the issue.
Um, so I know I kind of rambled there, but the emphasis on the
lengths that we go to for the investigation portion of the responsible
coordinated vulnerability disclosure process is just boggling sometimes.
Just the amount of data that we go through and validate.
It's really cool to see.
I'm a big data nerd in general.
Probably know from being the co-author of AWS Detective is, uh, it's big data.
We love big data.
You're working in the right place for it.
Oh, yeah.
And what's really cool for me as well, like previous lives of
being blue team and then going to red team and somewhere in the
middle is you get kind of a perspective of this is how somebody
is thinking to get in because this is how you would get in.
Now let's figure out how to break that way forward
so they can't get in, and now I'm a defender.
But seeing what gets submitted and seeing how some people think, and you'd
sometimes get to just put down the white paper, take a breath, and try to
think of whatever trauma this person has gone through that has made them
think possibly in this way to make this type of an attack vector work.
Like, what would ever make you look at doing something?
And it's incredibly fascinating to me because the
security community at large, they think different, right?
Uh, it's not how to get from A-to-B-to-C-to-D, but going from A-to-D,
and then B and C just worked out, and now you're backtracking, and how
you made your connections, and you look like one of those crazy people
maps on the wall where you're trying to find all the connections.
And, but the community is just putting them together based on their
usage, their experience, the artistry of kind of the intuition
of what they should keep poking at is always fascinating to me.
I love that aspect.
I've always envied the skill set.
It's one that I don't have.
I just blindly trip over things every once in a while.
It's as I am.
It's not the, "Eureka!"
moment.
It's the, "That's funny," moment.
What I always love to say is like describe what, you know, the
people do for a living in kind of our, our vector of the world.
And I say, "Well, they go to the dark places on the internet,
poke it with a stick and see what crawls out," right?
That's threat research and intel and all of that fun
stuff, but what we don't talk about is, well, when the
bugs crawl out of the dark, who's going to raid them?
Who's going to, uh, spray that thing down?
Who's going to make it known to everybody else that these bugs are crawling out?
Well, that's the research community.
They already, you know, they shook the tree.
They saw what came out.
So now that they saw the bug, it's really kind of that moral/ethical choice.
Do I tell people about what I saw crawl out?
And that's kind of where the VDP comes in is people can share that information.
They don't have fear of like retribution of it.
We have a Safe Harbor.
We have a very clear scope of what is and is not in bounds.
So as long as you read that you're, you're good to go, and I think
that additional, I don't want to call it a safety blanket, but the
additional assurances, I think, make the program very successful.
I think that that's, that's really, the proof is in the
pudding and in staying the course for a, for a period of time.
I want to thank you for taking the time to speak with me.
If people want to learn more, participate, or give you some of that
vaunted feedback you're after, where's the best place for them to find you?
The best place for feedback right now is either as the submission
to the HackerOne program, which is hackerone.com/aws_vdp, or
directly to our security inbox, which is aws-security@amazon.com.
Those are the best places to give feedback on the program specifically.
But if you ever have questions or concerns, things
like that, you can find me on the cloud security forum.
I'm on that Slack.
It's a great place to network with other people that
are working in the cloud, research or otherwise.
And then on LinkedIn, I believe my LinkedIn URL is
https://www.linkedin.com/in/cloudy-with-a-chance-of-security/.
I am always happy to talk with people and network.
You'll find me at a ton of conferences.
One of the big perks of being in outreach is
our emphasis on engagement with the community.
You'll find me at all the major conferences.
I like to go around and chat with people and hang out.
Black Hat, DEF CON, The B Sides, fwd:cloudsec, reInforce,
reInvent where we have a presence, and if you want us to come
check out a new conference, shoot us a message and let us know.
We'd love to talk with you and meet more, and we'd love to
probably host some more events and just get the community
together to talk about what they're interested in, how we can
improve, and what people like and don't like about it right now.
I really want to emphasize that this is a
program by a researcher for researchers.
I want your experience to be pleasant because I want to recognize the
type of effort that you're putting in, the type of time, the hours,
the creativity that you're putting in, and then also the fact that
you're taking the effort and time to write it up and let us know.
Thank you.
Like, I want to sincerely say thank you for
that because it's really, really important.
And we will, of course, put links to this in the show notes.
Thank you so much for taking the time to speak with me.
I appreciate it.
Thank you so much for having me.
It's an absolute pleasure as always.
Ryan Nolette, Senior Security Engineer at AWS Outreach.
I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud.
If you've enjoyed this podcast, please leave a five
star review on your podcast platform of choice.
Whereas if you've hated this podcast, please leave a five
star review on your podcast platform of choice, along with an
angry comment that points out a vulnerability in my platform.
Then I will follow my own policy internally, which at the moment is
a post it note that simply says, "PANIC," and then call the police.