Hello and welcome to another addition of Black
Hills information security talking about news. We have
a cast of characters here to talk about
the crowds thing, but before we get started,
I just wanna point out,
you're gonna be fine. It's gonna be okay.
Take deep breaths.
Unless you're in an airport, then you're screwed.
Or if you're running bit defender. If you're
running bit defender, I saw somebody recommend on
Twitter it's time to just like, walk out
of the office and give up at that
point. But other than those things, it's gonna
be okay. We think most likely,
So today we wanted to put this together
because ambulance chasing is fun. We wanted to
put together a bunch of people that kind
of understand what's going on. So we could
talk about it, but also comm commensurate
if you're on discord, I wanna see those
memes folks? Like, what are some of the
best memes you're seeing from this crowds thing?
This is cath.
It helps us get through through these things.
If I come across a flip and I'm
joking and all of that, I'm not doing
that to make light of anybody that's dealing
with this situation and really struggling today. I'm
doing it because we're trying to use this
as laughter as the best medicine and we're
laughing hopefully with you. We're not laughing at
anybody I also want make it very clear
that also goes for the engineers at Crowds
strike.
It's fun to make memes it's fun to
make jokes every once in a while, but
I want you to understand on the onset
before we get started on this. Software dev
for very large software projects is
incredibly difficult. Kernel Dev is even more difficult.
So please have a heart for the engineers.
As far as the salespeople people that are
trying to capitalize on this, don't don't give
them no quarter at all. But honestly,
we really wanna make sure that everyone understands
this comes from a place of love. We're
joking about it. We're trying to get through
it. And it's kind of a little bit
of gallo humor. I have Jo T to
help talk about Colonel stuff and what's going
on. Patterson Cake is here, runs our Ir
practice at Phi.
How many reboots is your son up to
now. We're it we're 18 and counting. We're
at 18 counting. 1 of those solutions was
to keep rebooting the system 15 times and
then magically the network stack would come up.
Get an update from Crowds strike. His son
is doing that for us, he's on 18.
Now Patterson
anytime any... Like we get into a wall
in this. It kinda gets into that just
shout out how many reboots were out. We
just need to know because I was worried
that that was just a lot of, like,
hot air, but we're 19...
19 counting.
I got John Hammond. If there's anybody that
knows malware knows virus knows with going on,
except for today.
All like, we're all like, what the hell?
So I called up, John, and I'm like,
dude, What the hell going on? He's like.
I don't know. I haven't slept much, man.
It's just really, really super busy, but we're
really happy that John is here to help
us talk about it.
Matt is also on who... By the way,
Matt works for Black Hills information security by
the wearing this today. He is he is
a full time employee of Black Hills information
security much of the soc and surprised to
other people at Black Hills information security.
But he also is, like, bypass and exploit
kernel level stuff at B I.
Derek been around for a long time at
B his. I think he's like, employee number
6 or 7.
Something. And he just keep showing up to
these things. But he also is the lead
on our sock services,
so we wanna bring him in on this,
and then we've got Jason. Who you all
know, Jason is just like, here at kind
of when we get 2 technical, dropping in
questions, bring in hot takes, buddy little jokes
and little jab, who And behind the scenes
we have Ryan,
and can everyone send Ryan, your your love
because he has courage.
He should not even be here. His boss
is an asshole for making him come to
work today. And, he's he put this all
together for us. So round applause. For, Ryan
for making this happen, we really really appreciate
you, Ryan. He makes us look good and
sound good, even whenever he doesn't feel like
doing it.
So, John, you gotta jump off because you
got a very, very, very busy day.
I first learned about this on Twitter, I
jumped on
and I saw a tweet from you early
this morning
about you're, like, hey, there appears to be
some weird things happening.
And I noticed it was, like, 05:00 in
the morning, and that tweet was 3 or
4 hours old. I was like, what the
hell? So you've been working on this quite
a bit. You're probably gonna have 1 hell
of a day. That wanna give us kind
of a rundown of when you first started
hearing about it some... Because you're 1 of
the first people in the industry that I
know of that was kind of, like, throwing
the alarm belt. You kinda give us a
little bit of... Timeline from your perspective
about what's happened here?
Yeah. Thank you so much.
Well, hey, I'll admit. I'm over in the
Pacific time.
Kicking it over in California. So it was
like, 10 or 11:30 ish, and I was
thinking like, well, goodness. We gotta get a
video out tomorrow for some other thing that
I wanted to do, But then I saw
this
fire start to burn in on the subreddit
crowds strike? Crowds strike subreddit.
And folks trying to, hey, starting to flag
it, and I really have the hunch. This
is gonna absolutely imp.
Now were you seeing some of the the
the earliest stuff that I saw appeared to
start in Australia and New Zealand. Is that
kinda of where you started seeing it pop
up there? I know. Due to the time
zone, those folks are awake and still tracking
things are the day? I'll admit. I think
the earliest I saw was 09:57PM
Pacific, and that was just the starting gun
on the crowds subreddit. But I believe everyone
chi in, but all the comments and responses
to said, hey, yeah. We're already down and
have been fighting fires here in Australia and
New Zealand whatever. Etcetera. Mh.
And then Us, and I think the rest
of the world started to wake up to
it slowly but surely.
What else can I color the picture on
here for?
So so whenever you first started seeing this
pop up, what was the first ink? I
mean, we know that as there was something
I mean, it seemed to me like we
knew very early on. It was Microsoft and
crowds together. That's what the issue was. Right?
But it took a little while to figure
out exactly what was the sis driver?
Like, how long did that kind take from
Has. That everything is on fire.
Here's possibly what it's causing to causing it
to be on fire? Was that, like, an
hour 2 hours, how long do you think
that time window was between Oops sees? Too.
Here's what we gotta do to deal with
it. It's actually pretty neat. I think you
can see, and I'm trying to see if
I can
track it down in the reddit post just
as well there was I think about a
20 minute delta from when... Hey, some sales
engineer representative from Crowds said, hey, we're aware
we're seeing widespread reports pretty good bath, etcetera,
etcetera. We'll try to get a a tech
alert or whatever they call their advisor messaging
and knowledge based stuff.
So that was a 20 minute delta, pretty
And then about 15 minutes following the technical
alert came out. And then
around that time, I would say 40 to
50 minutes, maybe an hour is when they
identified. Hey, we are seeing this cs s
agent dot sis colonel driver being problematic.
It looks like I... And I'm not super
a thousand percent confident on whether it's loading,
some of these c hyphen 00291
Dot whatever dryers pile. Yeah Yeah.
But those are the hiccups. Those are the
issues. So you've got folks trying to share
some quick scrappy workarounds and
mitigation of, hey, just rename the folder path,
rename it so that way it won't load
the driver.
Bear in mind that will completely triple crowds
strikes, so your mileage may vary with that.
I was kinda going back and forth screaming
and shouting on Twitter when I saw Brody
from, I think the director of Overwatch.
On at Crowds strikes screw. Saying, like, look,
here's a little bit more pointed a little
bit more targeted. Make sure you can get
into safe mode
and use the commands to remove the C29001
sis
suffix,
colonel files and drivers. And that was much
sharper and much more accurate than
just rename the folder path and Yeah because
it doesn't much strike Judge completely break crowds
strike at that. Right. Right. Right. Though.
I don't wanna, hey, ramble for too long.
So if I'm up he's strike look. No
No. Please. Please please. No. But, obviously, I
think a lot of folks will raise their
hand and say, hang on wait a second.
You really want me to do that manually
across... However, many 50000 points.
Yes. That is the crux of the issue,
and you can
expand that and extend that to whatever... Scale
and size you might like. And folks are
saying, like, well, wait a second. That only
works if you aren't using
bit bitlocker encryption,
it's going harder to do that at scale
because right, you'd need the bitlocker recovery key
to be able to get into,
that safe mode to.
On on reddit, that I'm hearing, like systems
administrators that are straight up, Like, yeah. I
need to get the recovery keys. It's on
my notebook computer.
Which is blue screened, and I don't have
recovery keys for that system. Like, it's bad.
Like, if... I mean I was joking about
the whole bit bitlocker thing, but This is
1 of those situations where our security products
are getting in the way of our security
products at point. Not just bitlocker. It's probably
any encryption. It is any.
Right?
Got it. Just watching discord explode with the
memes. Keep them going, keep them annoying. I
don't know memes. I just saw the Steve
Ko meme pop up. And
I never thought I'd see a Steve Ko
meme in an It security webcast, but here
we are in Ko Khaki.
May I... I'm sorry. I don't mean to
stop no. I'd I I'd love to have
all of your hot take in your ex.
Expert opinion pigeon because I know there's been
some chatter where folks are saying, hey, you
can use group policy, push out some automated
tasking and then forced the safe mode for
configuration change, blah blah blah, and that is...
I think what some folks are saying, oh,
might be some
to this headache. But wait wait a minute
know.
If we look at this, you can't even
boot. Like more back isn't coming up. Right.
So group policy, I don't know who... Unless
there's some kind of new thing that's out
there And Like you said, if you're on
discord with us. If you have a link
about that, that would be great. But, apparently,
1 of the things they said is reboot
the system 15 times, the network stack will
come back up,
Crowds strike can reach out and pull down
an update that nukes this file and solves
the issue. We're testing that now. Patterson, what
do we at for reboots?
I, I fear at, 18 or 19, my
my volunteer staff,
quit.
Yeah.
There...
My volunteer staff is directly struggling with this
issue in the enterprise, and so and all
seriousness,
They they were testing this as a potential
solution and and it's just a a non
starter. I I've seen reports that it's worked
for some folk. I have 2 2 direct
contacts and 2 significant enterprises in both have
failed to see any resolution literally up to
18, 19 reboots on a handful systems.
Yeah. Your mileage may vary, but that is
the pretty clearly not a silver bullet at
this point unfortunately.
Yeah. That that that test of 2 systems
that we're going on not looking good.
George just pointed out, let's go to 50.
Let's see what that does. Maybe that'll do
it. You never know. You never know. You
never know. Once you get to the hundred,
it's that's working out here. Yeah. Just. A
quick question for everyone here. Like, it it
used to be Info tech Twitter was the
place to go and So find answers and
solutions. Like, is there a centralized place right
now where people were...
I'm I still went... I still went to
Twitter.
And read it. Those are the 2 places
that I'm at.
This might get me back on the Twitter
because that's where I'm getting probably the best
information and the best communication with
some of the top people on this, and
I don't know what you all think, but
it's reddit and Twitter for me is where
I'm I'm getting all of my information on
that. And, of course, for the His discord
server. I would also add Linkedin into this.
I'd not want to usually promote Linkedin for
a threat intel, but, be very... It's actually
really surprising how much sharing has been done.
Just trying to follow along There's been a
lot of sharing of code in reverse engineering,
kinda figure it all out through Linkedin. And
so Put Linkedin.
Yeah.
Yeah. I'd I'd I echo that to you.
A Twitter Twitter and Linkedin combination there,
I I tend to lean into Twitter first,
and then, lean on my friends like Matt.
Yeah,
so, yeah.
Who is an employee of Phi by the
way, just for the wreck.
Oh Yeah. I've heard that. Yeah.
So it it's it's kind of interesting that...
Matt and I work very deeply
with lots of malware development. So the minute,
we heard,
of this issue. We both were like, we
know exactly what this is.
So oh, my camera's mirrored apparently. Okay.
Does it... Which is my best side. Is
it this side with this
but it's the side that has the hat
on? You need the hat. Yeah. I need
the hat. Right? I'm, I'm going to mirror
my camera. So that was scary. My god.
That's nausea. Yeah. I saw a lot of
people.
Can you give us a little bit of
information like, okay. So we know that this...
This file is a channel file, And I
scrambled all over the Internet trying to figure
out what the hell a channel file that's
actually, how I opened up my conversation with
John and invited him on.
And but, you know, because Crowds people are
saying, oh, it's a channel file. It's not
like a full update. So what I was
able to find out from channel files from,
Mike Fell, you stay ready, who's over at
trusted, a good friend of B his.
Is this is a file for minor updates,
definition updates, very, very small tactical updates.
And the thing that I was kind of
working off of and trying to figure out
is this this something that's generated through automated
means like God help me for saying this
artificial intelligence?
And threaten until pushing updates down systems or
is it manual.
And he said based on his information, which
may be out of date by a couple
of years
it was very much manual.
It was a manual like file for these
updates that could pushed down, but like little
quick tweaks.
So it wasn't an automated thing, and then
people started cracking this thing open,
and it was full of null. It was
full of zeros. Now Can you and Matt
talk a little bit about where does crowds
strike exist
as it relates to the hurdle?
And, like,
what what the hell going on with AAA
file full of null
coming into it and like, how that impacts
the kernel itself? Yeah. Yeah. Absolutely. So so
when you're looking at,
these defensive products,
a modern day, Ed, D,
marketing acronym of choice.
There are always,
we're typically in in in well developed products,
layers of that product that operate in the
kernel of the operating system and also operate
in what we call the user land space.
Right?
Kernel development, when you when you look at...
So the channel file that we're actually talking
about has an extension of dot sis.
Are in in Windows land, that means it's
a kernel driver. Okay?
Kernel... A driver development is a very specialized
skill.
It is,
it is a skill that you have to
get
exactly right. Because when you're developing code in
the kernel,
there are no safeguards.
You are responsible for managing all the memory
allocations you are responsible for for managing all
the resources and releasing them correctly. You are
responsible for managing the memory space,
of the a driver that you are developing
and maintaining and you... And you are God
at driver level. You can write to anything
in the kernel
and the user space of the operating system
that you want to write to. So it
is a very high trust
development
environment.
Okay? So if we go back historically for
a minute, there there was a time pre
Windows vista
where
most
antivirus products at that time
used to put hooks into the kernel,
Ss, there's, it's a des distributor table
that various kernel calls we're using, and they
with it was it the global interrupt table
that they were hooking into? Wolf that's local.
It
for each job. Okay. Go ahead. Yeah. It
was global. And,
This
Microsoft
started getting upset with the community because
if they made a mistake doing this,
they would destabilize the kernel and result in
in what we're seeing today, right, which is
a blue screen of death. Otherwise known as
as a colonel crash. Right?
And so Microsoft,
with the release of Vista kicked
the the
virus community at that time out. Of the
kernel, and they introduced the technology called Patch.
Patch,
basically,
said, you can no longer directly patch the
des descriptive tables in the kernel. You're not
allowed to period. And if you do,
we will bug check the system,
and others, we'll will crash the system. Right?
So
naturally, the
virus community at that time,
and the emerging, you know, Ed x d
community eventually.
They were not too happy with that. And
so what happened as we move forward is
These,
vendors started doing what's called user land hooking
in their products. So they would patch into
in the user space not the kernel space,
the D in windows that we all know
and love as nt D l dot d,
which means they were redirecting not Api calls
in the Windows operating system so that they
could get telemetry about what's going on in
the system.
At the same time,
They were petition Microsoft. You've got to let
us back into the kernel.
We really really don't like this situation of
having to hook every process
and do all this extra work in user
land just because you won't let us tamper
with the Colonel.
Eventually, and I don't know what actually transpired,
of course, I'm not a my Microsoft employee,
but eventually, Microsoft Rene and said, okay, we
are going to give you some limited means
for you to
develop drivers in the Kernel and listen to
telemetry.
And because that was their primary request. And
the telemetry came in the form
of what's called kernel callback
notifications.
And there's multiple different kernel callback
notifications that are involved with colonel drivers. There
are. 1
I wanna jump in real quick on that
history.
There was a huge dust up between S
semantic. And Mca at that time because they
were the main antivirus vendors. And basically, what
they were planning on doing was just exploiting
the Windows kernel to get the level of
access that they need it
and before it actually right really ugly in
the streets because that was the threat
antitrust. We're just gonna write exploits for it.
Microsoft caved, and this is the kind of
inter in a immediate intermediary solution. So go
ahead, keep going, jo. Yeah. Yeah. So that's
exactly right. So These kernel
callback
notifications are in the form of
a table in memory.
And what happens is
that when a driver installs itself and gets
loaded,
it will
register a callback
notification
that that driver wants to listen to. And
basically, it's telling the Windows kernel when certain
events happen, tell me about that event. And
there are names for these things.
There is a process create notified routine, a
create thread notify routine, a load image routine,
a object register callback routine as well as
a
registry,
activity routine. So
the community was relatively happy
that they got the telemetry back that they
need it,
But bear in mind,
the when you are developing
drivers that listen to these callback
notifications,
The kernel is still vulnerable
if that driver that it receives the notification
does not...
Do the right thing, essentially and process that
notification in a timely manner
and allow the kernel to continue operate. Right?
Now the worst of all possible scenarios which
we are experiencing right now is if the
kernel
notification callback routine in the developed driver
crashes or it is not able to be
called. Right?
And from our understanding to The colonel
notification callback that was registered
by the crowds strike driver,
called out for it to make a notification,
and
encountered a driver that had bad or missing
coat. And if that immediately causes the Windows
kernel to crash.
By design because it's a destabilizing effect.
And so
it it's a mixture of things that are
going
on here. Right? It's necessary for Ed and
X to receive
telemetry via these kernel callback
notification
registration events, but there's still a danger and
it's very critical that the driver is developed
very carefully to process these events, in a
timely manner and to not destabilize the k.
Right? So this is going to be where
the discussion is and, you know, you can
go further than this,
to understand this is not the only
component of Eds and X. While they are
si
telemetry, those drivers are also interacting with the
service process that's running in user,
typically,
and,
you know, providing that telemetry back to Crowds
strike,
maybe
getting some signature data, that kind of thing.
That that you know, a lot more of
the activity of defending is happening in user.
That kernel driver is typically acting mostly as
a notification facility. Right?
So that... That's what we're seeing here. That's
that's as near as I can understand what
happened. Now...
Our
information, and, I think Matt can put up
a slide on this is that the kernel
driver that was pushed with the update,
got installed,
was in fact full of null characters, which
means that the put
notifications that immediately hit that driver would have
hit essentially
no coat.
And that
absolutely would cause an immediate blue screen of
of, deaf. And, there has been some reverse
engineering that has already occurred to actually support
this. Matt, I wonder if you could put
some of that up can you share your
screen and talk about your your things come?
Yes. I actually just posted the pictures in
our chats through shared? Yep.
So we go to the other picture for
a second.
You'll kinda see that there's already
under the exception code and access violation, which
she's kinda of what Jo with talk about.
But... Yeah.
To you point about the update being full
of Null. What we kinda see right here
was this was paused because
as Josh mentioned that Arnold was expecting something.
And
by doing... I wanna stress a little bit
of research and a little bit of digging
before this,
it looks like it was expecting
an actual unit code structure of an address,
rather it got back this error code. And
because of that, it crashed
because obviously, the update was full of Null
and didn't have any routines or functions. So
that way, it didn't return valid result.
Causing this screen? Yeah. Just just to put
a finer point on that, when you see
an exception that is access violation in Windows
what that typically means is and I see
these all the time because I spend life
in a debug.
What that typically means is that what A
register in the Cpu has received an address
with all zeros in it, it has tried
to fetch memory at that address.
And that is by definition and access violation
because there is no address of 00000,
whatever. Right?
And this level. Just to be clear, there
is no structured exception handling. Like, when you're
working if there's no
Not There is no recovery. Like... Or it's
no recovery colonel, there is no such thing
it's structured exception handling coming. Yeah. It's just
not not a thing at all. So the
gotcha which is why Microsoft is so kind
of always been cautious about letting third parties,
develop drivers,
especially for this reason.
When you see this a lot with, you
know, in the gaming industry, especially the graphic
card drivers and stuff, 1 small thing crashes
a computer, and all of a sudden, if
your code is introducing that variance.
Send you a large player basic cannot play
and your game is dead in the water.
But what I actually wanted to kinda of
show is that second picture now,
which kind of let me down, there must
be some kind of issue with it. If
you see it under fail bucket Id. This
unknown function because of that structure, it was
supposed to jump to an address. To continue
on the code. But because it was full
of zeros and it returned to Null, it
actually changed and caused
to jump to an unknown address so far,
this is what I've been able to blow,
and that's actually been part of the root
cause of this crash.
Right.
And so the only way to recover from
a situation where you have a bad driver
in the kernel that already has pre existing
callbacks in a running system. You've gotta remember
that
the driver itself was written over,
but the pre existing callback already in the
callback tables. And so as soon as that
first
notification callback went to the new driver
immediate screen. Right? Mh.
The remedy
is remove the bad driver.
Okay? So that,
crowds strikes user code then once it boots
up
Presumably, I don't notice for a fact, but
other crowds abstract customers probably know this. The
user level code would reach out for a
new update.
And
as I understand it right now, Crowds strike
has the corrected
patched version of the driver again you'll continue
on your way. Right. So I wanna ask
you guys a little bit about load order.
Right? So it sounds like the majority of
the Windows kernel fires up, but
But we're not getting network stacks going enough
to be able to go to crowds stretch
to pull this off. Tried doing the 15
reboot trick. Right?
It seems to me like, this is, like,
where the where these drivers load. Yeah.
Is actually before the network stack
It is that you you will not get
a stable kernel with this bad driver in
place. I mean, in my opinion, Matt, Go
ahead. Well, so obviously, when we look at
an Ed, we think about it, they wanna
know what it's going across the wire as
well as what's going quickly across... Well that
boot. Process. Right? So they put... They're kind
of putting themselves at a higher priority to
be loaded first over the network stack. And
that's kind of where this problem of, you
know, process You even it on a reboot,
it's not getting the update is because by
the time, network stack gets called. The crowds
D or... So the crowds drivers already.
Loaded and it crashes. Right? And if it's
a driver full of null, it'll crash as
soon as it loads, tries to load that
drive driver So you're
I think we've said it a lot, but
I think the most interesting part of this
whole takeaway is a driver update full of
null. Driver. That's what I keep going back
to. This doesn't seem like a developer screwed
up and writing their code poorly. This almost
feels like the file was corrupted. Or something
something something in there. I've I've heard a
lot of mixed things about people saying that
if it's Ai generated. I don't believe that
I think... What other what not from what
I'm hearing, this is not a file that
Ai generates automatically. That's... And I and like
I said, the people, like, if we're talking
Mike, or you stay ready on Twitter.
Mike knows this stuff really well from a
crowds perspective use to work there did all
kinds of research for B his. He's not
trusted Sec.
It it's not Ai generated. As near as
I can tell. No. But something in that
Qa process obviously went a aw. For this
file ended up with Null in it. Right?
These things happen, I mean, I recently just
upgraded my whole home infrastructure. I got the,
you know, dream machine pro. And when I
plugged it in, had did you an update?
The update corrupted the whole os ass. I
had to, you know, ref flash the entire
thing. It's not unheard of, but I just
think it was some kind of secured, you
know, safety check before it went light. Yeah.
And we have a lot of people saying
well, someone f up the code. I really
don't think the code was left up. I
think it's the build process. So the Ci
pipeline somewhere
between point a to point z. Like I
said, this doesn't look like hands on keyboard,
Oops c's, I used gets at. Get ass.
It's like... So way, like, more basic than
that. Like it... It's definitely an Oops c's.
Right? But it's not poor code development,
yes no No 1 forgot comma sort to
speak. And is... So I got I got
hand it over to John. John has to
jump off. John, do you have anything else
to ask add because by the way I
wanna say thank you so much for giving
30 minutes of your time today. I know
that you're gonna be real busy. You've been
up all night, and we've appreciate you coming
on. Is there anything that you gotta say
before you have to jump off though. Oh,
goodness. Well, hey, I don't think so. Thank
you so much for letting me be here
with you all.
It's kinda interesting to think on and speculate
as to, okay, What were the root cause
or what are these
impacts and things as it's unfolding,
at the end of the day,
I think I would offer, you know, just
a gentle reminder, than the other side across
the screen. It's still another person. So big
hug ops to
crowds strike or fight fires, the best that
they can, and everyone that is going to
be doing this slowly in recovery and
efforts. You know what it's still it's still
all of us in the same fight. So
we'll be there to. Also, I wanna throw
a shout out to Hunters for not taking
advantage and, like trump to do marketing and
saying, look, Crowds sucks were,
because there are vendors that are doing that.
And by the way, trying don't don't party
with those people. Right? We just don't wanna
party with those 8 holes. So be sure
to check it out. Also, I'd like to
point out Hunter has a neighborhood walk program,
if you wanna give things a try and
play with it, and John is on Youtube
as all of you know because
you're probably all here because of him. So
Thank so much, Dude. Get out of here
and get back to your day and and
good luck and happy hunting. Thanks, all. See
you soon.
Does see where jeff black Jason's got some
questions. Go ahead. My question is, how do
I explain this
to regular people?
What happened?
Honestly, I would just say it was a
bad update that corrupted the operating system. Yeah.
Antivirus works at a very low level in
your in your cute, antivirus screwed up, like,
something happened with it, and it broke it.
That's that's how I would try to describe
it to friends and family. To be honest.
I don't know anybody else has a better
analogy. Know, I I think that's probably the
easiest way to explain it, but what what's
what's I'm interested in following now is,
you How exactly
Microsoft's gonna respond to the developer
community and to all of the X Ed
and defensive product community?
In terms of,
ideas for putting guard rails, more guard rails
around the tunnel without pissing them off. Right?
I think it's going to get to be
a very interesting
conversation because, you know, frankly in a in
a perfect world,
this shouldn't
shouldn't be allowed to happen. Right? And and
you know, we we'd rather not
see an entire operating system destabilize
by a single
Kernel driver, I also
worry greatly in our community and have for
a long time actually
that the kernel driver supply chain,
is an extremely vulnerable
and highly trusted
part of well everyday It operation that's very
certainly because of the compromise that happened all
the way back at Windows Vista. Right? Yeah.
You either let third party vendors into the
Colonel or you would let them out. And
Seriously, I think that Microsoft as much as
they were being, you know, very, very, very
mono in trying to lock vendors out of
the colonel. There was some damn valid reasons.
Yeah. To keep them out of it as
well. Well, those of us who lived through
that period of time. We remember back in...
You know, Windows Xp days just how
unstable the kernel would get
when you put lay it on some of
these products. And the reason was they were
hooking colonel tables, and sometimes they messed it
up.
Know,
and it's... You know. I I wasn't in
It or anything like that years ago, and
I remember how alt my computer crashed all
the time. It, like the blue screen all
the time, and then at some point it's
stopping being blue screen. Probably around SP2.
And Yeah. So if you're looking at Windows
xp, if we're looking at what protections were
put in place, it used to be pre
Sp 2 on Xp that any application could
access any region of memory
anywhere else in memory. So you had a
lot of things like video game trainers that
could actually hook right into another process memory
and start changing it that could mess with
the kernel.
Sp 2, they started putting in protections to
try to shut that down. When Vista came
out, Microsoft shutting it down completely and like
what Jo talked about is that compromise. Right?
So you probably did notice things getting more
stable,
and more secure. But now kind of, like,
an open question, I'd love to get people
a disc... Their opinion on this too.
Do you think that this is gonna push
Microsoft to shut this down even further.
I scored I have my hot take is
that this is definitely gonna start a conversation.
You know, Microsoft. I I mean, Microsoft... I'm
not... Microsoft does not like me, so I'm
gonna try to be as blood as I
can be, but have their what we do
here. Right? So...
But they have their own product to in
the space and Ed, and it doesn't really,
like, what Josh describes because they built the
Os, they have their own ability to kind
of built their own in kind of methods
inside there to get the same level telemetry
for the kernel without having to hook it.
So they are, you know,
ahead of kind of. They are, and they're
competing with a market space. Here.
I think this gives them enough fire to
kind of say, maybe we need to revamp
this pulse. See, and so pushed people out.
I mean, I wouldn't see that happen because
I wouldn't work eyes because all of a
sudden, their products gonna start looking a lot
more interesting and a lot more accurate.
Or do you think that they're gonna do
something like if you're crowds striker or another
vendor, you're gonna have to hook to Amc?
And it's like this is what you look
into? No. I... You know, I'd like to
see what I would like to see and
if I were at Microsoft right now, I
would be advocating for this, I would like
to see a kernel Api developed.
That had a series of driver integrity checks
built into it for every driver load, kind
of like
live dynamic qa as the system booted. Do
that. But that is definitely gonna be a
performance hit. Right?
No Do you want that? But but
Only on boot if they could pull it
off that the integrity checks are only on
there's a chance for the system to recover.
Correct. Oh,
okay.
But that's that's what I... You know, I
think is kind of a responsible middle ground
here. To maintain the telemetry that the vendors
need, but also introduce additional safeguards in the
driver load process
so that potentially these things would not happen.
Right? That's interesting. Yes, Jason.
Your muted
your mute. But I'm glad you raised your
hand. Why does stream work? Youtube work,
discord work and all these other things work
right now. Linux
Linux.
Yeah. Linux.
Well, or any uni may stop operating. Yeah.
Just be glad that this doesn't bring down
all the Linux... Server. And Very different. Yeah.
Let's let's answer that question a little bit
different. Right? If we're looking at Crowds strike
Foul, Falcon server The vast majority like overwhelming
majority of Falcon instances are endpoints like people
sitting down working at their computers. When you're
looking at a lot of server infrastructure,
that's actually running the Internet. Right? A lot
of that is running on Linux.
And if it is running Windows, a lot
of that server infrastructure, which we don't see
as mock for, like, these types of services,
it isn't gonna be running Crowds strike. In
those, like, those critical services that are out
there. So
that's generally why we're not seeing things go
down at that level. Now. What's going on
with the Faa and everything else, my guess
is
that the core infrastructure for banking, the core
infrastructure for airlines, the core infrastructure for a
lot of things is just fine
but the systems that the people use to
run those apps, those critical services every day
are down.
Sort of the people can't monitor if the
people can't maintain if the people can't poke
into those things,
then you've got stop airplane. Then you've got
to stop financial transactions, then you've gotta stop
these things from progressing.
So, yes, there are absolutely servers that are
windows servers that are running crowds that is
in fact, the thing. But in our experience
in testing, it is exceedingly rare for a
very high like, like high fidelity,
high availability,
high bandwidth service to be running on a
windows system with A falcon on it. And
by the way, Microsoft, if you listen to
my idea, you can write me a large,
check. I will... I will definitely fully accept
that. Oh, somebody just said, so should I
reboot my computer 15 times? I'm seeing that?
Seeing people that have gotten up to 20
and it hasn't worked.
So it you can do it if you're
real desperate.
The other thing that I wanna talk about
I wanna take a pivot, from what's going
on here,
from a technical under the hood perspective, and
let's let's take a step out. You're now
an enterprise, you're an analyst, your a cis
Okay. This is great. How it actually impacted
the colonel, where the virus hooked into the
colonel? That's all fine. That's all good. How
do I recover?
And right now there is...
I'm I'm gonna tell you
right out the gate that recovery
is bleak.
So right now, the recovery officially from Crowds
is you have to physically log in to
a workstation as administrator, which I'm gonna come
back to here in just a couple seconds.
You have to boot the system into safe
mode.
Then you have to delete a file seconds
and reboot it, and it will come back.
That
is like 4 or 5 sentences, that sounds
easy. Here's the instructions. Boot into safe mode
navigate to the drivers Crowds strike directory,
locate the Sis files, delete it,
reboot it, and then you're back.
So you're seeing on Reddit and you're seeing
on some people in Twitter, where they're talking
about it in terms of they have a
hundred thousand notes.
And best case scenario this takes 5 minutes
per node
to do.
That is a ridiculous amount of time.
To be able to go through and reboot
all of those systems manually. Jason, you had
a question, Does this include people that work
from home? Yes. But include people that work
from home.
So worst case scenario for desktop support teams
is they're sending instructions to people. On how
to move their computer into safe boot or
just out new laptop. But but if you
can't
use your computer to get that information? Then
how are you getting the information to fix
the computer? There there we go circular firing
squad. Yeah. And by the way,
Jason, it gets worse
because a lot of organizations have been told
to put hard drive encryption on their computer
system.
And using like Bitlocker or something like that.
You can't even boot it. And these kinds
of massive problems it. Know, was a totally
different problem back in the day, but I
remember when the the seed files were stolen
for our pay. That's right. And, you know,
I worked for a large company at... And
we had a hundred and 50000
employees that need your new Rsa tokens. So...
Remember, Rsa didn't replace those tokens for a
bunch of their customers. They didn't renew because
you're were out. Shipyard,
and it was Dod. Yeah.
So that deal. That sucks. Yeah. So
Ryan, can you share the thing I just
shot you about bypassing bit defender?
Or sorry, Bit bitlocker? So bit bitlocker,
this is opposed by what is it S
Cali,
that you can cycle through B to you
get a recovery screen. Navigate to troubleshoot advanced
option start setting, restarts, skip the bitlocker first
and second attempt. Navigate back to troubleshoot advanced
options command prompt, Bc at its set default
minimal safe boot.
This, I have seen pop up in 4
or 5 different places
I've seen it on Reddit. I've seen it
on Twitter. I've even seen it on our
on our on our discord server.
And I am seeing people to get around
bit bitlocker. Sorry, referred to it as a
bit to better. I'm seeing people that have
tried to bypass
bitlocker
that this works.
This is Anecdotal, you can try to do
it as a desperate measures. If you... Especially
if you have a critical system
that has, you know, app absolutely like the
recovery keys you need to get access to
this computer system. But this thing terri me
because this isn't even 5 minutes to do.
It's gonna take up half an hour to
45 minutes to do. You know, kia escrow
for the win. Right? Yeah. He's Mh I
wanted to bring in Patterson and Derek a
little bit more on this.
So we're saying you gotta sit down and
you gotta log in into to all your
Windows computer systems guys.
What if you what if you're... I mean,
if you go into safe mode. Right? You
don't need to have an administrator password for
safe mode on some systems or can you
kind of walk through what is the logic
of that? What if you're using lapse?
For administrator, like, password
randomization in your environment? It's not an issue
of just an admin sitting down? Like, What
are some recommended
kind of approaches for trying to handle this?
If you're seriously a helpdesk desk desktop
administrator. You're staring down this. What are some
pitfalls? What are some things that you've gotta
you gotta start figuring out right now?
Oh, Lord. I've been so far removed from
break fix stuff. Like, III honestly don't know
the answer to the question of does safe
mode, you, we're still require an administrator path.
I would say no based on
experience, but I mean, if it were me,
and I was an enterprise, and I was
dealing with because this is an incident Right?
This is, you know, basically a done all
the service.
I would say I'd probably be sending out
new
new equipment. And having it cross ship. I
know it's expensive, but I just don't know
how you're gonna get remote workers to successfully.
You'd spend more money on the phone with
folks I think, trying to figure it out.
I'd just... I... And then, you know, I've
seen people saying in chat, I've been trying
to, you know, read the fire hose of,
for the the folks that are like as
Crowds strike gonna be financially responsible. I'm sure
in their end user license agreement that there's...
We don't take responsibility for outage kind of
language. Right? You I I am equally sure
that there's going to be judges. They're gonna
throw that shit right out the window though.
But I mean, as far as, like, retain
customers.
Yeah. III, you know what and and so
I wanna continue to talk about recovery. We'll
talk about the out
Especially to crowds right here in a second
gaining I just... So ditch Oh, go ahead.
No. Just those manual instructions are really un
10
for for...
Mean, any anyone who has remote workforce stuff.
Yeah yeah. We've got we've got people freaking
out there, like, excuse me what? Do... So
you completely can bypass bit bitlocker using these
techniques.
Apparently, someone was sitting on a 0 day
for bit bitlocker.
Today's is their data shine. Maybe. I don't
know. I would like to get some more
confirmation as to whether or not this actually
works.
So some other solutions kind of kicking kicking
this out there. If you if you don't,
if you're not using hard drive
encryption, You can always have a linux system
on an Iso
that run some scripts to automatically mount the
hard drive and delete these files. Yeah. That
is 1.
It's just a Usb drive. It once again,
if you're dealing with remote workers, are you're
gonna ship them Usb drives, and it's gonna
get there on Monday or Tuesday and then
they it's gonna get scary.
The other 1 that I've seen Banter about
is pixie boot. But we Derek and I
were talking about this this morning. Been so
you gotta be on the same land. Right?
I mean, rather be on the same. Yeah.
I just... I I just... It's not gonna
work for remote workers as
Top Absolutely.
Scale is difficult.
Yeah. Yeah. And it just... Yeah.
It dip... That be the beeping in, You're
not gonna be able to pick boo. Right.
Internet. I think I think the biggest biggest
challenge here is is the bit bitlocker challenge.
Right? Because, you know, if you if you
have to go recovery keys or you go
to Safe mate. I mean, I know the
hacks out there, but, you know, who knows
if it's gonna work. But like this. Somebody
said it doesn't bypass bitlocker. It just boots
normally with a trusted platform module to unlock
the drive. You still need to have the
admin password. Yeah. Yeah. Yep. Yeah. But still
kinda kinda does.
I
so Can can I... Linux girl last, what
is pixie boot. So Pixie boot is where
Instead of your computer
starting its operating system from the hard drive,
it goes on the network through a broadcast
protocol.
And it downloads an image of an operating
system to load from the network rather than
the hard drive typically over T.
Yeah. T, it's very... Common. And and basically,
what happens there is do you you put
a option in your Dhcp server, which Yeah.
Which points at common the network server that
supplies the operating system image. Yes. Daniel just
have we need to pixie Boot on content
delivery networks. I'm like, there's nothing
Yes and by thought about that, that makes
me feel warm and fuzzy. By the way,
the acronym is is PXE
even though if we all say Pixie. But...
Yeah. Yeah. It Yeah. Sounds cooler. Right, Off.
Does sound cool. I feel like it does
sound cooler.
I feel like Patterson quietly in the background
finding a fixed for this and then the
very end he's like, alright. So here's how...
Here it is. He's the ninja that's quiet
in the back of the fight. He's jumping
in. What do you got, Patterson? I I
I'm for Oh, I'm I'm I'm border on
captain obvious, but I can't help myself in
that, as an incident responder, You, this is
the crap we've been warning you about. And
and but I I just wanna say, please
forgive me, you know, never let a good
crisis go to waste. All of the things
that we're wrestling with out of band comm,
prioritization of business critical resources, backups, resilience,
This may be the wrong time, but that's
just top of mind for me to to
walk out the other side of this as
an incredible learning opportunity,
to redefine and or,
define ways that you're going to confront these
kinds of issues because this is, and, this
is denial of service at scale
And again, it is... It's like, the the
world's greatest tabletop exercise.
Yeah. Like
the Ara code. Right? Like,
Oh, my gosh. It's like everyone gets to
play saudi of Ara. Yeah. I ordered it
out this morning. On the bright side, lots
of companies get a live ransom wear recovery
test this morning.
Yeah. Because you... You're worried about ransom, this
is the same. Like, these, like, just like
Patterson said, these are the same steps. If
you can do this well.
You're doing awesome. Keep going, but I'm guessing
a lot of organizations can't do this well.
And unfortunately,
there's no amount of money.
That you can pay somebody that they will
give you, like you pay them at Bitcoin
and they give you a recovery key that
recovers your ops. Like, there is no really
cool reset button with the Ransom ware group
where they're, like, give us a million dollars
and you get everything back. That doesn't exist
today. Yeah. Watch out for the fishing tactics
they're gonna go flowing
really.
Pitching tactics. A little bit. Before John left,
he showed a bunch of domains
that were registered, like, within the last 24
hours. And here some of them, crowds strike
blue screen dot com. Cloud crowds strike 0
day dot com. Crowds strike BS0D
dot com. Crowds strike doom day dot com.
Crowds strike fix dot com. Crowds strike down
dot site. Crowds strike token
dot com. So
it's brace yourselves. Like you think this is
bad like the attackers are gonna take advantage
of this? This is coming
right now.
So, Derek, do you wanna... You were talking
about that with me on the phone? Do
you wanna talk a little bit more about
this? Like, the I mean, we just saw
this. Oh, gosh. I just read about this
yesterday on another,
can't I remember the the the software package.
We actually had a sock customer that bell
for something similar where, you know, you Google
the problem. And it's, like, here's the solution
of the problem. So you couple either, like,
fishing or social or a search engine optimization,
and you get the the fix or the
update, and well, it's not the fix or
the update. Right at all it's backdoor.
And, yeah. So that's what I worry about.
Those are just some of the Io c's
that are known now, and I imagine there'll
be more to come. So from a defense
perspective, 1 of the things you might wanna
do with your Dns security provider is basically
looking at the age of domains slick.
Like, what is the age agent's domain? I
mean, you should be... And this goes back
to what Patterson said.
This is all shit. You should have been
doing too.
Right? Like,
isn't like, oh, god. No. We never saw
this coming. Take advantage of
This is that really hurts me is like
the past people that are listening to this
webcast are lighting up discord and I cannot
keep up with what's going on on meeting
right now.
They're not the ones that worry me the
most. The ones that worried me the most,
so the ones that logged into Cnn dr
report, whatever.
And they're like, good god. What happens? What
is this? And they're starting from ground 0
right now. Yeah. That is what really scares
me.
So
I believe we're gonna have a new back
and bridges and inject card. Do.
Yeah. I but I think we 1 the
card is gonna be.
We did a we did a tabletop yesterday,
and and we actually pulled the Ed, you
know, endpoint security, and then they rolled the
failure and everyone on in the audience was
inc. Are always works. I'm like, no. Really.
Back to him to get you like me
see Sometimes things don't go as planned. Like
Yeah.
I what timing is that? So somebody actually...
I just call something going by, so updates
are bad. Right? I mean, these are the
kinds of things got that make
you know,
organizations think, well, now we're not gonna update
because look what happened. And
right answer either. So so the new the
new... Back to us some breaches cards. I
think think it needs to be named Colonel
Driver of F buy. That's what we need.
So at the at the the first crowds
deployment ever did. A long time ago, we
had issues with,
basically denial service through over saturation of internet
connectivity based on updates. And then you could
group your updates thereafter. You could actually manage
distribution of
Crowds strike updates to endpoints based on grouping.
And and you not do that anymore, and
you exert some manual control to say, testing
know. Somebody posted a minute ago, and III
just caught it, and I'm not sure if
it's true that even if you were trying
to stay behind, like that this
basically, you got this update either way. And
that's 1 thing I never really agreed with
in the... Certainly for personal users when Windows
did it, but know, I I never really
agreed with forcing updates on
organizations. And I know there's ways around it
for sure, but, you know I think the
automatic update thing It needs to be an
option. It shouldn't shouldn't on. Okay. So I'm
gonna throw this out there. I'm gonna disagree.
This is bad. No question. But God it
if takes us back 10 years with auto
That's political it. That's what's gonna happen. We
don't... We we can't have that. Right? No
disagree. No, but with follow up. But we've
used to be I used to teach way
back in mid 2000.
It was like, you should always test and
validate your updates for, you know, a couple
of weeks before you roll it out to
production, and it's like, we ain't got time
for that I I stopped. Saying that and
more of, said in it and it does
apply to this too. Just get get good
at fixing it when it breaks. Right?
Immediately deploy to, like, your It team that
would be,
you know, responsible for fixing it Right? We're
get to auto out q
You do go ahead 1 crowds and hunters,
and then at least only 1 third of
your environment already the space Yeah. Yeah. Well,
that's an idea. Let in depth, not really.
So I you know, have joke about that,
dude? We joke about that, but I guarantee...
And...
Is that a bad idea today. To basically
say that we have multiple offices. We're gonna
use different Ed r's to make sure that
we don't go down a hundred percent.
We used to recommend doing a bifurcation of
endpoint security on workstations versus servers, partially for
that reason. Yeah. Not a, again, not a
terrible idea. Sure. And the article I was,
trying to remember during my class, but I
remember Dan Gear and Bruce Schneider, the mono
culture,
paper, where they were talking about the dangers
of mono culture. Like. We're still there. Right?
That is absolutely a problem
that exists. But
I I just don't know where the hell
this goes. I don't think it goes anywhere
good. Like, I think that you're gonna have
Ceos are gonna be, like, rip and replace
crowds strike now. Oh, yeah. And Crowds strike,
just to be honest, Matt, I wanna get
your take, off I wanna get your take
They aren't a garbage Ed. Right? Like they're
not bottom feeder. Right? So we might be
seeing 1 of the better Ed r's in
the market.
Get completely blown out of the water because
of 1 stupid Ci
mistake.
And I'm not dude Crowds strike doesn't pay
me. Right? That I get no money from
them. In fact, we have a love hate
relationship with Crowds strike. We absolutely do love
to hate us. They love to hate us.
They keep shutting down our upping licenses little
researching.
At the end of the day, I mean,
I'm gonna throw this out there, everybody. God
damn. They're a pain in the ass.
That's a good thing.
Come Right? So you want it to be
a pain in our ass. And, like, if
they handle it in a away. Like if
I was a ceo or they handle it
in a way where you know, they take
ownership and and and and try and make
it as right as possible. I don't know
that I would say. Alright. Because this might
happen to any product. You Yes. That's the
key. If you think you're gonna move off
to another vendor and they're never going to
have this. I've got this.
This great poster that I saw today where
they went through and they listed out all
of the other vendors that have had Ed
vendors. That have had the same type of
issue. They just aren't as big. Ryan, could
we kick this up? Yeah, with it Quick.
Yeah. Yeah. It's just... It's just not. Ed
vendors as well. Right? It's not.
I think that this could happen to any
enterprise,
class software,
But. It's and any product at all that
involves itself with the tunnel, which is a
lot. You know, And I've seen thousands of
networks since I've been at. At black hills
like both large and small. And if you
think the really large shops aren't held together
with, like, duct tape and pops sticks behind
the same wrong, Right? They are are. And
so I could just imagine this happened because
a filed didn't transfer right. Nobody paid a
etch. Right? Yeah. So I... So the tweet
talks about blue screen of death in
what is this cortex.
Another 1 in K.
Another 1 in s semantic endpoint protection. Another
1 in Sop net filter. So,
you know,
God damn at Crowds strike. You're putting me
into a situation where I have to defend
you. I did not
want to be at this point today. I
did not wanna be at this sprint today.
So
no. I I don't... Was 1 I wanted
to jump into. I know going it back
a bit.
A lot of Ed vendors
have the ability with their updates to actually
do staggered updates where you can do grouping
of updates or, you know, push live to
a big group or you can actually set
it to that way, depending on
new settings, you know, you can have a
small subs section, get the latest bleeding edge
updates and then your court brew. So like,
when to say this whole thing about, like,
people are gonna stop doing updates. I think
the only thing people that benefit from that,
are threat actors when you do that, and
I wanna stress that. There are some settings
and controls like free prod that can deploy
this. I think well, this is gonna definitely
have a bigger conversation, how is your update
policy set up? We... For a lot of
time in security focusing on, you know,
How you're rolling out these updates, How they
deployed how they're being tuned. We focus a
lot about that, but no 1 really focuses
it on update tests. And I think this
is a great opportunity to look at these
products. Because I know a lot of Ed
across the space have the ability to do,
like,
too staggered behind updates from the bleeding edge
or the current 1. Yeah.
And I think this... This is a great
opportunity that if you somehow we're able to
set your thing not always push the live
up update blindly and have it go to
a small control test. You might be 1
of the few companies that are not having
this issue today. Yep.
Yep.
Now the big question that I'm getting from
my family is, like, when do we buy
on the dip?
III
honestly this is...
III hope I'm wrong, but
I think this is, like Crowds strikes going
go down. It's not gonna go out of
business completely, but it's
it's going to go down. Like, there is...
Ceos having very
blunt conversations with their tech team, and they
do not give a shit about any of
the things that we just said,
and they're like, we need blood. We need
a head on the platter and
crowds strikes going to be the sac artificial
lamb.
I I think that that's going to happen.
You think the dip already happened possibly. Right?
It might be. I don't know. But III
would not people... I would not look at
this as like, 2 weeks, everything's going to
be okay. It might be. It might be,
but this is this is bad.
Especially if it gets to the point where
flights are down for, like, the next, like
week or if it led to people dying.
Like, there's hospitals that went down, operating rooms
that went down. Dialysis
machines that went down,
at least that's what we're seeing. Right? A
lot of that shit's not confirmed,
But, you know, standard outage, companies can recover
from. If you actually led to loss of
life in you're a hospital, and you got
a story of another hospital that went down
because of this product.
Right wrong or in different small mistake, anybody
can make that mistake. If you choose to
keep that software in your environment and something
like this happens again, The liability is not
going to be on the vendor. The liability
is going to be on the hospital
that decided to keep that vendor that had
something like this happening in the past. And
once again, I don't think that's fair to
Crowds strike. I don't think it is, but
there's a lot of organizations and lawyers that
are looking at it through that type of
lens. And they have to do something.
And I... And that's that's my big fear
about what we're seeing this right now. Yeah.
I, I have a lot of empathy for
the people are gonna work the crowds strike
booth that black hat.
Oh, yeah. Yeah. It's gonna be Page even
just close it down. Yeah. It might be
entirely possible that Crowds strike this isn't worth
it. Because think about all the memes and
the people on Twitter, they're gonna be going
by and taking a picture of them flipping
off the crowds strike booth. Right? That's gonna
be the hot thing to do at Black
hat. I'm gonna plug Crowds strike. If you're
thinking of going to Block hat, don't booth
Well, that's nice it. We can not... We
can... As a community, we're not always the
nicest foe. No No. No. No. No. No.
Can can I Can I suggest that you
prepare your speech now
when somebody says, you deployed Ed
on my biomedical systems and risk the lives
of patients that
not to minimize that concern, but I'm here
to tell you for a certainty if Ransom
was detonated
on your biomedical systems, it would directly impact
your patients? So be prepared to have a
logical
conversation about the cost benefit
of Ed versus
exploitation versus threat actor acts this and right
cannot overs stress that... And a logical conversation,
not an emotional conversation about there is a
risk this they're running Ed,
and it's this. There's a risk to not
running Ed and it's it's it's this.
Well, and I'm also gonna kind of, like,
r on that a little bit. If you're
a crowds strike right now, you need to
within the next 12 hours.
Be saying, what are you doing moving forward
that you were not doing a week ago.
Yeah. You cannot say that an intern screwed
up. You cannot say that an admin screwed
up. You cannot say that somebody oops seed.
You've gotta say what are we doing that's
fundamentally different moving from this moment on, that
we were not doing last week. And I
also think being very open about what happened
and saying, oh, there was a problem in
our Ci pipeline
Not good enough. Like, with something like this,
you're gonna have to say here's specifically where
the boo boo is. And I would recommend
looking at Man,
and whenever they were breached and of course,
that led into the whole solar winds thing.
He is open as you possibly can. Like,
if you're trying to minimize this by doing
obscurity, if you're gonna try to minimize this
with this community by trying to double speak
corporate speak and going around it, you're f.
Not only are you f you deserve to
be aft. If you get in front of
this and you get very detailed technical
explanations of where the program broke down the
Ci pipeline,
The validation process broke down and what you're
doing to rectify that, you may survive this.
I don't have a lot of hope for
Crowds to do that.
I I just seriously don't. But I think
when this community, like, we react better to
Sunshine, which is weird because we're It geek
and Sunshine makes us burn in ways and
impressive that are uncomfortable.
But the more information that you give us
on what's going on on this, The more
we're going to trust you moving forward and
your only marketable thing that you have at
Crowds strike right now is trust. And if
you start burning that with corporate double speak,
you're done.
Yeah. IIII
wanna
not to not to be shameless plugging, but
I'm gonna do a slight plug. Back in
February.
I released,
blog,
February 20 second, called,
initial access operations.
Part 1, and it talks about the Windows
endpoint to affairs technology landscape. So aside from
the things that I've said today and that
Matt Ar said today on this,
news cast.
You know, it it behaves us to to
do a little research and read up on
some of these things to to to to
you know, get get educated about some of
the things that are going on in the
systems that we're managing every day. So I
just wanna to put that out there. Go
read it,
and I think you'll learn a few things
and it helps reinforce some of the stuff
that I said today.
Alright.
Alright. Everybody. Let's wrap it up We've gone
a little bit over. Thank you so much
to those of you that joined. Thanks to
John for joining. Thanks for
Ryan for being here when you stick with
Covid, it's not cool that he's here,
speaking of processes that need to be fixed.
We need people that can back him up,
but he's so good. No. He's he's asleep.
Don't tell him. He's getting a big head
is what you're saying. He's got Covid. He's
not gonna remember any of this anyway.
But but seriously, I appreciate you guys jumping
on and really the community jumping on. I
think we had, like,
3000 people come on, which is huge. We
really appreciate all of you in the community.
You blew up my desk discord in ways
I didn't know was possible. I couldn't keep
up with you.
But we appreciate all of you and good
luck and, god bless to every single 1
of you. It's gonna be a rough weekend.
Take care.