Ready to party.
By party, do you mean nap?
Uh, yeah, we could have a nap party.
I’m not even mad about that.
That sounds delightful.
[snoring]
. Hello, alleged human, and welcome to the Chaos Lever podcast.
My name is Ned, and I’m definitely not a robot.
I am a real human person who does not make random
updates to your computer and then crash entire airports.
That would be wild, and I am definitely not
behind that update through my Skynet uplink.
With me is Chris, who was also here.
Let’s talk about not Skylink.
[crosstalk] . Whatever [laugh]
. Wow.
Yeah, I don’t know if you go into this or not, but you know, they are already
conspiracy theories out there that this was an actual attack of some kind.
Or a test for an actual attack.
Yeah, I have seen that on the Reddit
threads, [background noise] and other places.
Ohh, that was loud.
That was way louder than I thought it was going to be.
Sorry [laugh]
. Who likes fizzy water [laugh] ? Yeah, on the Reddit forums, and in
other places, I’ve seen the conspiracy theories starting to propagate.
I feel like those start out as jokes a lot of the time.
Like, “Wouldn’t it be funny if this was like a state actor,”
or something, but like, when you actually drill down into
it at all, you realize that this is just incompetence.
Don’t attribute to malfeasance what is more likely just gross incompetence.
There’s a pithier way of saying that.
Well, maybe now we should just talk about what we’re actually
talking about because we’re already talking about it.
Oh, CrowdStruck.
UnderStrike?
Dunn.
[laugh]
. Those of us who have been around the tech industry for a while,
and have peeked behind the mysterious curtain to see what actually
supports this endeavor that we call modern information technology—
It’s terrifying.
It’s three monkeys in a trench coat.
Barely.
I feel like you quickly become aware of how fragile this entire construction
is, and just how many redundancies and safeguards have to be put in place
to prevent the entire edifice from crumbling into the proverbial sea.
Yeah, and just to put a pin on that, in terms of, not
only is the technology fragile, so are the people.
I saw a joke on LinkedIn today about power-washing the back
of your servers to let the packets go faster, and I guarantee
there’s somebody out there going, “I haven’t done that.
I should do that.”
[laugh] . If nothing else, it’ll clean the air filters, so that’s probably good.
It’ll make everything a lot quieter.
[laugh] . I suppose it will.
Oh, silence is golden.
The packets go faster in silence.
To quote the second-greatest sci-fi movie of all time, Men
in Black, “There’s always an Arquillian Battle Cruiser,
or a Korilian Death Ray, or an intergalactic plague that
is about to wipe out all life on this miserable planet.
The only way these people can get on with their
happy lives is that they do not know about it!”
I love that quote.
Yeah, so just kind of apply that to technology instead
of aliens, and it’s pretty much the same thing.
The CrowdStrike debacle may not have been a Korilian death
ray, but for 8.5 million Windows devices, it basically was.
Everything, everywhere, is breaking, all at once, and it is
only through the heroic efforts of thousands of ops people
diligently doing their jobs that the public is unaware.
Of course, the public does occasionally become very aware,
and then senators have to hold hearings to grandstand
about things they do not even slightly understand.
They’ll hold some CEO’s feet to the fire for an hour, make self-serving
proclamations and possibly even attempt to levy a fine or two.
Good luck with that now that Chevron Deference is dead.
But hey, we’re not a Supreme Court podcast.
Go listen to 5-4 for that.
Solid plug for 5-4.
Definitely [crosstalk] time.
After all, the hubbub dies down, honestly, one or two C-level executives
will probably fall on their swords to appease the investor public.
I wouldn’t feel too sorry for them.
It is a metaphorical sword after all, and it comes with a guaranteed
payout of several millions of dollars, and a cushy job as a lobbyist
or CEO of some other poor unsuspecting private equity firm-acquired
disaster where they can oversee another unavoidable catastrophe.
It’s the circle of life, Chris.
I’m not going to sing it.
I don’t want to get sued again.
I—no, you’re seeing it in your head though, and I can see it [laugh]
. [laugh]
. Oh, so rather than talking about CrowdStrike for the next 30
minutes, I think we should all just go watch The Lion King—
The original.
Which is the best sci-fi movie of all time [laugh]
. I’m not really sure where to go with that [laugh]
. [laugh] . I don’t either.
I’m curious to hear the comments that we get in.
I did recently watch Dune: Part Two, which was excellent.
Took you long enough.
Listen.
Some of us responsible citizens saw in the theater.
I have one word for you.
That word is children.
Anyway.
You don’t think they would like it?
I think nightmare fuel would probably be the closest, yeah,
Feyd-Rautha—Routha—however the hell you say his name—yeah,
those scenes in particular, God, that dude is creepy.
Yeah, he really inhabited the creepy level of the character.
Like, Jared Leto levels of creepy.
No, but except good.
Yes [laugh] . Yeah, because he’s playing a character, not himself.
Oh.
Anyway.
CrowdStrike.
What the hell happened?
On Friday, July 19th, 2024, at 5:24 UTC—that’s 1 a.m.
for our East Coast peeps, and the day before for California
because you’re a bunch of weirdos—security vendor CrowdStrike
released an update for their Falcon sensor platform.
Falcon is an endpoint detection and response solution meant to protect
systems against viruses, malware, and advanced persistent threats.
The update type was a content update, or what CrowdStrike calls a channel file,
which you can think of is, like, the virus definition, except as a modern EDR,
it’s a bit more complicated than that, and we’ll get to why that’s important.
When we get to the root-cause analysis, or what we know so far.
Once the channel file was loaded by the Falcon sensor
platform, it caused a memory access fault at the kernel
level that forced a system crash on all Windows clients.
The old Blue Screen of Death popped up, and then the
system either rebooted or sat at that screen for a while.
Possibly forever.
Yeah, until somebody touched it.
Pretty much.
So, if you happened to walk into a major airport around that time,
you might have been greeted by giant display signs that just had
the sad frowny face on it, because now the blue screen has an emoji.
And it was kind of funny, actually.
I mean, funny for the people, you know, seeing the screens; not funny
for everybody who had to deal with the disaster, [unintelligible]
Right.
And were sitting in airports for three days while waiting to, you know, go home.
Yeah.
Depending on which airline you were working with, you
may have been not impacted at all, impacted slightly, or
still sitting in the airport listening to this right now.
I’m so sorry.
Maybe don’t fly Delta [laugh] next time.
Actually, I don’t know if it was Delta.
It might have been United.
They’re all terrible.
It doesn’t matter.
But one of the few that wasn’t affected was Southwest.
Is that because they’re running Linux?
Allegedly.
Again, this is unproven internet theory, but allegedly it’s because
their systems were so old that CrowdStrike wouldn’t run on them.
[laugh] . I feel like we did cover Southwest in a Chaos Lever, or possibly its
precursor, when we talked about old, out-of-date systems that are super fragile.
Am I remembering correctly?
I mean, I had that theory, or that thought as well, but I’m also now like,
did they just post that, and it became a memory, or is it a real memory?
[laugh] . It’s hard to say.
I will say that it was in fact Delta—and is Delta—that’s having
the biggest struggle because they use BitLocker extensively.
Right.
I assume you’re going to get into that.
Oh, yes.
Okay.
I don’t want to interrupt.
Carry on.
So, we had all these crashes—
Whenever you’re ready.
And you know, when your system—
Just go with [crosstalk]
— [unintelligible]
— —whenever [unintelligible]
— [unintelligible]
— At anytime—
[unintelligible]
— When you could—why would—who—
[laugh] . We have all these crashed systems,
and what do you do with the crash system?
You restart it.
But unfortunately, attempts to restart the afflicted
systems just resulted in another blue screen of death.
Because Falcon sensor is loaded as a driver during system
boot, and it has been marked as boot required, meaning
it must be loaded for the system to boot properly.
As soon as Falcon started, it would load all of its channel
files and, predictably, the system would crash again.
This rendered all effective systems completely
unusable and inaccessible through in-band management.
So, you can RDP into this thing and fix it.
So, this makes sense from an EDR perspective, right?
Yes.
You want to protect your computer.
No matter what tool you have, it’s going to have this boot requirement
because you don’t want your system booting without endpoint protection.
Right.
Because endpoint protection, ostensibly, is good.
Ostensibly.
The problem, obviously, comes in where your endpoint management
is now, effectively, malware that’s crashing your system.
Right.
That would be what we call ‘the downside.’
[laugh] . Yes.
And we will definitely get into that as well.
Microsoft has published a blog post where they claim, according to their
telemetry, about 8.5 million Windows devices were impacted by this.
Now, that’s only about one or 2% of all Windows devices out
there, so this is not, as a percentage, a ton of devices.
However… it’s still a lot of devices, [laugh] and the impact was pretty severe.
As we discussed, airlines had to suspend or cancel
flights, retail stores suddenly couldn’t accept payment.
Medical devices and hospitals crashed in the middle of
surgeries, bowling alleys had to hand out paper and pencils to
individuals, who just looked at them like, what the hell is this?
How do I track ten frames by hand?
How does a turkey even work?
[sigh] . Dark times for all of us, Chris.
That’s the kind of math podcast that needs to come out because I guarantee
there’s no one left on earth who knows how to score bowling by hand.
[laugh] . True story.
I was up in Cape Cod, and we went duckpin bowling—which is a real thing.
Look it up—
Oh, it’s so fun.
It’s super fun.
Definitely look it up.
Super fun, but the bowling alley was so old that
they did not have a computerized scoring system.
Wow.
Yeah.
They gave me a piece of paper and pencil, and
I was like, “Uh, score is not important, right?
We’re just here to have fun.” Oh… now to get these systems back to a working
state, the offending channel files had to be removed before Falcon was loaded.
There’s a few options to do this, and none of them are great or easy.
You can boot the system into Windows safe mode, which only loads the
absolute bare minimum of Windows drivers, and then remove the files.
For virtual systems, you could mount the system disk on another system and
remove the files, and then reattach the drive to the original system, or
if you had snapshots or a backup, you could roll back to a prior snapshot.
Fortunately, CrowdStrike did pull the offending file from the update servers,
so you wouldn’t then immediately redownload it and be back where you were.
While it is a huge pain to fix all of these virtual systems, the real pain
is those physical systems that don’t have an out-of-band management option.
Someone will need to physically sit at the terminal, invoke
safe mode, and perform the remediation steps, or use a separate
boot device like a thumb drive to perform the maintenance.
This is very bad.
You forgot about the other way to fix the system, which apparently
did work on some—at least a number of people’s, which is just
keep rebooting it until CrowdStrike Falcon updated, and deleted
the file on its own before it crashed because of the file.
[laugh] . I guess if it does load and the network stack loads
in time for it to pull the update and replace it, maybe?
Maybe.
Between 15 and 20 reboots.
Sometimes people were getting it to work.
Wow.
That’s awful.
But okay.
So, another option.
Microsoft has published a USB tool to assist with the
removal of this file, so you have that option as well.
As I mentioned, the BitLocker thing does throw a bit of a wrench
in the whole plan because in order to access a BitLocker-protected
system drive out-of-band, you have to supply a BitLocker unlock key—
Yeah.
And that can be hard to get.
Well, it’s not like people want their end-users to have that.
Again—
Yes.
—this is a security concern.
Also, the BitLocker key is 48 characters long, so not only finding it
but typing it in before BitLocker times out… which it does, apparently.
So, a bit of a nightmare.
Not a great situation.
No.
And so, that’s part of the reason Delta is still struggling.
I would love to say that, as of right now, we know exactly what caused the
error, but honestly portions of the supply chain are still pretty murky.
Instead, I will try to explain how a simple update for an EDR caused millions
of Windows machines to blue screen, and we can also have fun pointing all the
fingers that we have at all the other parties because we’ve got jazz hands.
[whispering] It’s all your fault.
That works better with a—
Visual medium?
Yeah [laugh] . So, to start with, we have to consider
what the Falcon [Center’s] actually trying to do.
Falcon Center, as I mentioned, is an EDR product, and it’s meant to
scan all activity on the host operating system looking for threats.
Most applications aren’t granted that level of access
to other applications or to the system as a whole.
As you mentioned, Chris, it needs to be in a privileged position.
But that’s the point: you’re trying to prevent other pieces of software from
getting themselves into privileged positions to compromise your computer.
To understand what it means to be in that privileged position,
I’m going to briefly talk about user space and kernel space.
Please feel free to interrupt me when I get something wrong, which I will.
[whispering] Yes, thank you.
Your operating system, whether it’s Windows,
Linux, macOS, I don’t know, Solaris—
AIX?
—sure—it is responsible for managing the hardware on your system.
That includes stuff like memory management, writing data to disk,
sensing input from peripherals, and scheduling threads on the CPU.
This all happens in what is called kernel
space, and it’s considered highly privileged.
If something goes wrong in kernel space, the system may have to halt
or crash to prevent damage to the hardware, or corruption of data.
Ideally, as little as possible should be running in kernel space.
Instead, most applications run in user space
which does not have direct access to the hardware.
Applications running in user space interact with the operating system,
and make requests based on that operating system’s published APIs.
Do you want to write a file to disk?
You make an API call and pass the correct information.
Need to access memory?
Make an API call and specify the address and range.
The operating system will evaluate that request,
make sure it’s valid and allowed before executing it.
This means when an application runs into issues or it crashes, the
operating system is able to handle that crash gracefully—most of
the time—and keep other processes and the system as a whole running.
Right.
And there’s one point to note here.
So, first of all, some of the terminology, they call it the
kernel; also, they call it Ring 0, meaning it is the lowest
possible level of the system, and it has access to everything
else that is going on in the system without restriction.
Necessary to make sure, for things like EDR tools, that it can
scan not only all of the files, but all of the activity, all of the
network, all of the I/O, the disk, et cetera, et cetera, et cetera.
Right.
One thing people always get upset about is, why does Windows crash so easily?
And—
[laugh]
. While there is an argument to be made that it is fragile and poorly
designed and should have a better way of handling things like EDR
that needs this access—which is true, and I assume you’ll get to that—
Yes.
The other thing is, again, remember, completely unfettered access.
If something goes wrong at the kernel level, we
get our old friend, unanticipated consequences.
And this is extremely bad.
So, for example, let’s say you have a system that is running a database.
Databases, as you know, are kind of important.
A kernel-level job is trying to write a new file, or
a new table, or a new row, or record, or whatever, but
it runs into an error with, say, memory misallocation.
What is it going to write to the database?
It could be writing absolute nonsense.
It could completely corrupt the database.
Therefore, the kernel crashes preemptively whenever it detects a
failure because the consequences of trying to soldier on might be worse.
Right.
It’s that, “Out of an abundance of caution, I’m going to fail.”
Right.
Which is the same thing I did in high school.
[laugh] . Yes… it was better if you didn’t succeed, Chris.
[laugh]
. So, Windows applications have been able to request
access to run in kernel mode for a long time.
Generally, that’s a bad idea, for the reasons you just articulated.
But Microsoft wasn’t super strict about it.
Microsoft is nothing if not accommodating
to developers and their terrible ideas.
Some applications actually do need to run in
kernel mode, in particular, antivirus software.
Applications running in user mode are not generally allowed to
access the memory and monitor the behavior of other applications.
Microsoft Teams can’t just decide to read the memory space of
Slack or kill the Zoom processes, as much as it might want to.
I was going to say, it totally would.
[laugh] . The operating system just doesn’t allow that type of nonsense.
But, you know, an antivirus application needs a privileged
level of access and monitoring to defeat the bad guys.
So, antivirus companies like Symantec wrote
their application to run in kernel space.
Now, Microsoft actually tried to push back on the rampant abuse of kernel
mode by antivirus outfits—and others—when Windows Vista was being rolled out.
Yeah.
What, what, what?
Vista, for you youngsters in the crowd,
Vista was the Windows 8 of the early aughts.
Hopefully that puts some perspective on things.
While Vista was a disaster as an operating system release, they did add a whole
bunch of additional functionality and features that brought the client OSes more
in line with what the server OSes were doing, and added a bunch of security.
And one of the things they really tried to do was lock down kernel mode access.
Unfortunately, antivirus companies didn’t like that, and they threw a hissy
fit, claiming that since Windows Defender could run in kernel mode, and their
stuff couldn’t, Microsoft was abusing their influence, a la Internet Explorer.
And Microsoft, still reeling from their decade-long battle
with the FTC over antitrust, kowtowed to the AV club, and
allowed them to keep their precious kernel mode access.
It’s not an unreasonable request because all the
other players wanted was an even playing field.
Right.
The fact that even playing field was a wide-opened
security nightmare is still a Microsoft problem.
[laugh] . Right.
Microsoft did add an interesting requirement, though, if you
wanted to play in kernel space, and that was driver signing.
Antivirus applications would present themselves
as device drivers to get to run in kernel mode.
A device driver to nothing, but a device driver nonetheless.
Microsoft created the Windows Hardware Quality Labs Testing Certification—aka
WHQL—and once a driver had gone through that lab and gotten its certification,
Microsoft would digitally sign the driver and give them the Certified for
Windows logo, so they could proudly display ‘Certified for Windows Vista’—or
Windows 8 or whatever—on the box when you buy the software, or on their website.
Now, vendors could still choose to sign their drivers
internally, but the antivirus folks wanted to get that
WHQL certification and all the cachet that went with it.
As long as your driver code didn’t change,
the digital signature would remain valid.
So, that means all these antivirus companies—like CrowdStrike—would
get that certification, which meant that it had gone through
some level of rigorous testing when it came to the way the
driver was written and the way it interacted with the kernel.
Seems like a good idea.
I’m for it.
Unfortunately, external data could be loaded by the driver, like—
No.
—virus definitions.
But in theory, the actual running code
should all live in that signed device driver.
So, read in some config, but all the logic in the
actual code should live in that device driver.
That’s all well and good for loading virus signatures and
looking for matches in memory and CPU threads, but Falcon
sensor is a modern EDR, and it doesn’t just use signatures.
Instead, Falcon uses machine learning to develop behavior
patterns, and then it needs to detect and respond to
emerging threats that match those behavior patterns.
The channel updates Falcon sensor receives to model that behavior, those updates
appear to include some amount of pseudocode that is executed by the driver.
And it is that injected code from the channel—or lack
thereof, actually—that seems to have caused the issue.
According to people who have looked at the channel
file in question, it is entirely filled with zeros
[laugh] . Now, you would hope that the driver would look
at a file full of zeros and just ignore it, like, “Nope.
That’s invalid.” Falcon sensor chose a slightly different route and crashed.
Right.
So, what we have here is a driver that is legitimate
and was tested and proven resilient, which is good.
Yeah.
And we have updates that come down the wire multiple times
a day and interact directly with that driver, that were not.
Precisely.
And it would appear that of the many tests that were run against
that driver, none of the tests were, “Here’s a file full of zeros.
What do you do?” Because no one thought that was a thing that would ever occur.
But it did.
There is a popular breakdown of the Falcon sensor crash dump by Twitter person
Perpetualmaniac, which I won’t be linking because after assessing that it was
a lack of null pointer checking in the dump, he then went on to make weird
disparaging comments about the Rust community and blamed the whole thing on DEI.
It got strange and kind of fasci, so fuck that guy.
Fair.
Instead, I’ll include a link to a different Twitter thread, by
someone who actually debugs stuff like this for a living, and he
basically said that Perpetualmaniac was wrong and thinks that it is
uninitialized data being read from a table that caused the crash.
Now, considering that the input file was entirely filled
with nothing, uninitialized sounds like an understatement.
Unfortunately, we won’t know for sure unless CrowdStrike shares
their source code for their driver, which seems unlikely.
Maybe they should, but I don’t think they will.
The point is that the channel update caused Falcon sensor to attempt to access a
memory location that didn’t exist or wasn’t initialized, and the driver crashed,
forcing the system to halt in order to prevent possible data corruption.
So, that’s where we’re at.
Now, it’s time to point fingers.
Cool.
[It’s] Everybody’s favorite part.
Predictably, in a fuckup of this magnitude, the blame
game and armchair quarterbacking is in full effect.
Thought leaders are tripping over themselves on
LinkedIn to have an opinion about the whole mess.
And I’ve seen posts ranging from ‘this is all CrowdStrike fault.
How did this update ever get out the door?’ ‘This is all Microsoft’s fault.
How could they let third parties run in kernel mode?’ ‘This is the
customers’ fault for not having phased rollouts.’ Et cetera, et cetera.
And then there’s all the conspiracy theories about how this was a state actor,
or a planned thing, or I don’t know CrowdStrike did it on purpose, for reasons?
Anyway.
Solar flares?
Oh, I like that one.
That’s what made it all zeros.
There’s plenty of blame to go around, and none of it is
actually helpful while the fire is burning, but now that
we’re over a week out, maybe we can take a more nuanced look.
Or not.
So, how did this update actually leave CrowdStrike’s front door?
That’s a great question.
The truth is, we will not know until CrowdStrike tells us or
a lawsuit forces legal discovery, and we find out that way.
The former could come any day.
I’ve checked their [unintelligible] blog posts several times as I was
writing this piece, and so far, they haven’t said, but maybe they will.
Uh, actually, so they did—
Ooh.
—at about three o’clock this morning.
[laugh] . Of course they did.
They released an official—well, an official
unofficial preliminary post-incident review.
Okay.
It’s a good name.
And basically what they’re saying is, it went through automated
testing, but the automated content validator had a bug in it.
So, they passed it—quote-unquote, “Passed, but it was an invalid file.
Ah.
“Once the file went out, it was immediately picked up, read by Falcon
sensor, and it caused an out-of-bounds memory read, triggering an exception.
This unexpected exception could not be gracefully handled,
resulting in a Windows operating system crash BSOD.” Unquote.
So, it seems like their testing harness or whatever they’re using
also doesn’t know what to do with the file that’s all zeros.
Well, yeah.
There’s a lot of problems here.
First of all, clearly they did not test the tester enough.
Yeah.
Because if you have a bug in a testing system
in an automated deployment, that is a problem.
That is a huge problem.
And the fact that simply loading the file caused the blue screen pretty
quickly makes it sound like they don’t actually push these updates to
test machines that then run the update to see if the system crashes.
They’re using some other testing process.
Right, which they do not go into any detail about, unsurprisingly.
So, I am sure that the lawsuits are forthcoming, and maybe we’ll
find out more when legal discovery happens, if it gets that far, but
the truth is, CrowdStrike pushes these channel updates frequently.
Like you said, Chris, they push these more than once a day.
And they have automated testing in place, but they’re trying to stay
one step ahead of the bad guys, which means time is of the essence.
This specific update was meant to address
something, a new vulnerability found in named pipes.
They wanted to get that update out before any
attacker figured out how to abuse this vulnerability.
So, maybe what they’re doing is sacrificing
quality or testing in favor of speed.
This is a systematic failure, and it’s not the fault of one person.
Yes, maybe someone screwed up and accidentally saved the file
empty, but something else in the chain should have caught that.
Right.
If a single person can unwittingly push an update that
takes down eight-and-a-half million Windows clients,
that’s an organizational and systematic problem.
There’s also some indications that this isn’t
the first time such a transgression has occurred.
It appears that Red Hat Enterprise Linux, Debian, and
Rocky Linux have all encountered similar crashing problems
earlier this year after a channel update was pushed.
I think it was April and May were the two months where the issues were found.
The issue with Debian in particular was traced to a specific
version of the kernel that wasn’t included in CrowdStrike’s
testing matrix, but was on their list of supported kernel versions.
macOS seems to have weathered the storm, for reasons that we will get to.
Yeah, and I mean, that’s an important point.
And, you know, a lot of times people will say, this only
happens to Windows, and that’s absolutely not the case.
Anytime something runs unfettered in Ring 0 of any operating
system of any kind, you run the risk of causing an immediate crash.
It’s just not usually so public-facing because you don’t tend to
have Linux running your displays that’s also running CrowdStrike.
Right.
For whatever reason, we like Windows for that.
I don’t know.
We’ll get to that, too [laugh] . So, what about Microsoft?
Shouldn’t they prevent this kind of thing from happening?
In an ideal world, they could.
And we’ll get into the technical solutions in a
moment, but this is largely not Microsoft’s fault.
Yes, Windows has its flaws—many, many, many flaws—and Microsoft
hasn’t always produced the stablest or most secure software.
No one could call them blameless with a straight face.
But in this specific instance, the system is working
as designed, even if the design kind of sucks.
Should we be shaming all these organizations who let the update
barrel through their environment like salmonella on a cruise ship?
That’s an image for you.
Think about the counterexample for a second, let’s say that a zero-day
attack was discovered using this named pipes thing, and it was
leveraged by a hacking group to infect a major airline with ransomware,
and later it came out that CrowdStrike would have protected them
if they had been running the newest version of the channel updates.
Stupid CISO decided to stay at n minus one for updates.
Do you think the defense of not running the latest channel updates as
a resiliency strategy would appease litigators and the public at large?
I’m going to go with unlikely.
[laugh] . So, I mean, another point that’s important to note
here is that the kind of patch that came out—or the channel
update—would not have been stopped by an n minus one effect.
N minus one would stop the driver update.
Remember, that’s the part that was signed by Microsoft, and is noted as good.
The actual kernel—or the actual channel update itself
happens automatically, and you can’t do anything about it.
There is… I was reading through some Reddit posts, and some
people did say that there is a way to run a little behind
the channel updates, so to postpone them by certain periods.
There is a way to run, kind of like, n minus one for the
channel updates, but there’s an inherent risk in doing that.
Yeah, like you said, it’s certainly not the
sort of thing that a CISO is going to encourage.
Right.
And there’s also a regulatory hurdle with that, too, because there may be
compliance and regulations that say you have to be running the latest version.
So really, it’s just a rational decision based on balancing priorities and
political realities, and trying to protect your customers as best you can.
So, the blame ultimately should reside on
CrowdStrike for putting out a floud update—floud?
Flawed.
Floud.
Words.
I love them.
I like floud, actually.
It’s like loud, but with an F.
It’s floud.
It’s like a flan that has opinions.
An opinionated flan.
I like it.
Let’s talk about solutions [laugh] . The reason macOS hasn’t encountered
a similar fate as the Linux and Apple installations is that Apple
doesn’t let CrowdStrike—or really anything else—running kernel mode.
Starting in macOS 10.15—I didn’t look at the codename,
so please forgive me—Apple offered System Extensions.
These allow an application to stay in user mode while
requesting special access to hardware managed by the kernel.
At the same time, Apple phased out Kernel Extensions—often
shortened to kext, or [pronounced] kext, I guess—
Yeah, it’s pronounced, unfortunately.
[sigh] . They phased those out starting in macOS 11.
So basically, CrowdStrike doesn’t run in kernel mode on
macOS, and thusly, it cannot crash macOS the same way.
I don’t know no Mac, so I don’t know about any of that [laugh]
. No, it’s true.
And for a while, it was extremely annoying because a lot of programs
relied on kexts for similar reasons: to have instant access.
Like, a good example is if you have an external audio
device and you want that to work as fast—as efficiently as
possible, you would want it to work and run in kernel mode.
Right.
So, there are actually ways to get around the
security that you just talked about in macOS.
I don’t recommend it, but it is doable.
And the whole point here is that you have this little secret
enclave, effectively, where things run in this sort of in-between
mode—sandbox, if you will—which we’re going to go into in a second.
But if it crashes there, it doesn’t take down the operating system.
Right.
And Linux actually has a similar option with eBPF,
which I struggle to say because it’s awkward.
And apparently, it’s no longer an acronym.
It’s just its own thing.
So… that’s weird.
eBPF lets applications load into a sandboxed
secure kernel execution environment.
So, once again, gives them kernel-level access to resources, while applying
stringent safety checks to make sure the application doesn’t crash the system.
CrowdStrike now offers running Falcon in user mode on Linux—what
they call user mode—which actually uses eBPF under the covers.
If you were running in that mode, those previous crashes
that happened with Red Hat, and Debian, and—what was
it?—Rocky Linux, you would not have been affected by those.
I mean, CrowdStrike would—Falcon still would have
crashed, but it wouldn’t have crashed your system.
Which is better.
I think so.
Windows has some similar functionality available.
There’s the Windows Filtering Platform, Windows Defender
Application Control, and Windows Defender Device Guard, all of
which have APIs, but none of them have the same mechanisms present
that, like, System Extensions for macOS or eBPF for Linux have.
So, they provide an API that applications could be rewritten to take
advantage of and get, you know, almost kernel levels of access and speed,
but they’re not the same as this, sort of, sandbox, secured enclave.
There is a project to port eBPF over to Windows, for what it’s worth.
I don’t know if that will be the ultimate solution, but this catastrophic
calamity should at least prompt Microsoft to try something similar.
I have heard some folks—and we could call this a technical
solution—I’ve heard some folks say that you just shouldn’t
be running Windows in most of these environments.
Like… you’re not wrong.
If I could wave a magic wand and turn back time, if I could find a way, Chris—
Stop it.
—I would take back all the Windows that hurt
you and replace them with Linux variants.
Okay, it doesn’t rhyme.
[laugh] . It’s the best I could do.
If you’re out there, and you’re building a net-new system,
that’s, like, an end-user terminal, an IoT device, or even a
server running in the cloud, I think anything but Windows is your
best bet, and it would probably be malpractice to do otherwise.
But like it or not, Windows remains the most popular desktop
operating system, and that doesn’t appear to be changing anytime soon.
We need a short-term plan to make things better—through some sort
of update—and a long-term plan to ditch Windows for most use cases.
Thoughts?
So, a lot of in—I’ll put, ‘in my opinion’ around all of this—
Right.
A lot of this comes down to the never-ending battle between speed and
security, and making assumptions that things are just going to work.
After all, like we said, they’ve done multiple channel updates
a day for years and years and years and years and years, and
while they’ve had a few issues in the past, it’s not very many.
This is the sort of thing that leads developers—and, you know,
engineering teams—to have a false sense of security, and a false sense
that everything they do is golden, and they will never have a problem.
Therefore, checks get skipped, checks get removed from the process—because
after all, they’re just slowing us down—and that’s a huge issue.
The other issue is, when you push everything out all at once, the problem
can occur—like it did this time—that everything will crash all at once.
There needs to be some type of a fuzzed deployment.
So, let’s just say these things get released
on a schedule, I don’t know, every four hours.
You get a customer that’s got a hundred servers.
Those servers should get that update five minutes apart in, like, groups of 30.
That way, if there is a catastrophic failure, it
only takes down a percentage of your platform.
Now, that’ll happen for every single customer on Earth, and
that’s not great, but the assumption is, and should be, that
there is high availability built into this, so if half your
systems go down, theoretically, the other half can carry the load.
Mmm.
Yeah, and that’s something that CrowdStrike could change today.
That’s within their realm of control.
Yeah, and I suspect that they will [laugh] . Because the only
other option—if this is the situation—is people are going to end
up with half of their environment running one antivirus solution,
and the other half of their environment running another one.
That seems worse.
It’s just as insane as running—insane and difficult
to manage as running in a multi-cloud environment.
Or a super cloud, as some might say.
Ugh.
I hate you.
Yeah, these are all technical solutions.
I don’t know if there’s any policy solutions, but my biggest
concern coming out of all of this is that regulators and
litigators are going to get into hubbub and pass some poorly
thought-out legislation that makes things effectively worse.
I can’t quite figure out how they would make
things worse, but I am excited to see them try.
[laugh] . They’re nothing if not creative.
Well, hey, thanks for listening or something.
I guess you found it worthwhile enough if you made it all
the way to the end, so congratulations to you, friend.
You accomplished something today.
Now, you can go sit on the couch, update your CrowdStrike channel
file, and watch everything crash in beautiful synchronicity.
You’ve earned it.
You can find more about this show by visiting our LinkedIn page,
just search ‘Chaos Lever,’ or go to our website, chaoslever.com.
You’ll find show notes, blog posts, and general tomfoolery.
And if we got something wrong, or you have strong opinions
about what CrowdStrike should have done, leave us a comment.
Leave us a voicemail.
We might even listen to it.
We’ll be back next week to see what fresh hell is upon us.
Ta-ta for now.
What a mess.
Mmm.
A glorious mess.