Subscribe
Share
Share
Embed
Threat Talks is your cybersecurity knowledge hub. Unpack the latest threats and explore industry trends with top experts as they break down the complexities of cyber threats.
We make complex cybersecurity topics accessible and engaging for everyone, from IT professionals to every day internet users by providing in-depth and first-hand experiences from leading cybersecurity professionals.
Join us for monthly deep dives into the dynamic world of cybersecurity, so you can stay informed, and stay secure!
Today, we will debunk one of the biggest myths in
cybersecurity, and that is that patching is a solution.
Welcome to Threat Talks.
My name is Lieuwe Jan Koning.
And here from headquarters at ON2IT we
bring you the next episode of Threat Talks.
And the subjects of today
is about the patching strategy:
what works, what doesn't work,
and what should you really do?
Let's get on to it.
Welcome to Threat Talks.
Let's delve deep into the dynamic world
of cybersecurity.
Let me introduce our guest of today.
He is a returning guest,
so we see him a lot.
His name is Rob Maas.
He's the Field CTO of ON2IT.
And his job is to make sure that everything works,
stays working, and doesn't get hacked.
Not just for us, but for all
our customers as well.
So, Rob, welcome.
Patch management comes up a lot.
And we got a lot of questions and
a lot of misconceptions always.
And so it's about time
we talk a little bit about that.
And to what extent is
patching a solution?
Because, sometimes we get the feeling
that, patching, if you don't do...
patching is like the Holy Grail.
The silver bullet.
And, as long as you patch,
then everything is going to be fine.
A lot of people tend to believe
that they should patch everything
and always, and immediately.
Which definitely is not true
and is not even possible.
Yeah. So let's start with
the first reality check.
And that is actually that the majority
of systems can't even be patched, can you
elaborate a bit on that? Yeah. So that
depends a bit on your organization.
But a lot of systems cannot be patched.
So hospitals, healthcare have a lot of systems
that have been built to last ten,
20 or even longer, in years.
So look at your MRI system,
for example, MRI scanners,
they often cannot be patched
or they only will be patched
for the first year or the first two,
three years, but then it stops.
So then you also have these legacy
systems that no vendor supports,
that cannot be patched
or only can be patched
and then you lose your complete
support if things break.
You have your of course,
OT systems, which,
well, maybe you can patch them
but [it’ll] have a big impact.
So like downtime of production lines
or recalibration of systems.
So, yeah, you end up with a lot of systems
that simply cannot be patched
or not patched easily.
Yeah.
And another thing is,
a patch needs to be available
first of all, so you can
actually roll out. Yeah.
So if you know of a bug, and there's
no way to patch it, then...
Yeah. Then it also stops. Yeah.
Or if the vulnerability is there
and we don't know; the zero days,
I mean that's even...
That's another one.
Yeah.
If we don't even know how
the hacker will get in, and they
know, because that's the problem.
Then yeah.
Yeah. Okay.
And the other group of, the other
reality check is about,
patches are scary.
Well, I would not say scary,
but they can break things.
Yeah. Therefore they’re scary.
They can be really scary.
And especially if you patch your
crucial systems for your business.
Yeah.
Suppose I'm the national railway system,
and there is a new patch for some
system that is in all the trains.
And I'm going to, am I going
to press the install button?
Hopefully not.
No. That's a good example.
I think, the best example
I can give was CrowdStrike.
That was a patch that was released
and automatically installed.
And of course, we all know
how that went down.
We did an episode about it,
but that had a big impact.
So I can imagine if you then,
as CrowdStrike would show,
hey, there's a patch available.
Would you like to install
that? People think twice.
So that's a good example.
We have lots of examples with Windows
updates that cause problems.
Recently, last August, we had
a Windows update, which updates
the NDI systems within Windows,
which is used for streaming services.
And a lot of applications could
not run after that update.
For example, OBS, which is an open
source video streaming solution.
So that's an example.
Also, we have seen updates that cause
antivirus to stop working.
So then you patch your system,
but as an end result,
you get less secure, because your
antivirus is not starting anymore.
What's your opinion on this?
Is this lack of testing or
is it simply so complex,
all the combinations of software
that can run on systems,
that it’s impossible to...
I think it's a combination,
because it is complex.
You have so many combinations, but that
also means that it is really hard to test.
So you can test scenarios you
know that are out there or that
the biggest market share have,
but in the end, you cannot test everything.
So, yeah, it's a combination.
You cannot do it all. Yeah.
It's always the CIA triad.
Confidentiality and integrity that.
for that you need to add
the patches to be installed,
but availability, probably not.
No. So, yeah.
If things run, then don't touch it.
That’s what people often say.
I would always say if things
run then you can
improve them, because you have a stable
starting point, but that’s not
how it works for availability.
Yeah. Yeah.
Well let's talk about how we can
do it then a bit, about the solution.
One thing maybe is,
if CrowdStrike back then,
we covered it in that episode.
Hey, why would you put the patch in,
in your whole ecosystem all at once?
And to be clear, that wasn't possible
before, in a CrowdStrike product.
They now have installed it.
And the idea is that first in 5% of
your machines, you do the update and then,
and then half a day later or a day later,
you do it on the rest.
Of course there's a little bit of security
that you lose then, but at least,
if something goes wrong, then
it's not like catastrophic. Just then,
it gives you the idea, within
24 hours, you know,
you have probably found someone
who knows which button to press
in the [ ], to keep the update
from spreading further.
Yeah. So that's one mitigation action,
I would say, if you have
a large patch of systems
that you need to update or maybe
multiple critical systems
if there is an opportunity to do it
in phases, I would always do that
because then you have always
a fallback if things break.
Maybe it's also better
also for your uptime.
So that's a good approach.
So, I think CrowdStrike is doing that now.
They learned it the hard way.
But you can also do it
in smaller environments.
So if you have multiple, well,
let's say Active Directories,
quite crucial for a lot of companies.
If you can do it,
don't patch them all at once.
But just one by one, so the rest keeps running.
So what you're saying is it's actually okay
to not patch everything that has an update.
It's safer, I get that, but it's also,
this would also be our best practice.
I would even argue
if it is safer. It depends.
So one of the most important things,
if you think about patching is
you need to know what you have and
how it's placed within your architecture.
So, maybe a good example
if you have edge devices.
So we recently saw a lot of
problems with the NetScaler.
They had a lot of vulnerabilities,
those you want to patch immediately.
They're on the outside.
So they have a really big attack vector.
That means a lot of people
can connect to it and try to,
yeah, abuse that vulnerability,
exploit the system.
So those systems you want to patch immediately,
there's no question. Needs to be routine, then.
You have a patch machine, like in a [ ].
Yeah. You need to act immediately.
We saw it, unfortunately, with a few
customers here in the Netherlands,
that even if you patch within a day,
you might be too late.
It was a zero day before,
but now it's open,
now even everyone is going to try.
The crucial thing is you specifically
single out certain systems.
So the edge devices have no other
protection than to rely on the patch.
Yeah.
Which is kind of,
we said itin the beginning,
I mean if you have to rely on patching,
you have already lost more or less.
But there's simply no
other way of doing it.
For the edge devices
there is not, but
let's go back for Active Directory
as an example.
Authentication system in the core
of your network and shouldn't
be exposed to the internet.
Probably isn't.
Hopefully it is not.
And even within your internal network,
you should only exposed the
protocols and, well, I would say
the application that you need.
So for example, LDAP is
the protocol that is being used
to see if a user can login or not.
That's exposed.
But if you, if there is a patch that
patches a problem, for example, in
the SMB protocol,
which is the file server protocol,
if it's not needed on your Active
Directory, and if you have not exposed it
to your internal network,
there is no rush to patch the system.
Yeah, but now are,
this is a requirement that you mention,
because I don't think that's the reality
in most organizations,
that the Active Directory
is actually singled out in a network.
So that means that all other servers,
when connecting to AD,
do go over a firewall
that forbids access on the SMB ports.
Right? Correct.
So you need to have a Zero Trust
architecture already in place.
Yeah, a Zero trust architecture
really helps here to lower the priority
and... It's better than patching.
It's really much better than patching
because, you know, the system will run.
It's more of a static setup.
So, you know, okay, these are
the protocols that need to work.
It's not only for AD, but you can
do this for every server.
The server has a purpose,
has a certain set of protocols
that are needed to communicate and to work.
Shut down, close off the rest.
Hopefully, not only on the firewall,
but ideally also on the system itself.
That takes away a lot of
need for immediate
patching, especially if it comes
to the security patches.
I remember that we had, I will not name
the vendor, but the solution that,
that was in the authentication realm
and it was RADIUS requests.
And that software only ran on
a very old version of Windows,
and it was simply not available
for the later versions.
And, yeah. There were no patches,
so and it was as buggy as it can be.
So, but the only thing
we actually needed
from a business perspective
is the RADIUS ports.
And so what we did is, we just,
it was already in a segment.
And, first it was easy, but we could
easily validate that although this,
machine had thousands of known vulnerabilities,
it was not in the RADIUS protocol part.
So therefore it was okay.
And then, but yeah, that's not, often people
say, yeah, but it's such a bugged system.
We should really get rid of it.
But it isn't really, not for
security purposes then.
I mean, it's always better to run
current versions of everything.
Yeah, I won't advocate that
you don't need to patch at all.
But the prioritization, you should,
I think, play a lot with it because,
there are so many patches
now, and most companies
have so many systems,
they simply cannot keep up.
You cannot patch everything, test
everything all the way, all the time.
And also, patches might break things.
We also already discussed about legacy
support, or a legacy without support.
Do you want to patch them? And what then,
if things go sideways?
There's a lot of guides, guidelines
that you mentioned,
but it's like a lot. You need to,
kind of be an expert in the field
to decide, okay, this patch we do
or this patch we don’t.
Do you have like a guideline
that you can follow like or like a recipe
that is easy to replicate, that always works?
Always, I know, I asked
a technical guy about
something that always works. I think it will work,
but it will require a lot of work.
And, that's the downside here. But
the first thing is you simply need to know
what you have and where it's located
in your network and how crucial it is.
And that sounds very simple, but unfortunately,
for a lot of people, that's still hard,
knowing what you actually have,
how important it is, where it's located.
Yeah. So if you have your e-commerce
company, you probably have a
what we would call protect surface,
that is the e-commerce website, right?
That's one thing. And probably your
Finances and your HR systems.
You need to make sure that
you know where they are.
Yeah. And then? Because then,
how do you get,
how do you know whether a patch is
available for... Yeah, the second thing here
is, you need to have the, I would say
this is also threat ntel information,
especially for security patches.
You need to make sure that you
get the emails from the vendors.
Or application maintainers,
keeping an eye out,
hey, this application can be vulnerable,
for example, this e-commerce website,
if there is a problem
within the application.
So within the e-commerce
application itself,
you probably want to patch
that immediately,
because you said, okay,
this is our website.
So it is publicly available
if there is a vulnerability there,
in the application, people maybe
are able to purchase stuff without paying.
Yeah.
So that I would then classify as-
You need to put effort
in aggregating all the information
from different sources.
Yeah. It’s quite similar to what we do
in the security operations center upstairs.
Of course. And that is indeed, newsletter
of vendors, it's also certain websites, certain
channels on Twitter, for example.
Those kind of things you mean, but
so every organization needs to have that
or hire someone to do that for them, then.
Correct.
And then I think you can
do that to a certain extent,
but you will-
I can tell you to a lot of data.
This a lot of data, and especially, so
the generic updates, I would say
like a NetScaler, Windows, etc.,
they are available for everyone.
So that's easy also for us
to provide a customer,
hey, this one is really crucial,
please update, but a lot of companies
also have internally developed
software or maybe niche
software we don't know of.
So then they, preferably
the application owner or the internal
IT team should really closely watch
if there is any update for those systems.
Yeah, but you can make
that infinitely more simple
by, like we said, segmenting
out the applications.
So you only need to do the protocols.
You still need to monitor it, but then
it's less crucial if you miss something.
Because if you can,
let's say Zero Trust, or micro segmentation,
whatever you want to call it.
But you place your systems
in certain segments,
which their own risk profile.
Then you can say, okay,
this niche application,
I only check every month for an update.
Only this port is open. This protocol.
So the chance that things go sideways or it’s
being abused or exploited is really, really low.
Okay. Hey.
And then, so let's assume
that we have this stream of
possible problems, possible
vulnerabilities that need to be patched.
We send it to a mailbox or ...
And the poor guy on the receiving
end of it, then gets all those emails.
Can we help him triage in a way?
Do you have some guidelines for that?
Yeah.
So, I would try to classify
the updates where needed.
So filter out what's not relevant to you.
You probably also get a lot of
updates of systems you don't run.
And once you have
the updates that apply to you,
you need to specify how important it is,
that’s the triage process.
So the AD systems, patch them immediately.
Then you probably have classified a couple of,
what we call protect surfaces,
but important systems within your
environment, as your crown jewels.
So you should patch them... well,
maybe not immediately, but quite quickly.
Especially if those vulnerabilities that
are being fixed are open to the rest
of the company. You can see this
in the CCSS score, right? Yeah.
That's a big help.
Yeah. CCSS score is a good help.
But also there is the Known Exploited Vulnera...
What was it, the KEV, I believe?
But there is a list. If a vulnerability is
out there, that you can see
if there is an exploit and if there
is a known exploit for it,
and if there is a known exploit
you really want to patch
because then the information,
how to exploit
that vulnerability is out there.
Yeah. Someone will try it.
Someone will try it.
And of course
there's still the difference
if it is externally exposed
or internally exposed
or maybe, within your internal network,
not exposed at all.
Yeah. Okay. Clear.
Any other practical advice that we could
give to our viewers on this subject?
I think the most important thing is,
forget about patching
everything immediately.
And be fine with it.
And be fine with it.
That could be a hurdle.
Like an emotional hurdle.
Yeah. Still...
Because you feel guilty, right?
If you don't patch everything
immediately, and then, y’know,
the next thing you know
is it's impossible.
Yeah, unfortunately, we see still
a lot of IT departments,
which have more or less the requirement
that they have always everything patched.
And that's simply not possible.
You always find systems that you simply
cannot patch or there’s no patch available or,
that you don't get support or there are
all kinds of reasons we already discussed.
So I'll leave that aside and focus
on where it's really, really important.
Make sure you have a good process in place
there and then, also have a list of
systems where it's not that
important at all, or where you only
take a look every quarter, for example.
Okay. Clear. Well, thank you very much.
It's nice to have some peace by,
if you can't cope with
patching everything, but there is a work
to do, for every organization.
More on a process level then.
Okay. Well, thank you very much.
And to our viewers, thank you
as well for tuning in today.
If you liked what you saw today, please like it,
because then we can spread it further.
And while you're in that area
where you can do the likes,
you can also press the subscribe button
and that make sure that next week
you will have our next episode
right in your inbox.
Thank you very much and see you next time.
Thank you for listening to Threat Talks,
a podcast by ON2T cybersecurity and AMS-IX.
Did you like what you heard?
Do you want to learn more? Follow Threat Talks
to stay up to date on the topic of cybersecurity.