Welcome to the VCD Roundtable Episode 47.
We want to talk about the importance of hardening.
So hardening software.
So we added the software at the end.
Oh, I didn't know that.
I thought...
The topic is around hardening, but OK.
If it goes around VMware products,
maybe I can help as well.
Hi, this is Matthias,
also a part of the VCD Roundtable stream again.
Today's guest speaker with us is Fabian.
Fabian, for the introduction.
Hello.
So I've been here many episodes ago.
I don't know, maybe one of the first and never came back.
But now if things get hard, I'm back in the team.
Fabian, architect at comdividion.
I do a lot around architecture and consulting service
providers when things get tough.
Nah, just kidding.
No, but the topic around hardening is something
I dropped in during our Kickoff
because we see that a lot of service providers
need to be prepared
against any kind of cyber threats
and attacks and also audits.
So be prepared for the audits.
That's why hardening is an important thing.
And maybe I can drop the one or other story
to give you guys out there some feedback around that.
Absolutely.
What's also interesting, based on the questions
we get from many customers and service providers,
is do we need hardening only if we need to get certified
or is it also relevant in normal environments?
Yeah, I mean, hardening is a general concept for many years
to change your software probably in a way
that differs from the default settings.
Yeah, there were many years ago
when... was telling you
how to harden your Windows systems
and all kinds of software can be
hardened by disabling features,
by also giving you guidance around operational steps.
You need to patch your environment.
You need to make sure people are trained.
All those things come together
when we talk about the topic hardening.
That's why I always see it as a three-part thing.
One is the architecture around solution.
That must be hardened.
One is the configuration, how is it applied?
Those hardening settings.
And also how is it lived?
So how are daily procedures hardened against security?
This is something you always need to observe
as a complete item from my opinion.
And if someone promised you,
"Hey, we deploy Aria Operations or VCF Operations"
and you have automatically your environment hardened
or get reports of the hardening status.
In my opinion, that's not true.
It's one important toolkit for it.
But there's much more to hardening
than just installing something
and saying, "Now it's hardened."
I wouldn't phrase it that simple
because honestly speaking using Operations Manager
and the implemented policy or the implemented framework
with the hardening configuration is pretty straightforward.
It just reflects Broadcom's default view:
which configuration should be taken
or which feature could be reconfigured
to have a hardened infrastructure
from a Broadcom perspective.
That is one of many approaches.
It might fit your needs.
It might not, but you get a proper
report based on Broadcom's default view,
how it should be done.
If you don't agree, if you have a different approach
on hardening your infrastructure
or you have different security
and recommendations/needs,
you just need to reconfigure the
framework within Operations Manager
to fit your needs.
Then you have a proper report based on your infrastructure
and you can start reconfiguring everything.
Yeah, totally right.
But from my point of view,
as you know Ops Manager has a limited view on things.
If I think about the environment of the service provider
who might also have NSX, Advanced Load Bancer
or I don't know Cloud Director in place,
that's typically something the
Operations Manager does not get.
If we talk about those, let's say hardening relevant items
like "Hey, your Active Directory must be secured."
Let's say, immutable against any kind of attacks.
That's also something we cannot.
So we have only this limited view
and I really love using those built-in compliance options
from Aria Operations as a starting point.
But still, and you did a great point by saying,
"Hey, it's everything related to risk
and everyone needs to decide fon their own
if they want to accept the risk of not applying this item
or maybe this item does not fit
for your specific use case environment."
And from my experience over the last years,
I did a lot of... I was involved in a lot of audits
around Enterprise solutions.
And the important thing is you
need to have something in place.
That's the first thing.
So you need to do this kind of hardening.
VMware has done a great job over the years
together with the community of
creating those hardening guides;
in the past they were called
security configuration guides.
Those big Excel sheets
where you have the default setting,
that's what it should be.
That's the argument why you should have that.
And what we always did, what we still do nowadays
is taking that extension for certain customers
and have the discussion -- item per item with the customer.
"Hey, how does it look like in your environment?
Is it a risk we take?
Shall we live with the operational downside
of enabling lockdown mode?"
Which obviously makes sense
from a security perspective in many scenarios,
but sometimes from operational perspective,
it can be quite annoying.
So we need to get the risks together,
calculate what's relevant for the customer
and then make a decision.
And I think that's a very important point you just covered,
because even though a guide tells a customer or a CSP
setting ABC needs to be reconfigured to whatever,
you can still come up with a decision
against the recommendations.
Say, "I'll configure a feature
in a certain way because of..."
And even if you have an audit...
so an auditor has no chance to make any...
He cannot force you to reconfigure it
to a certain configuration.
But if you have it configured differently
based on common guidelines,
you always need to have a justification in place.
You need to take a proper decision and justify
why is it configured that way?
And as long as you have a justification,
you're good to go.
Because if you have a justification,
you made the decision and you're perfectly aware
about the pros and cons.
Yeah, depends on the justification.
I don't encrypt because it's complicated.
[laughter]
Would be the...
I mean, from my point of view,
most auditors I work with, most of them,
they audit a broad area of IT.
They're not experts in VMware.
But they react differently
if they have a question and want to see something
or some documents, artifacts, whatever.
And you take something out of the box.
Here, that's a complete list.
It looks great with bullet
points and score and justification.
You show it to them.
You then take an extract from the Aria Operations report.
So they see there is a mechanism in place
to prove that certain configurations are in place.
This is something that really helps you
during the audit.
And for sure, for your own sake,
it's up to you to still make the best design decision
that secures you, but still keeps operations alive.
And this is really relevant from my point of view.
That's why, I mean, if you do those security engagements
or hardening engagements,
three to five days of architecture is easily gone.
You have one, two, three days of workshops,
then everyone goes back together and make maybe different
decision about those 10, 15 complex points.
You need to, let's say, also,
or try to get the proof for certain statements.
Yeah, maybe that the backup is encrypted.
Simply something you cannot see.
You need to assume that the backup is encrypted.
But still, when we go out there, I ask them,
hey, give me some proof or some configuration proof
that the backups in the back end are encrypted.
And this is really relevant.
So those engagements are really important,
but also take a little while.
And that's what I said.
You cannot simply deploy Aria Operations.
Say, now it's hardened and let it run.
You need to discuss that.
It's an architectural decision you need to take
around the environment.
And even if you have some configuration monitoring
in place, like Operations Manager, for example,
it's just a supporting tool,
because you need to make the architecture.
You take the decisions, you justify.
The monitoring is just a tool supporting you
to prove that you implemented what you have discussed
and designed.
That's one point.
But what we also often saw on the health checks,
for example, that there is hardening
or hardening was done, hardening was documented.
But then they had an issue and needed to open SSH
or direct access to hosts
again and never configured it back.
So from that perspective, I think, it is necessary
to have proper control of all mechanisms and changes made
during the hardening process to ensure they remain active.
And totally, totally true.
And one example that always comes to mind is,
for example, I'm a big fan of
segmenting the VMkernel ports
of the ESXi, onboard firewall.
We simply say only management networks can interact
with the ESXi VMkernel, which is easy to do
and gives you direct or reduces the attack vectors.
And even though there's some
malware in your infrastructure,
they cannot reach the ESXi network.
Except, I mean, you could, if
you have some within data center
firewall, you could do something similar.
But it's one way to make sure the packets don't arrive
from unknown networks at the ESXi.
And when I look at all... fortunately, over the last two
years, I've known many companies had
some issues with crypto attacks
on their VMware environment.
Fortunately, none of my direct customers were
attacked by that, or it was broken or encrypted.
But the primary way I got engaged afterwards,
it was always the same thing.
Some older version, non-patched version of ESXi,
they were reachable from one of the desktops
that had some malware installed.
So client network, server network,
server network, they could scan the network,
figure out ESXi host, figure out older versions,
and did some attacks.
And the other perspective was always
around this active directory integration of the ESXi host.
So also one thing, when the ESXi host was integrated
in Active Directory for use authentication,
there were a lot of attacks happening over this path.
And today, I think in the hardening guide for security,
there's still the recommendation
to add the ESXi host to the Active Directory,
to have a proper directory or non-route-based log on
as possible, even though that's the official statement.
And you also see in the CIS guidelines
for security or other documentation
when you want to reach a certification,
I always argume," hey, let's not do that right now."
There were so many bad things happening over this way.
We do log them out.
We segment the ESXi from the network,
and we make sure we have some alerting in place
when the root user is used on the ESXi host.
So you have Log insight configured, making alerts
if roots are doing any actions
out of our role-based authentication mechanism,
because that is really important
to make sure every action is accountable
to a specific user or system.
But that's... all the topics you've mentioned.
They're all part of a proper hardening,
because hardening is not a tool.
It's not a solution.
It's a process, right?
It's something--
It's a lifestyle.
It's a lifestyle, right?
That we're not going down there.
Yves is not here, Fabian is here, and we're drifting.
So it's a process you take.
It's an approach, and also monitoring,
and many different things that come to your mind
might be part of a hardening process.
Like you said, if a user is used to log into a system,
or you mentioned the AD thing with the ESXi host,
it's also pretty fine.
But I think even if a recommendation exists
and you come up with, "no, I
don't think that this is a good idea,"
like joining an ESXi host to the Active Directory,
you just justify it.
The downside of it is, and just being really blunt,
is there are many auditors out there without any clue--
No, I should not say without any clue.
That's not good.
Not being in depth with a certain product behavior.
We'll tell you in the report, your ESXi host
is not joined with Active Directory, so you get a minus
score for something which does not make any sense
because they have no clue.
And then if they have a good name,
if the company has a very well-known name,
they are always right, even though they are wrong.
Absolutely.
But the good thing is, those are the items,
even if everything's correct,
those are the things they can find.
But they feel good.
We feel good.
But still, the Active Directory thing,
and if anyone is here
listening, I would be really interested
how you are dealing that around the ESXi hosts,
because I have a clear opinion just based on what I saw.
Because I saw things, and I don't want to see them again.
I think if you're coming up with your approach
and how you would configure infrastructure,
I think as long as you have,
as long as you take the decision
document the decision, that's the proper justification,
you're good to go because that documents
that you're perfectly aware of what you're doing.
I think that's one very important point
around the whole hardening and those topics.
True.
Sasha, how can Cloud Foundation help us with hardening?
I want to hear some sales talk, come on.
Yeah.
It's the best solution.
Absolutely.
So you get a lot of configurations pre-configured by VCF,
so that's interesting.
But I think the one topic also for me,
a big part in hardening is to work with certificates.
Work with public certificates,
make sure that you have a trusted CA
behind your certificates and not working with
self-signed certificates out of the host.
And having a hardened certificate authority.
Now we start again.
Yeah, I mean, that's also a thing.
Absolutely.
It doesn't matter if your
certificate authority is compromised.
That's a huge thing.
I always get the feeling when you go down the security
in Hardening Road, it's opening the box of Pandora
because from one item, you get to the next one, ask,
hey, what kind of CA certificate authority do you have?
Is that something we can
assume as being reliable and secure?
We don't know to be honest. Especially if you talk
to smaller corporations where
that was set up by, I don't know,
Erwin and Dieter.
And they left the company 20 years ago.
You know, Erwin and Dieter, that's a common German name,
just for Mike and Jane, if you're from the US.
But still, this is something where you then also say,
okay, perfect.
When we are done with our, I would say, assessment
or hardening sessions, and everyone is really hard.
No, that's not the right thing.
And everyone is really getting items to work
in the VMware infrastructure.
There's also a lot of to-dos to figure out
if we assume the right things around Active Directory,
certificate authorities, and other topics.
Erwin and Dieter.
Come on, you said we can cut this later on.
So therefore--
Well, we'll let it slide.
So, cool.
But again, it's also the combination
with the monitoring tool.
Because you need to have a report,
because we are all just human beings.
And if we are tasked to configure something,
and even though it's very well prepared and documented,
and we receive a monkey, see monkey, do instruction set,
there is still the chance that
you're fat-fingered something.
And that's the reason you need to have a pool
in the backend, monitoring the infrastructure,
being aware what should be configured,
and compare it with what is configured.
And, Sascha, I love the example you came up with earlier on
with the SSH.
So that's a common use case, like,
"oh, I need to troubleshoot something."
I enable SSH.
What is perfectly fine, right?
And at the end, because you disabled all the warnings,
because they drive you bonkers in the UI,
you forgot to turn off the SSH service
at the end of your troubleshooting session.
So latest and next day,
the compliance report within the monitoring tool,
which is Operations,
you shall have only one Operations Management tool,
pops up saying, "I found this...
"design and deployed by comdivision"
Yeah.
... here is a new host, and this host has
SSH turned on," and then you receive an alert,
saying, all right, I detected a configuration drift.
Please help me turn off SSH.
And even though you have an incident,
so you violate one of your policies,
but if you have a monitoring tool, you get an alert,
and then you can react on, like, "oh, I forgot.
I turned the SSH servers off."
That's a proper approach, and that's perfectly fine.
The worst thing is you are not
aware of that a configuration
drift happened, and you are not
compliant with your old policies anymore.
That's the worst thing.
Yeah, totally something we need to keep in mind.
I mean, the good thing is over the years,
the core VMware products, and
if you now look into the security
configuration guide for vSphere 8, there's not so much more
in it anymore.
The product team did a great job over the years
to bring the core hypervisor to a
level where you should typically,
or you don't need to tweak that much.
I think Mike Foley, who was also in the product team,
he's very often shouting out on LinkedIn that he
was very keen after getting the hardening into the product.
We see that VMware is getting
better and better in a lot of areas.
That will also be much more better than with VCF 9.
I'm pretty sure about that because
now they all try to make it secure
right from the beginning.
But again, there are a lot of things we need to decide.
Second factor of indication, for example,
that's nothing you can bake into the product.
We need to make a proper decision, and
then also how to configure it concretely.
Even if that's not a real word, I think.
But also Cloud Director.
I mean, we are big old fans of good ol' Cloud Director,
and also there are still a lot of
things we need to tweak afterwards
to make sure it is hardened according to the recommendation
that VMware put it out also many years ago.
Maybe not Broadcom.
Maybe the recommendation which
comes from either comdivision
or even better from the customer themself.
Because that's one thing we do with our CSPs and customers
is interview them.
What's your expectation?
What are your business
requirements towards security hardening?
And those bits and pieces.
Another interesting topic.
Let's move away a bit from the base infrastructure.
And let's maybe for the last few minutes
focus a bit more on the workloads which are running
on the already successfully hardened base infrastructure.
So we have guest operating systems.
We have applications and a ton of stuff running.
And if we are now back into the CSP game, it's a CSP.
You need to make sure that you
have proper network implementation.
Because you have no clue what your
customers are doing inside the virtual machine.
They install the guest user and you just don't know
if the customer applying all the
security patches as recommended
by the operating system vendor for example.
Yeah.
And remember those site channel attacks a few years ago.
You know where they said, "Oh, disable hyper threading."
Because there were chances of
whatever this is something that could be.
Could be a real...
So winning the lottery had a higher chance than that one.
But...
I'm not sure.
I mean there were attack patterns
where you could extract certain things.
And this is something you
cannot control as a service provider.
The thing here.
What was the name of it?
Spectre and Meltdown.
Winning the lottery has a higher chance than that.
No, I prove it wrong in the show notes.
Do we have show notes?
We will get some.
So I can tell you on the side why.
But the thing is you have your...
And that's not a guest OS thing.
It was a hypervisor attack.
So two different things.
But you have no clue what your
customers are doing inside the virtual machine.
And if they're patching their OSs,
if they apply security patches to the applications.
And we are perfectly aware, because
virtualization is a big invitation to keep
expired software running year after year
after year out of support, out of everything,
not supported anymore.
So many companies do that.
So as a service provider, you never know
what's happening inside the infrastructure.
So it's even more important
from our perspective to take care
that the underlying infrastructure is properly secured.
And also you are logging all the
traffic between tenant networks
and the CSP basic network infrastructure.
Separate.
And additional to that and additional to hardening.
So really take a look what
options you have to help your customers,
your end customers, to bring more
security inside their environment.
So take a look at advanced threat protection with NSX,
which you can buy as an add-on.
So that's one point.
And the other point is take a look at advanced ALB.
So advanced load balancer.
So for all the web services your customers hosting.
So you can activate web application firewall on top of it
and make your environment or the
customers' environments more secure.
So ATP, so I first mentioned
ATP, that's a different ballgame.
It's an amazing feature.
It's there and even with DPUs it gets usable.
Because you're offloading everything to the smart name.
And use those features because I haven't.
So I've been in IT for more than 25 years.
So over time, year after year, you just see more different
different patterns.
It got more and more and more.
There was no year where you had less
threats running around the internet
than the year before.
So it gets worse and worse.
Okay.
And then in the end, you always need to have
Operations in the back end for monitoring.
Awesome.
So if you are not hardened already, what can we do?
Sascha, how much is our hardening?
Standard offering.
What do you say?
You need now to turn on the red light.
And then...
Wait a sec.
So if you need...
Not red enough, I would say.
So if you need it really hard.
No, wait a sec.
Guys, you are aware that it's a podcast.
Most people won't see the video.
Really?
We're now waiting for the audio description.
That's why we talked about the red light.
Yeah.
So for all the audience who's not watching the video,
the light in my room is now pretty red.
And joking aside, if you need any guidance around that,
if you feel your VMware
environment could use a third party look up
around the topic of security
hardening, just drop us a message
and we can make you pretty quick and offer to help you
nearly on all continents nowadays, isn't it?
Sure.
Yeah.
Feel free to ping us.
Feel free to drop us messages on
LinkedIn, as an email, over our webpage, etc.
and we can sit together and talk
with you about what we can do for you.
How can we solve your problems and provide hardening
or a design for hardening for your environment?
Yeah, also from my side, thanks for tuning in.
Famous last words, I think hardening is very important.
Also monitoring and implementing
the stuff, but always keep in mind
that you keep your risks and hardening
aligned with your business requirements
to have a solution because in the
end you need an infrastructure solution
which supports the goals the business has.
Otherwise it doesn't make the best sense out of it.
Short outlook Ep. 48.
I heard a rumor.
I heard a rumor that Episode 48 will
be around Orchestration Automation.
Yeah, I want to come back.
So that's the rumor.
So we won't discuss VCF
Automation because that's not released.
So Orchestration Automation will be
around Cloud Director and Orchestrator.
That's the plan.
Interesting.
I'm looking forward to this one.
Who will present, not me.
I'm sure you.
He's Mr. Orchestrator.
Come on.
I know.
Okay, thanks for tuning in.
Have a great day.
We're more than happy to help just ping us.
See you in the next episode.
See you.