As we were wrapping up TalosCon 2024
I was in the pub talking to one of my fans,
Rio Kierkels, a DevOps Specialist
from the Netherlands.
And at one point Rio said, you should
talk to Mischa van den Burg.
So here we are today.
Thank you, Rio.
And it's great talking to you, Mischa.
Welcome.
Thank you.
Thank you, Gerhard.
So we talked a little bit about this
before we started recording.
The more I was researching the content
that you produce and the blog post,
the more excited I was getting.
I knew this was going to
be an amazing conversation.
You've put so much good stuff out there,
not just in terms of content,
but in terms of ideas and you live them,
which I find very inspiring.
Like you are the embodiment of
everything that you stand for.
And it's amazing.
Thank you.
That's the biggest
compliment I've had in a long time.
Thank you.
Four years ago, you switched
from nurse to DevOps engineer.
That's right.
Why DevOps?
I had been living what you
can call it a colorful life.
I've tried out several things.
I moved to Norway.
I worked in the oil industry there.
I worked in management.
Then I lost my job in that industry and I
was very fed up with it.
So I wanted to try something else.
So I transitioned to nursing.
I wanted to work more with people and
like feeling that I
made a change in the world.
And I did.
Every day I was at the beds with the
patients and I could literally make their
world a little bit better by just being
there for them and
putting in a lot of effort.
But I'd also noticed that like my
personality, I'm rather introverted and
it was costing me a lot of energy.
And I wasn't sure if I was going to
be able to do this like
for the rest of my life.
Throughout this whole period, I was
always ever since I was a kid tinkering
with computers and scripting,
automating, especially video games.
I was very into online RPGs.
And at some point I figured out that you
could write scripts for those RPGs.
So I could mine my ores and earn
my in-game money while
I was doing my homework.
Also Diablo II: Resurrected bot, seriously,
like what I saw that like, hey, I think
that's what that was.
That was used to be my favorite game.
Still is by the way, I'm still playing that.
So yeah, like you like you and I never
built a bot by the way for Diablo II
Resurrected, but you did. Amazing.
Well, I didn't build it.
It was an open source project and I was
involved with it, but
by no means built it.
But I did run it and
made videos about it.
So I did do that.
That's when like my fascination with
computers and
automation especially started.
And that escalated to me having like
renting servers in Germany and having
hundreds of accounts playing the games for me.
And then selling that in-game currency
for a little bit of extra money.
I could by no means live off it, but it
was just so cool to know
that I had this army of bots.
playing for me as I was doing my work.
So in that process, I learned Python
coding, Linux. Everything was running
running on Linux, and
I was doing all of this in my free time
and I was always like, well
what if I could make this my job?
Like not automated games specifically,
but like working with infrastructure and
working with code, automation.
And I tried a few times, but it was
always like, well, you don't have a CS
degree and you don't have
experience in the field.
And this was when I was living in Norway.
So that was very tough to get into, but
at some point I was again reaching that
point in my nursing career where I felt
this is not going to be the thing either.
And then I just went, I asked myself the question:
What do I want to do?
And I answered that by saying:
what do I like to do in my free time?
And that was what came out.
And then to get back to your question,
I started talking to people
who I knew in the IT industry.
And one of them said, I
think DevOps is your thing.
I think that's what
you're describing to me.
You're what you like
about code and automation.
I think DevOps is the thing.
And that put me on the track on DevOps.
And as soon as I heard that word for the
first time, I was basically sold.
Okay.
So as you were discovering DevOps,
as you're getting more and more into it,
what are the fun things that you didn't
expect to begin with,
but you really enjoyed?
You've been doing this for four years
now, five years, four years?
I'm going into my fourth year now, yeah.
I really love to learn, and learn new
skills, new technologies.
And I hadn't fully appreciated when
I started out that this field,
or IT in general, is just such
a constantly changing field.
And you're basically required to tinker
in the evenings just to stay up to date.
It gives this constant intellectual
stimulation for me and something to keep
up with and to share about.
Which is something that I didn't
realize in the beginning.
I thought, okay, I have a
skill set I need to learn.
I learn it.
And then that's what I'm going to do.
But that's one thing that really
surprised me and that
I still enjoy a lot.
What about the hardest part?
I did have this vision that I would sit
behind my computer and code and tinker
with what I like to do.
And that was it.
And I heavily underestimated
the soft skills part of it.
So that in DevOps, you're all about
breaking down silos and helping teams to
reach their goals as a DevOps Engineer.
And I didn't realize how much
interpersonal communication and meetings
and all of that is involved with that.
It's not necessarily the hardest part
about it, but that was something that I
had to like readjust myself to.
But then my nursing skills, and my
project management skills like
really came to shine there.
So that's, I think, part of my
success as a DevOps Engineer.
That background that I could apply those
skills in those environments.
And yeah, another hard part about DevOps
is that it's sometimes hard for me to
realize that not everybody is
enthusiastic about these things as I am.
So if you come into a company and yeah,
we're all about Kubernetes now.
Let's start.
Let's do everything Kubernetes.
And this is why.
And then it's like: hold on there cowboy.
You said the K word.
I said the K word.
Get out of here.
Mischa will no longer join this meeting.
He said the K word.
That can be a bit challenging.
And that's one thing you have to learn
if you're passionate about something that
not everybody is going to be as
passionate about as you are.
What does it mean
"DevOps - breaking down silos" in practice?
Do you have a story or an example to give
that illustrates that concept?
We were basically the cloud platform team
and then we had development teams relying
on us running that platform logically.
What would happen is that we would get a
ticket and then: oh, there's
a problem with the database.
And then it wasn't necessarily that
they were saying you should fix it.
It wasn't necessarily implied that there
was this expectation
that we just handle it.
But the way it was working in this
company is that there wasn't much
collaboration between those divisions.
So even though we were a DevOps platform
team, it was again, just Dev throwing
stuff over the fence to Ops.
What I emphasized was: is this urgent?
How urgent is this?
Okay.
Well, it can wait a day, okay.
Let's plan a meeting,
and let's go over it together.
And I would, I would, of course, look
into it myself and see if there was like
something immediately going wrong
with the database itself.
But then I would actually jump on a call
with the developer who
sent in this ticket,
and rather than just me fixing it,
I would do it together with him.
And not necessarily because
I didn't know what to do, but I was actually
showing genuine interest in
what is the application doing,
and how does it work.
And when you start doing that, then the
people, what they're working on, they
start to get very enthusiastic about it.
And in my experience, then you get into a
bit of a different mindset together
instead of like, well, there's this error
and we're fixing it.
It becomes more of a personal project that you're
a fun thing that you're solving together.
And when you do that,
you become more relaxed.
And then in my case, we found
the solution much faster.
And more often than not, developers were
pointing out stuff, like we were
screen sharing, and we were in the monitoring.
And I thought that I had seen
that I had analyzed
the problem and I thought I had it,
but he said: "Hey, what about that one?"
And I hadn't,
that one hadn't occurred to me,
and we dug into it,
and then that was the problem.
So that's a very long
winded explanation, I think.
But that's one thing that I have always
emphasized is: okay, we are DevOps team,
it's not supposed to be that way,
but how do you actually implementing that
collaboration between the two?
And for me, this was one way of doing
that, to actually do it
in meetings, in calls together.
That is a great example.
Great example.
It starts with working together on a problem,
not you throwing the problem
over the fence, and letting me handle it.
Because that in itself is the first step
towards building a silo.
Yeah.
The second step is accepting
the ticket, and going with it.
Don't do that.
Try to have the conversation.
Try to work on it together,
and be open to working together.
Full stop.
Yes.
We are in this together.
It's not me, and I'll figure it out,
and you'll just get the end result.
We will go through it together.
Bonus points if you can record it.
One of the best things which I've done is
recording internal meetings like this,
because we are a remote team.
Jump on Riverside, or any
remote software exactly like this.
Do the recording, do the screen sharing,
do some minimal editing,
and then you have a video of how you
solved the problem together.
Yes.
Post it internally and it's a reference
for example:
what does it mean to be on call?
How do you handle an incident?
What happens when you take
an application into production?
If a migration, if a database migration
fails, how do you tackle it?
How do you build preview environments?
Why do you need preview environments?
All sorts of questions like these that
come up, build a knowledge
base, a wiki of some sort,
A Zettelkasten. It's a system for
wikis, but still something
that captures that knowledge.
And it's not different
people doing things in a corner.
It's people coming together, recording
that, and then sharing that knowledge.
And with all the tech that we have
nowadays, starting
with AI, transcription,
this is easier than ever.
You still have to do it.
Totally agree.
And it's one of the main emphases I have
when I work somewhere is
usually when I jump on a team,
the documentation sucks.
And I always make it a point
of improving that from day one.
Like there's always the onboarding
process where you can't really do much
the first couple of days.
I always start with reading the
documentation, asking questions and
improving it already as I go along.
That's amazing.
Document first,
automate second.
But always document.
I used to do things
wrong for many, many years.
I used to like automation.
There's this Makefile.
How do you mean?
It's like self-explanatory.
And people would tell
me, no, where's the docs?
Who needs docs when
everything is here in the Makefile?
It took me a while, but I finally got it.
Documentation first, automation second.
Are there any hard-earned lessons, any
hard-earned DevOps lessons that you would
like to share with us?
It's really important to keep an online
presence and keep that well up to date.
So the moment you start, like you can
always lose your job.
And the moment you lose your job and you
haven't updated your
LinkedIn profile for three years,
or even not having a
LinkedIn profile at all,
is one of the biggest mistakes you can
make as an engineer, in my
opinion, in this day and age.
And I don't necessarily
like that it's the case.
Like I still am the guy when I'm on the
street and I'm taking a
picture with my cell phone.
I feel weird.
It feels still like this
abnormal thing to do to me.
Like I wasn't a
social media person at all.
But once I recognize the potential it has
and the importance it has
these days for your career,
then I think this is one hard-earned
lesson that I would like to give
everybody to share your journeys,
even if it's just one post a week.
But be active online.
It will only only give you benefits.
Build your personal brand.
Whatever that is, whatever works for you.
If it's TikTok, if it's Reels, if it's
Instagram, it doesn't really matter.
Twitter, Blue Sky, say
Twitter... X, shall I say or Xitter.
I've heard that's like a new thing, which
is like catching apparently.
Xitter, that's a funny one.
In Mandarin, it
actually means something else.
So anyway, I just read recently,
the point is whatever works for
you, videos, podcasts - do that.
And if it's too much to start one,
or maintain one, join one.
There's so many around.
Yes.
So do whatever you can do to
put good content out there.
Do that.
Personal blog. Does not matter what it is
as long as you do that,
because it's like compound interest.
You're putting in the
bank of your personal brand.
It's you. Be who you
are and show the world.
Exactly.
If you have a stack of 200 CVs,
and there's one person there
with a thousand followers and a
personal brand, he literally has a better
chance of getting the job.
Again, I don't necessarily like it this
way, but believe me, it is this way.
Another hard-earned lesson that you've
shared, and I picked it up from the
various YouTube videos
that you posted and blog posts, is:
you waited too long to start a home lab.
Yeah. I'm an academic-oriented person.
I studied at university.
I love reading books and getting into
theory and writing notes
and getting more and more
and more information in my head.
But when I started learning Linux
seriously, professionally,
I'm ashamed to say this, but I wasn't
entering the commands into
the command line for months.
I was just reading about
them and I said, "Oh yeah, okay,
well that's how the file system works."
Great.
But when I sat down and actually tried to
do it the first time,
it wasn't that great.
It hadn't stuck in my mind at all.
One huge mistake I made that I wasn't
applying what I was learning in practice,
and you really need to do that in order
to make things stick.
And one of the best ways to do that is to
start a home lab in
whatever shape or form that is.
Anyone can do it with any hardware.
That's it.
It's becoming more and more accessible
and you will learn so much.
So just go and learn.
Have fun.
Explore.
Tinker.
Exactly.
Yeah, I also always used to associate
this word "home lab"
with these huge server racks
with blinking lights.
That's what you see on YouTube.
I never understood that you could
actually do this on
cheaper hardware or on something
that you already have.
So my students, I always
say, Okay, go to your closet.
There's 99% sure there is one old laptop
in there with maybe four
to eight gigabytes of RAM.
Install Ubuntu server on it and you have a home lab.
You have a learning environment where you
can learn enough skills to get a job.
That's literally all the
hardware you need to get a job.
I went through every
single article on your blog.
It's amazing.
So thank you first of all for taking the
time to write them,
and then to share them.
And out of all of them, there's one that
resonated with me the most.
Do you want to guess which one?
My favourite one is:
"I'm in love with my work"
So I'm not sure if that's the one,
or it's "The power of writing".
It's either that one or the other one.
Okay, so I'm going to read a quote and
you'll recognize it, I think, after
the second sentence.
Once you decide on your occupation, you
must immerse yourself in your work.
Yeah, I know.
You have to fall in love with your work.
Never complain about your job.
You must dedicate your
life to mastering your skill.
That's the secret of success and it is
the key to being regarded honorably.
Who said that?
Who wrote that?
Jiro.
Who's Jiro?
Tell us about Jiro.
Jiro.
Oh, Jiro is the protagonist or the main
character of the
documentary "Jiro Dreams of Sushi."
And when I first saw this documentary, I
did not appreciate how
deep this actually is.
But Jiro is a sushi chef who operates out
of a subway station in
this tiny little restaurant
with like four seats in it.
And he is the master of
sushi of the entire world.
Like there's nobody better than him.
And I know nothing about sushi.
I have no interest in sushi.
But this documentary is the story of him
like committing to his craft
of becoming a sushi master.
And what they call in Japan a "shokunin".
Someone who is completely dedicated to
their craft and just doing
the same thing over and over and
over and over again and making it perfect
and even more perfect the next day.
That is the shortest
description I could give you.
That's a great one.
So this restaurant is in
Tokyo, by the way, Japan.
So if you're there, I forget the name of
the underground station,
but you will find it because there's just
one like this in the
entire world actually.
It's a three Michelin star restaurant.
Only one of its kind in the world.
And some say Jiro is the best sushi chef
that has ever lived.
Now, this documentary was recorded, it
was actually was it came out in 2011.
Jiro was 85 years old.
And he's been doing this.
He's been doing sushi
every day since he was 19.
Yeah.
Today, Jiro turns 99.
I say today when we're recording this
actually maybe a few
weeks ago, he turned 99.
I'm wondering, is he still making sushi?
Well, the last time I checked, that's a
couple of years ago.
Then he was also in his 90s and he was
still there every day in his restaurant.
That's amazing.
I didn't realize this, but in the
Japanese culture, apparently retired is
not that well understood.
No.
When we first talked, I had a title in
mind for our conversation.
Do you remember what that was?
The Note Taking Santa Claus.
That's it.
Okay, we will get to that in a minute.
I hope or a few minutes we'll see.
But after watching "Jiro Dreams of Sushi",
and reading all your blog,
I came up with a new title proposal.
Let's see if it's as
good as DevOps Sushi.
I was thinking:
"The Bear that Dreams of DevOps".
The Bear that Dreams of DevOps.
Wow.
Okay.
Why is the bear an important thing for you?
Because I know it is, but why is it?
Because people listening
to us, they won't know.
You really have read all of my blog.
I'm so honored that someone...
This also is featured on the blog and why
I chose the logo at the time that I had.
And Mischa is a Russian
name that means"little bear".
So I'm conveniently
ignoring the little part here,
but it might not seem on the camera, but
I'm Dutch and I'm almost two meters tall.
So I'm a pretty big guy.
And in Norwegian, they
have this word called Bamse,
which also means teddy bear.
But they usually describe that with
for long, big hairy guys.
And well, if you're watching this on a
video, you'll see the connection there.
For our listeners, imagine Thor.
Wait, let me get my hammer.
Exactly. Yeah.
That's the only
thing missing from the picture.
Yeah...
So another thing which I watched over the
weekend on top of "Jiro Dreams of Sushi"
which is a great movie,
10 out of 10, highly recommended,
is "Why I use Kubernetes for my Homelab."
A video that you
posted not that long ago.
We'll put a link in the show notes.
And I'm wondering if you
could take us back to the day
that you decided to use
Kubernetes for your home lab.
I think it's a very interesting choice.
How did it start?
That was when I was
working in my first DevOps role,
and we were using Ansible for everything.
And they had built this super, actually
very impressive system
of deploying Docker servers and then
having three Docker servers,
having running the same container and
then putting them there.
and then putting them there.
But when one of those
containers would die,
then we would have to
go in and fix it, right?
Because you're limited by the tooling,
even though there's probably ways of
doing it with Ansible alone.
That was my reality of the day, of the
situation I was working in.
And when I then for the first time
created a Kubernetes cluster,
and I created a
deployment with three replicas,
and I killed one of those containers,
and I killed one of those containers
and then it came up
automatically, I was sold.
And then when I did a rolling update,
where I would update the image tag,
and then it would go one by one, and then
update the containers.
And if something would go
wrong, it would roll it back.
That's when I fell
totally in love with it.
And I knew I could not go back.
and I knew I could not go back.
For me, that was the
moment where I thought,
this is what I want to work with.
When I then decided to do my
home lab, to create a home lab,
like it wasn't even a thought,
I didn't ever even
consider anything else.
It was just, yeah, I'm
specializing in Kubernetes.
Of course we need to run
Of course, we need to run
Kubernetes in my home lab.
Even though it's completely
overkill, and it makes no sense,
it had to be done on Kubernetes.
Well, actually, I think
it makes a lot of sense.
And you talk about this
in your video as well.
Why it makes sense,
and why it is important.
And the majority would react that way.
It's overkill.
Of course it's overkill.
Why not use Docker?
You started with K3S.
Why?
Part of me wanted to go the kube-adm way,
Part of me wanted to go the Kubeadm way
and just do it properly.
Even kube-adm is a bit
easier than doing it the hard way,
like compiling the
binaries yourself, etc.
Still a goal that I
have for maybe next year.
I went for K3s because I recognized
I went for K3S because I recognized that
the sooner I got up and running,
and actually working
with the Kubernetes API,
and feeling that I had
something stable to work with,
the quicker I did that,
the more I would learn,
the quicker I would start learning.
So I think it might felt
like a shortcut at that time.
but I thought long-term and I thought,
But I thought long-term, and I thought,
well, with kube-adm, like,
like running your own CNI
running your own CNI, and having to
update your certificates,
like, it will work in the beginning,
but if something breaks,
then I'm just sitting
there tinkering for hours.
And I wanted something that was a little
bit more thought through,
and K3S was a very good option for that.
Why not Kind?
I had a few Linux machines
I had a few Linux machines,
I had a few Linux machines
and I really like this
idea of that k3s does,
where you install the binary,
and then you can get a token,
and then you can add
another node to your cluster.
And I actually, you can of
course do this with Kind as well.
But I really like this idea of
maintaining a Linux machine,
maintaining a Linux machine
where you install something on,
but you still have the
possibility of SSHing into it,
and you still are a Linux administrator
at that point, right?
And I really like that balance,
that I could have those
Linux machines running Linux,
I could tinker with it,
I could still run other
things on that machine as well,
alongside K3S,
and then just expand my
home lab as I went along.
I really like that concept.
Okay, so you mentioned
SSH, which is interesting,
because then you switch to Talos,
because then you switched to Talos
and Talos prides
itself on not having SSH.
Yes.
Why did you switch to Talos?
Well, I mean, that was
that stage in my home lab.
And then later, I was running
applications that I was thinking,
well, this is
actually running very stable,
and my home lab has
now escalated to a point,
where I don't want to
live without it anymore.
It's actually containing applications
that I want to keep running 24 seven,
that I want to have
accessible from the internet.
And then the question
of security comes around.
And you need to, when you start exposing
things to the internet,
you need to really
know what you're doing.
It's a very dangerous situation.
If you don't know what you're doing,
and you're just opening ports and letting
people into your cluster,
they could break out of your
containers and do bad things.
So at that point, I felt, okay, I want to
have things more secure.
And then Talos came on my radar,
which is production grade
security out of the box.
And it wasn't necessarily because I felt
incapable of securing my own cluster.
It was more a combination
of learning a new technology
and also having this peace of mind,
knowing that what I was
doing was actually secure.
So then I switched to Talos
and yeah, I'm never going back.
I think it's been amazing so far.
If someone wants to get really good at
Kubernetes and start a homelab today.
Yeah.
How would you recommend
that they do that?
I'm currently in the process
of building a course for this,
the Kubernetes Homelab.
It's 60% done and people are responding
really well to it in the community.
They love it.
And the approach I give there is,
first I have my
Kubernetes Fundamentals course
where they run Rancher Desktop
on their own machine
and then to learn Kubernetes.
And then the next step is
to actually get a laptop,
a Raspberry Pi or some
sort of dedicated hardware
and then run k3s on it.
That is the approach that I advise to do.
So first learn it in a local setting
and then move to dedicated hardware,
use k3s and learn
GitOps as soon as possible.
So where could people find this course?
The course is currently
available in my community,
KubeCraft that I started at
the beginning of this year,
where I invite engineers to come in and
engage with each other
and where I also have my note-taking
material and my Kubernetes courses.
Obviously we'll add a
link in the show notes.
I had a quick look.
I haven't joined the community yet.
I'll put yet there.
I've seen that it's
on skool.com with a k.
Can you tell us a little bit about that?
Well, maybe I can just briefly describe
why I even started doing this.
As a YouTuber, I was just making videos
because I like sharing knowledge
and then at some point people were
actually telling me that:
"Whoa, we really like your
approach to explaining things
and how you approach things.
Don't you have a course on X or Y?"
And when I realized that my audience
started asking me for this,
I thought: "Well, maybe I do have some
potential of going
beyond sharing on YouTube."
So at that point, I was
like any engineer considering:
"Oh, I should build my
own course platform."
Let's code my own platform.
And then you spend a weekend on it and
then you start looking into
authentication and user
management and all that stuff.
And then I figure:
"Okay, maybe that's not the way."
And then I researched the platforms and
of course there's
Udemy, there's Kajabi, etc.
But what Skool really emphasizes is
combining education with community.
And when I heard that sentence, I was like:
"Yep, this is it."
Because I don't just want people to buy a
course of me and then that's it.
I mean, that's fine, but it's much better
if you can create a group of people
who are gathering around
their interests for those courses
and then can get support from me,
but more importantly from the collective
knowledge of that community.
They can actually draw from each other
and discuss tactics with each other.
And that is proven to
be very powerful so far.
So this community, when you started
earlier this year, started at zero.
Where is it today?
We are currently at 401 members.
Wow, so just past 400, okay.
So it'd be interesting to see how many
members there will be
by the time this comes out.
And you'll have the
delta to see how that grows.
I also know that
obviously this is a paid community
because of the value that you get.
But there's also a free community that
you have also on Skool.
The free community now has 2000 members.
And I only started that a month ago.
Wow!
So that has been very
successful in that sense.
Well, I do ask, I charge a monthly price
for membership of the premium community.
The services that I provide for KubeCraft.
I decided: well, I'll
create this free community
that people can at least come together
and meet people who are...
They're watching my content.
So there is for sure some level of
overlap in their interests,
in note-taking, DevOps and such like.
So people can come there, make friends,
and they can discover the platform.
And I give away a lot of material in that
free community as well.
And then if they like, if
they see the value of it,
then maybe they can consider
joining the premium community.
Okay, great.
We will make sure to add
both links in the show notes
so people can go and
check them both out to see
where they want to start
and what they want to do next.
What would you say is the main reason
that people go from
free to the premium one?
I think there are two kinds of people in
the world in terms of employment.
There is the guy who shows up at work,
does his work, goes home,
and then plays video games
or does whatever he wants
and then he shows back at work again.
And then there's the guy
who is entering the stand-up
who excitedly talks about a new feature
that came out on Talos
or he managed to get this self-hosted app
up and running in his homelab.
And he talks about that.
And I'm obviously the second guy
who likes to tinker with what he does
in his free time as well.
Well, imagine putting 400
people like that together,
all sharing what they're discovering
and what they're learning.
And what you get from
that is a sense of belonging.
I was never able to find this place of
geeks and nerds sharing homelab stuff
and learning Kubernetes and learning my
approaches to
note-taking and personal branding
and doing that together.
And I really found my tribe in this.
I created something that wasn't there and
I really feel at home there.
And there is, we have calls
there as well a few times a week
where you can ask me questions directly
on how I approach things.
But what's even better, I don't try to be
the guru of the community.
I'm all about letting the
community answering things.
Because I can only know so much.
I only have this experience.
When you put 10 experienced engineers in
a call and when you come in there,
maybe someone who wants
to transition to DevOps
and you're in that
meeting and you're saying,
"Oh, guys, I have this
problem in my homelab."
or "At work",
or, "I don't know what my next
certification should be."
If you have all that
experience sitting there in a meeting,
you're going to get the answers and
you're going to get good answers.
And it's amazing.
I'm amazed how well it all works and how
much progress people are making.
Coming back to the technical side, we
have a couple more
things to go through before
I'm very keen to start screen sharing,
and showing our homelab,
because that's what this is about.
Talking about it, but then also showing
what that means in practice,
which I'm very excited about.
What was the hardest issue
that you've hit with Talos?
When I set up my Talos cluster, I had
baked a custom image.
There's the image builder, and then you
can add certain drivers that you need.
But I did that, and I apparently didn't
document that properly,
because I need the
iSCSI tools for my Synology
to be able to provision
storage on the cluster.
And I had updated my Kubernetes cluster
with the Talos update,
so new Talos images are
loaded onto the nodes.
And all of a sudden, all
of my storage was broken,
and I can be honest, I panicked.
Fortunately, everything
is backed up to the cloud.
But that was a big problem at that point.
And then after a lot of debugging,
thinking it was Synology
pods on the cluster itself
that maybe got updated
or weren't compatible,
I finally realized, oh yeah, I had that
iSCSI driver that was missing.
So that was the biggest one.
Yeah.
Anything that happened in the recent
update that you haven't posted yet?
I ran into something where I would
initiate a reboot of the node,
and then in the CLI, it will then, say,
now waiting for the node to come back up.
And then I could see in my closet where
the nodes are standing, it was up.
It was up and running.
But it wouldn't work.
I would have to turn it off and on again,
and then it would reboot properly.
Like, I decided not to spend too much
time into why that was.
At least I knew how to fix it.
But that was an
interesting one that I had this time.
So this is a tip from the experts.
Turn it off and on
again, and it will fix itself.
That's one.
Which version did you
upgrade from and to?
I went from Talos
version 1.7.5 to 1.8.2.
That's it.
Yeah.
I assumed this was it.
From 1.7 to 1.8.
Oh, really?
I did went from 1.7.5 to 1.7.7.
So I was a good boy, and I followed the
latest patch upgrade,
and then went to the next minor version.
But you had the same
problem too, apparently.
So I don't do in-place upgrades.
It's one of my hard rules.
I never do that.
Actually, my clusters, they have a date
on the cluster name,
which tells me exactly
when that cluster was created.
And I never upgrade minors.
I rarely upgrade patches.
I just need to know when.
Even Kubernetes, sometimes I never
upgrade, for example,
from one Kubernetes minor
to another Kubernetes minor,
even though most experiences are good.
The problem is everything else that you
have running in the cluster
is very difficult to
reason about the combination
of the different tools that you have,
because it's not just Kubernetes,
not just the operating system.
There's also everything else.
What I found is that if
you do in-place upgrades,
you're risking it.
The more upgrades that work,
the more likely you will
hit a bug at some point
where you wish you hadn't done that.
Yeah.
If you don't do in-place upgrades,
it forces you to encode
the system in a certain way.
It forces you to restore from backups.
It forces you to have a certain approach,
which is operationally more mature.
It's a bit harder, but you
can sleep safely at night
knowing that you can restore everything,
even if it was to burn down,
because everything is configured,
everything is backed up.
You just point to a new cluster,
everything gets restored
or you create a new cluster,
you give it a new config
and it boots as a new cluster.
Most of my workloads,
they have built in a
way to restore themselves.
What that means is that
in the init container,
when a workload starts,
it looks if it's the
first time that it's booted,
it's like it's a bootstrap phase.
And if it's a bootstrap phase,
it will go and restore
from an S3 compatible API.
Nice.
In my case, I use two.
I use Backblaze B2 and Cloudflare R2.
And I do this via rclone.
So rclone is the first thing that restores,
even MySQL.
You'd be surprised
how well this stuff works
for even for large data,
because you're pulling
down from object storage
that is supposed to be performant.
Backblaze is not as
performant as Cloudflare R2,
or at least in my experience,
but I have both because you
don't want to put your eggs
in one basket, especially backups.
So I have two of
everything, another rule that I have.
The other one is don't
do in-place upgrades.
It's not worth it.
So when you do an upgrade,
then you would actually just spin up an
entirely new cluster?
Okay.
So also just like completely new,
well, not new, but
like different hardware,
spin up a new cluster,
and then you decommission the old one,
and that becomes your staging cluster.
I'm thinking of it blue-green,
but long-term blue-green.
I'm always between blue and green.
And it removes all the pressure
from having to do the
upgrades straight away.
It'll either work or not work.
No, I'm setting things up.
It's almost like buying a new car,
but in this case, it's more practical
because it's not a car.
You do need to have a bit more hardware,
but you have that anyway,
because you have generations of hardware.
You have your old laptop and the older
laptop and whatever.
So that's my case as well.
Also, very controversial,
my clusters are single-node.
Ooh, spicy.
I know, right?
Yeah, you can get some pretty big
single-node instances.
And unless you really
understand your networking
and can deal with etcd,
and you can do uneven numbers,
knowing how Raft works
and how leader election
works and all of that,
you need three, five, seven,
you're pushing it a bit.
If you go beyond nine,
you're basically like
in a whole new world.
I think very few have more than nine
nodes in their homelabs.
But that should be the absolute limit
for many reasons.
These systems are so advanced
that they will continuously converge,
and you can have problems
and not even know that
you're having problems.
That's how refined they're getting.
And usually when you get into DNS
and if you get into network latencies
and switch upgrades,
and you have to deal
with all these scenarios.
Single-node, you don't.
In that metric,
I am currently at four,
but I had five for a particular reason.
But you have a single control plane?
Currently, yes.
Right.
So that's slightly different.
So single control plane,
that's almost like the middle ground.
So the control plane is not HA.
In my case, my control plane is not HA,
but all my workloads fit
on the control plane node.
Yeah, I see.
And then I have two right now,
and then basically one
is blue, one is green.
So I have two home labs,
and I just switch between them.
And it takes me months to
go from one to the other.
So nothing is rushed.
I can experiment, I can try things out.
And if one was to break,
I always have like a
spare that I can go back to.
That's the other thing.
I used to have a staging cluster
and that ran exactly the same.
And I would just then update the GitOps
for the production one
if everything went right.
However, I actually
started needing to expand
my production cluster with extra nodes.
And I cannibalized some from my staging.
And so that one's not
operational anymore.
And to give you an
alternative perspective
on not doing in-place upgrades,
I found that it makes
me think about my systems
to have them more
resilient and easier to restore.
And I'm not in the stage where I have
auto restore like you,
but what I emphasize a
lot on my current cluster
is I don't want to have anything
persistent outside of databases.
So my databases are
running in with the EDB operator,
Enterprise DB.
And then using Barman,
I'm also storing those in object storage.
And that has been a really good exercise
like four times already.
I lost all of my databases
because I was a bit too experimental.
Tinkering.
You were tinkering.
Yeah, it happens.
That's how you know that
you're tinkering the right way.
When you break stuff,
you're pushing the limits.
So I've actually gained a lot of
confidence in my setup now
that I can actually
just delete the whole thing
and then restore it from object storage
because everything is in databases.
And anyone who's listening to this
databases on Kubernetes
is easier than you think.
And it works extremely well with the
Barman object store.
It's that.
I'm so, so excited about it.
It's super cool.
In my last Homelab
presentation, which I gave at Taloscon,
I use CloudNativePG, but
also Barman and object storage.
And that's exactly how I migrated from
homelab to production.
Because in this case on homelab,
I was running a workload
that was backing out via Barman
to R2, Cloudflare R2 in this case.
And when I set up production,
I was restoring
everything exactly the same way.
Would you recommend iscsi-tools?
Would you recommend it?
Like does it work well,
the extension in Talos
and like the whole setup?
Because I'm thinking of
setting up the same thing.
I don't have that yet.
But would you recommend it?
Like I can't speak to
iSCSI tools specifically
because I just run that and it works.
So I have a Synology NAS,
and then I use the iSCSI tools in Talos
that needs to be installed.
And then there is a CSI driver for
Synology for Kubernetes.
And that allows me to
provision persistent volumes
on my Synology NAS.
And I must say I'm really
impressed how well that works.
I know that you started with Argo CD,
but you're currently using Flux.
Yes.
Why?
I work with Argo CD at work.
And when I first learned Kubernetes,
I was super excited about Argo with UI.
And still, it's like a fantastic tool.
That was my first GitOps exposure.
But then I'm a
specialist in Microsoft Azure,
and I'm also a Microsoft MVP.
So I focus a lot on
Azure Kubernetes Service.
And in terms of
enterprise-grade Kubernetes,
I like to, like, as I mentioned before,
I like to explore and experiment,
but I also need to focus myself.
If not, I get too broad
and I just get overwhelmed.
So in terms of
production-grade Kubernetes,
I decided to focus on
that ecosystem for a while,
which has been a good
decision in my case.
And the native GitOps
offering in AKS uses Flux,
which I was very surprised by
when I first tried that out.
Like: "Oh, that's an interesting choice.
Why wouldn't they go for Argo CD?"
And I don't have the
answer to that question.
But all I know is that when I started it,
I was really surprised how well it works
and how well it fit me.
It fit me better than Argo CD.
What specifically
about it fit you better?
Argo CD has a beautiful UI,
which makes the
navigation of a Kubernetes cluster
possible almost
exclusively through the UI.
So you can do almost
everything through the Argo CD UI.
And I noticed at work as well
that I was kind of
losing my CLI ninja skills
in terms of kubectl
because I was deploying
so much through Argo CD,
which was, I saw as a problem.
And what I like with Flux
is that it's much more CLI focused.
And of course, like, usually you're not
actually interacting
with the cluster in that way.
But in terms of my homelab experience,
I want to be in there and tinkering
and just talking to my cluster directly.
And then the Flux CLI
felt very intuitive to me,
even though Argo CD also
has a CLI, I'm aware of that.
But the way how Flux does
it with the custom resources
such as Helm releases and Kustomizations,
and being able to check those resources
and Git repos, all of that,
I really like the way how that is set up.
And also, I completely love the way
how Flux bootstraps onto clusters,
how you can make your cluster like a
self-feeding machine.
It's just super awesome.
Okay, interesting.
How was the migration
from Argo CD to Flux?
Because you had like a bunch of things
all declared in Argo CD, I imagine.
Was it straightforward to migrate to Flux?
At that point, when I was using Argo CD,
I was getting very Helm focused.
And at some point, I realized,
well, not everything that I want to run
has Helm charts readily available to me.
So I need to dig deeper into this.
Then at the same time,
I was discovering Flux.
And there is a lot of
emphasis on Kustomize.
Basically, everything
works with Kustomizations.
That was basically an incentive to
finally learn Kustomize.
I had been putting it
off for the longest time.
I just tore everything down and rewrote
everything in Kustomize,
which has been a really good experience
in terms of my development.
And Kubernetes learning.
When Rio mentioned you,
he caught my attention.
And that was him mentioning how you spent
nine years in Norway.
Roaming the wilderness.
Yeah.
Reconnecting with yourself,
reconnecting with the nature
and remembering what is really important.
Well, I must say, like, it wasn't that I
was doing that full time, right?
I was just living
a normal life and working.
But basically, if I wasn't
like tinkering with computers
during the evenings in the week,
every holiday, every weekend,
every opportunity I got
was spent being in nature.
This will sound so cliché.
But one of the reasons that got me into
this was the movie "Into the Wild".
And just that phrase, "into the wild,"
I have since become less of a fan of
Mr. Christopher McCandless,
who actually turned
out to be rather stupid.
But when I was young, I didn't actually
appreciate his mistakes.
Let's keep it that way.
But that idea of being
out and being self-reliant,
that really applied
to me, and spoke to me.
And...
I have since come to realize that
that is also not possible.
So that's also why I was...
that I went back, basically.
But the idea of the experience of just
strapping on a backpack
and driving your car somewhere with a
vague idea of what you're going to do.
And then I would just bring a sack of
oatmeal and my fishing pole.
And I have since turned plant based,
but then I was still
killing animals and eating them.
But that did allow me to be out there
alone on a self-reliant matter.
And when you do that, then you...
No phone, alone,
in the wilderness, no people.
First of all, you become
really aware what you're doing,
because if you make a wrong step and you
fall, you break your leg.
And that is almost
certainly going to be your death.
So you have to be...
You have to know what you're doing.
And you have to be really
aware of what you're doing.
And when you're out there
not talking to people,
just there with your own
thoughts, things emerge
that would not emerge in the
day-to-day stress & busyness
that we all experience.
And I've had some very
deep philosophical insights
and discoveries about my own life.
Just sitting there, by the water, staring
into the fire that I built myself...
for hours on end.
And you'll be amazed what
the mind then brings up to you.
So that's an indication
of what that was like.
Do you still do that?
I mean, do you still make time for going
out there and being with yourself?
No phone, no people.
I think an extension of this whole
wilderness roaming that I did
was that I got really into meditation and
Buddhist meditation specifically.
And I have now found ways
of getting that experience
decoupled from being in nature that way.
So I'm a very consistent meditator.
I meditate for several
hours on end sometimes.
And I moved back to the Netherlands,
and here nature is much
more difficult to find.
It's here, I have
nice areas around me,
but I do really miss being in the forest
and just knowing that
you are alone there.
And the idea that you cannot
meet anyone,
or it's not likely
that you'll meet everyone.
Like here in the Netherlands, there's
always someone around you.
I did go back to
Norway a couple-- one year ago.
I spent three weeks
driving around and just roaming
and having the time of my life.
Last year, I didn't do it, and now it's
really itching to go back.
So I try.
I know that each of us has a
place where we feel at home.
And it's rarely where home is.
It's usually like somewhere maybe nearby,
or it can be a
different country for sure.
But when you're there,
you feel like you're home,
and it's very hard to explain because
everything is the way
it's supposed to be.
Everything smells right, it sounds right,
you feel very at ease with yourself.
Is Norway that place for you?
That's a deep question.
That's one that I...
Like Norway is
extremely close to my heart.
Like one experience that I had
was when I moved back home,
I hadn't visited for two years.
And then last year, I went there
and I drove off the boat
and it felt like it didn't feel special.
That was interesting.
It felt like, oh, I'm here again.
So in that sense, yes, it does feel like home.
But I'm also really starting
to like my little boring life
in just a one-room apartment
and focusing on my
business and the technology
and having a more
routine life in that sense.
I think this is potentially
the most important takeaway
from our conversation.
Going outside.
Slowing down.
You don't necessarily have to go outside,
but usually it helps.
Slowing down your mind, and...
finding your happy place.
It's usually in your mind.
The happy place is in
your mind, by the way.
Outside, walking, being...
free from distractions
usually helps that surface.
But ultimately, you
still have to do the work
and you still have to settle your mind
and go through all the steps.
But it's definitely out there.
Everyone has it.
Yes. And nature is a
great facilitator for that.
And I think in my case, nature was the
gateway to inner peace.
Nature was the place where
I could disconnect that way
and sit by the fire and experience that.
And still, of course,
still nature is that for me.
Last year I was there,
I walked 150 kilometers
in the mountains and just alone.
It makes you feel very small
when you're in those big mountains
and a thunderstorm breaks out
and it puts so many
things into perspective.
And how powerful those forces are
and how little we are
in comparison to that.
That is also very healthy for me
to have that humbling
experience every once in a while.
That's it. And when you come back
to whatever you normally do,
you'll be so much more productive
and things will flow a lot better
because your mind took a break.
Absolutely.
Okay. Well, Mischa, thank you very much
for joining me today.
It's been a pleasure talking to you,
looking at your homelab, you know...
Going through that
stopping and starting phase.
That was very, very fun.
And I'm very much
looking forward to the next one
because I haven't shared mine yet.
I mean, I really look
forward to seeing your homelab.
And we have to do a round three.
We will. Yeah. Yeah.
For sure. Thank you.
I'll see you next time.
Thank you, Gerhard. Bye-bye.