I think companies of all sizes can benefit from Kubernetes.
Like the, the line people like to use is, you know, you are not Google.
You don't have Google size problems.
You don't need Google size solutions.
Welcome to fork around and find out the podcast about
building, running, and maintaining software and systems.
Welcome to fork around and find out the most
animated infrastructure podcast on the internet.
I am Justin Garrison and with me is Autumn Nash.
And today's guest is Sam Rose.
Welcome to the show, Sam.
Hey, thank you for having me.
I've been waiting for this episode because Sam's animation fire, like I love
art and technical and when that like comes together, it makes me so happy.
And I want to get into that so bad, but I want to first talk to Sam
about Where you've been before, what you're doing now, because I did
not know that you were doing this crazy, awesome animation as in your
back end engineer, like you're just like, Oh, no, this is on the side.
I don't know how to do JavaScript.
I mean, you just did a lot of JavaScript.
Yeah, this surprises a bunch of people.
My background is almost entirely back end infrastructure, devopsy kind of thing.
So I started.
my career way back in like 2012.
Oh, I feel so old.
Don't say it.
I feel old now and I'm getting like gray bits in my beard.
Like the tides are turning, but like 2012 Ruby shop doing rails and stuff like
that way back when, uh, I think Ruby was kind of hitting its peak, like 2010.
It's very on brand.
Yeah.
2012 Ruby rails.
Everyone was doing it.
Then no JS came out of nowhere and things kind of shifted.
But back then, yeah, working for an ad company mostly.
And then.
I went from there through like an acquisition.
To another company I didn't stay long at.
It was a bit of a, not like hostile acquisition,
but it wasn't very nice for everybody involved.
I think it was fairly clear that we were
required for technology, not for people.
So over the course of a year, most of it left there.
But then, I ended up getting what I thought at the time was my dream job.
So I worked at Google for four years.
Uh, I joined as a Site Reliability Engineer on the Android team.
Stayed in that for two years and then did two years
of software engineering on the Android Studio team.
Afterwards, and like at the time, this was probably a little
bit after what people would call the golden years of Google.
Like, I think, uh, it had already begun a bit of a decline in public
perception, but it was still an incredibly good place to work.
And I learned a ton of things.
I'd say probably the main thing I learned after the four
years there was big companies aren't really my thing.
I did the initial switch away from site reliability
engineering to software engineering because I'd missed
writing code a bit, like as a site reliability engineer.
At least on the team I was on, the things I was doing,
not an enormous number of opportunities to get to get into
code and write features that users will end up seeing.
And I missed doing that.
What were you reliability in, uh, on the, on Android as a, as a site?
Like, cause that's, those are, that's a phone operating system.
The phone
talks to a whole bunch of stuff.
Uh, so at the time, the Android SRE team was split into two halves.
So there was the, the Play SRE team, which took care
of the Play Store and everything related to that.
And then there was what was called the Cloud SRE team.
And this is all ancient history now.
These things have changed substantially.
But we were responsible for The phone backup system, the authentication system.
So the thing that maintains your Google login all
times, we were on call for the push notification system.
I think by far the biggest thing we were on call for, we
were on call for a couple of like the device OTA system.
The thing that told your phone it needs to
update a couple of the smaller things like that.
Those are kind of the main hits.
The two that I focus most of my time on
was the backup system and the auth system.
So the team.
There were kind of six of us in London and then the six over in the U. S. and
people kind of specialized on each of these services because these services.
It's Google, they're huge, like, they serve hundreds of
thousands, if not millions of queries a second and it's hard
to specialize in all of them, so I took backup and auth.
My main, I would say, like, crowning achievement while I was
there was taking both of those services from two nines to four
nines of reliability, uh, through, like, lots of iterative change,
working with dev teams, capacity planning, tweaking code paths
to be more reliable, quite a lot of, Small nitty gritty things.
So that's broadly what I was responsible for when I was at Google.
Now, I mean, usually going from 2 to 3 nines is
like, we can just write some better software, right?
But going from 3 to 4 is a lot bigger lifts
and usually involves more process, right?
Because going from 4 to 5 is definitely like the people side of things.
It's not necessarily your code is just amazing.
It's, we handle this better.
But you said you weren't writing a lot of code.
You said you wanted to get out of SRE.
Because you weren't writing code.
So, uh, kind of
user end, like, uh, end user features is the stuff that I was
kind of interested in, rather than, like, trawling through.
Tons of logs to find some specific error that's caused by some race
condition somewhere like that's fun But it's not fun six months of the year.
It's kind of interesting going this like four nines to five nines It's
very challenging But the fun thing for us there was there's really
no point in doing it because the the telcos themselves were not five
nines So there was no point being more reliable than the network.
These phones were operating on So we we never saw five
nines of the goal for most of the stuff on Android
That's interesting that the phones weren't as reliable.
So you had to almost kind of walk back
how much you were expected to be reliable.
Yeah, it was nice.
It was, it was almost like a freeing thing that we had these kind of slightly
lower reliability targets than some of the other core Google services.
And just being able to calculate that out of just
like, Oh, the thing I rely on is not reliable.
So I'm not going to try to like beat it.
And people often move to the cloud and they're like, I'm getting five nines.
I'm like, the service you're using is three.
Good luck.
Right?
Like, this is not, this is not going to add up.
Right.
I don't think people realize that there's different
reliability for services versus like the cloud in general.
Like, I don't think people actually realize that.
Can you explain what being an SRE like at Google was?
Because I think like being an engineer on a release team.
and building language is so different than the traditional SCE job.
So I think, like, it would be interesting to hear what, like, your
SRE job was, versus being, like, doing software development, you know?
The SRE job at Google, as I experienced it, and it does differ between teams.
So Google, for example, If you're working on some core service that
everyone depends on, so say you're in like a networking SRE or something
like that, your life is very different than if you're working on like
product based things like Android, which is more of a product based thing.
But there's a lot of looking into the future, so a
lot of kind of capacity planning, like can we service
the expected growth over the next 6, 12, 24 months.
There was quite a lot of communicating with development teams, say
if, if something in our monitoring starts to like kind of break down.
strange or unexpected because maybe some
release or something unexpected has happened.
We will kind of work with dev teams to help them understand
how that was affected by changes they may have made.
There was an aspect of you were encouraged to, how do they phrase it?
Something like automate yourself out of a job every 18 months
or something like that was one of the taglines they had.
So it's like, whatever you're doing now, there should be a percentage of your
time dedicated to making sure you don't have to be doing that in 18 months time.
In my experience, that was more of a, a wishful kind of
goal than something that happened regularly in practice.
Some teams certainly achieved it occasionally, but I think
most people were still doing the same stuff 24 months
after that they were doing, 18 months prior to that.
It was a lot of almost babysitting, but I mean that like, I think
that has some negative connotations associated with it, but it
was, it was like, there's gotta be a better word for this here.
Uh, not championing, not like, You are, you are the person
like responsible for this being a reliable service, right?
You are like the vanguard, you know?
Like anything, like anything getting in the way of the
service being as reliable as it needs to be is on you.
And the way you are babysitting them,
because you're keeping it alive.
For sure.
Yeah, yeah.
Being on call is a big part of the job as well.
So, uh, on call like roughly a third of the time.
And nobody that I have worked for before or
since does on call quite as well as Google does.
Just in terms of sort of emotional well being, I guess.
So you're never on call 24 7.
There's a limit to how much on call you can do in a given amount of time.
You're always paid for on call.
So it's a very fair system.
Every so many hours of on call that you do, you get an hour of time off in lieu.
So I would End up racking up something like 35 to 40 days worth of time
off in lieu over the course of a year and just take big blocks of time.
Yeah.
I'm And time off, which is really nice.
Yeah.
It's like, I'll, I'll see you
And you have to be, you know, mindful of the team and, and what
everyone else's responsibilities are, but you don't want to kind
of put like furloughs on subsets of people, but, but you wouldn't
be, you said not on call over 24.
Twenty four seven.
So is that like just waking hours?
Like you're never on call when you're supposed to be asleep?
The, the way that we did it was
in the UK it was 7:00 AM 7:00 PM.
during the week.
So actually, no, the weekends as well.
It's, it's, um, and then four hours of that
12 hour, uh, is what you're being paid for.
You didn't get paid for time where you were in work.
So during the week, you're getting four hours a day of time off in lieu time.
It wasn't actually, it didn't directly correlate to four hours, but
then the weekend was the full 12 hour block would count as paid time.
So yeah, 12 hour shifts.
And then you switch over to the U S.
That's wildly fair compared to most big companies on call.
And I almost think that it's like a buy in, you know?
On call sucks, but when you get the time off and paid and all of that, and
you're not being on for 24 seven, you'd probably have a better attitude
towards your on call, but you also wouldn't be burnt out from on call.
So you'd probably be a lot better at your job.
Completely agree.
And then the atmosphere in site reliability engineering, really good.
And I miss it a lot.
I will be honest, I think people really cared for each other.
People really cared about the workload you had, and they really
cared about how much Because not all on call shifts are born equal.
Like, you know, you could be dealing with a week of absolutely
nothing, like nothing goes wrong, which would be great.
It's rare, but it's great.
Or you could be dealing with like the worst outage you've ever seen.
Right.
And then maybe you had scheduled where you're covering somebody
else's shift the week after that, and then you completely burn out.
So you don't want to do that.
People are generally really good at making sure everyone's okay.
And the education aspect of being an SRE at Google,
seconds to none, like anything I've experienced this.
There's so much material that's been produced to help you onboard as
an SRE, and teams will have kind of personalized material for that team
for, you know, we care about these things, here's the onboarding doc,
here's some onboarding videos, you know, we'll, if you want to, we can
sit through it together and whiteboard it out and things like that.
Really, really good, and I've not seen anything like it since.
Documentation and run books can make or break an on call like shift, right?
It's like, Oh, if I don't have a run book for this
alert, like I don't make me start from scratch.
Like that is just so hard.
And yeah, teams don't invest enough in that.
One of the things that I remember from the SRE
book, cause this is, I mean, SRE book came out.
Like 20 18, 20 19.
And you were probably there during this like . I I was there.
Yeah.
I have my
free copy of it.
. One of the things that I always point out to people that
are trying to start an SRE team is, is from the book.
I say the, the thing that Google, at least from the
outside in the books did, was they gave the SRE team power.
to say, we're not going to support your service.
In the book, it says that like you are embedded on a product team.
If the services are not reliable enough, SRE can
pull out and say like, Hey, you're on your own now.
Dev team, go do your own thing.
Is that how you experienced it?
Is that actually whatever happened?
It sounds like you were embedded full time with Android backups.
It
is true that SRE gets quite a lot of autonomy in what they do and don't support.
And I was involved in a fair number of what
they call production readiness reviews.
So you might imagine, um, there's a much smaller number of
SREs than can reasonably look after all of Google's services.
You know, Google runs an incomprehensible number of services and production.
And at the time when I was there, there was
like, you know, a few thousand SREs globally.
So it was kind of rationed out, I guess.
Like, you couldn't come to SRE and ask for support unless
you had something that was a pretty critical service.
And the process for being onboarded into an SRE team Was this
production readiness review where SREs would sit with you, you know,
you'd whiteboard the architecture, you talk about how it works,
you kind of, as an SRE, you'd read through the code of the service.
And generally speaking, Google services are fairly like homogenous.
There are accepted ways of writing applications such that you
know roughly your way around anything that comes your way.
I'm not sure how it is now.
I'm imagining like my ex colleagues at Google
laughing at me now because things have really changed.
But back in the days, it's kind of how it was.
And it wasn't perfect.
There was certainly edge cases and weirdness and, you know, been
on call for plenty of things that made no sense at the time.
But the broad idea was there was a process you went through to get
SRE support, and I did two or three of those while I was there.
They're very intense, like, they take quite a long time.
You end up iterating with the team that you're working with.
Like, it's not a yes no process.
It's usually a It's almost like a car MOT where it's
like, these things need fixing before we'll support you.
But if you do fix them and we'll work with you, then we
will onboard you as a, as a product we're on call for.
I never experienced the pager going back the other way.
So this is referred to as taking the pager and giving back the pager.
So I've never seen The page of being given back.
I've, I've, I'm not sure I like the threats of it, to be honest.
I know some teams threatened it more than others as a way to kind of
get teams to pay attention and do things that they want to prioritize.
I didn't particularly like that practice.
It felt way too adversarial to me.
I think it's good to build up trust with the teams that you work with.
One of the pain points on some of the products I was
working on was they, they didn't get full time dev support.
So backup and auth at the time.
There were long periods of time where they were considered, you know,
maintenance projects and the devs were working on something new and exciting.
So it was sometimes very difficult to get dev.
Time and buy into to work on stuff that needed it from
reading the book, a lot of companies, the thing I saw that they did
wrong was they're like, Oh, look, we renamed our sysadmins to DevOps.
And now we renamed them from DevOps to SRE.
And I'm like, you didn't change.
You didn't give them any power.
You didn't give them any authority over what the devs were
shipping over the products, what the services they were running.
They were just on call for it and it was just another wall between dev and ops.
And the thing that I always pointed out to
people when they're like, we're doing SRE now.
I'm like, what power does SRE have?
Can SRE say no?
If they can't say no, it's not SRE.
It's just sparkly dev ops.
And that was always the point where I, I always push
back on companies that were like, we have SRE now.
I'm like, if you're doing this to pay them more money.
Fine.
That you should pay them more money.
They're on call.
They're doing stuff.
Or again, like give them trade offs, right?
Like give them some time off at Disney plus.
Like we, every on call shift we had, we get a day off.
So it's like we would rotate on Thursdays and we'd get Friday off.
And it was just like part of our rotation.
We're like, Hey, you take Friday off.
If you were on call, I was an
engineer at the wrong company.
In
that time off, like it's.
It's not a complete, you know, one to one sort
of like, we didn't, we didn't track it as well.
It sounds like you did.
And we were on call over overnights or whatever, but it was
like, Hey, if I'm on call, I know that I get the Friday off.
I get a three day weekend.
And that three day weekend is really helpful to get stuff
done, to be able to go out and not carry my laptop with
me and go do some stuff during the day or something.
That stuff really does help.
Do you feel like you ever automated?
yourself out of having good documentation
or is Google really good at documentation?
Because I think like sometimes when you're building infrastructure
or kind of in the, I won't say SRE because I wasn't an SRE, but
kind of like in the releasing or just kind of infrastructure and
keeping everything up and going, automation is so heavily focused
on that people forget to actually document it and teach others well.
So like, do you think that with that push of automation, do you
feel like they still balance it out really well with documentation?
There was a meme at the time, and I'm sure it still exists,
where some documentation would refer to as strawberries, and it
harked back to a photograph from one of the canteens at Google.
I forget what it was now, but there was something
labeled as strawberries, and they weren't.
It was like pears or oranges or something like that.
The meme was like, you know, this documentation is a strawberry
because, you know, it's outdated now or things have changed.
Google's not immune to these things.
Like the, there's plenty of outdated documentation
and there's plenty of situations where.
You'll be following some docs and something doesn't
work because the world has shifted from under you.
I'd say it's probably better than most companies I've worked at, but
there was still, you probably wouldn't go like more than a couple of weeks
without finding some bad documentation somewhere, but there really was.
It was considered important and if you found outdated documentation and you
said to your boss or your team, I'm going to take a day and it should really
go over this and make sure it's right, then people generally didn't argue.
So I think the culture around documentation was
really good, but the actual documentation itself.
Sometimes not great.
I think that's kind of like as good as you're going to get though,
because the world literally does change underneath your feet, you know?
So I think that's actually like really cool to hear that they
incentivized almost documentation and gave you the space to do it.
Because I think a lot of times when engineers are trying to get promoted
or you come in new to the engineering role, there's no incentive to do.
The maintenance work or, you know, fixing something that's important for
security or documenting things like there are a lot of important jobs that
we need to do outside of writing code and none of those are incentivized.
So trying to get people to, like, do, like, the hygiene of things is so
hard because it's not incentivized towards promotion or getting leadership
to look at you and but they're very important parts of being an engineer.
So that's pretty cool.
Yeah, no, I agree.
At the time I was there, there was this attempt in, like, shift of
mindset on what work counts towards promotion and how it's weighted.
And, uh, the internal line was, we need more of a
focus on landing projects than launching projects.
And I thought that was quite good, like, It's
pithy and nice and it evokes a good, a good visual.
I don't know like how well that went.
I, I, I think I left shortly after that was a big thing and becoming a thing.
I've talked to friends I have still there probably a few years
ago at this point and I'm led to believe things have really
shifted for the better in that direction, so I'm optimistic.
I think it's so rad that you did SRE and you were so,
like, back in, deeply technical, but your blog just reads
as the most warm and fuzzy thing that I've ever read.
I felt like I almost got to know, like, a little bit just with, like, your
little, like, one paragraph, and it's condensed in, like, what, four sentences?
But it's, like, I just knew you were nice from this paragraph.
Like, I just felt warm and fuzzy from four sentences.
And usually like, I think we get this kind of like rep that
engineers are like not warm and fuzzy and that they don't like
people, but that's like the warmest fuzziest thing I've ever read.
I also don't like wasps and I buy way too many more books than I can read.
I was like, me and Sam are gonna be friends.
Like
. I, um, no thank you very much.
I, I also, the
buttons work on your like , sorry, I had to fan girl over that.
type in Moo, MOO.
Oh
my gosh.
There's a whole bunch of Easter eggs
in there.
Oh boy, that is absolutely ridiculous.
Obviously it's a terrible audio show.
If you go to samhu.
dev, Sam's websites, he has a little keyboard on top, a
little like six key keypad, and in the buttons you can
hover over them, and then you can also click on them.
And if you type M O O, I'm going to let you
go to his website and figure out what happens.
This is the
coolest personal website.
Like you do this in your spare time, dude?
Like what do you mean?
Yeah.
Like mine's is so ugly.
Like this is so great.
Wait, before, before we dive into that completely, you
went from SRE at Google, then you went to the Android team.
It sounds like you're writing a lot of Java.
Like in general, was this, yeah, it sounds like a lot of Java here.
I'm a Java apologist.
I think I, I still think Java is a nice language let down by a bad tool set.
That's what I normally say.
Ooh, I like
that take.
That is, that is the, that is like the mid take on Java.
It's not really spicy.
It's like, yeah, no, it sounds about right.
All the other JVM type languages try to improve
the tools more than the language, right?
Like the Kotlins, the Scalas, the other things around it seem
like the toolchains are better, or at least more opinionated.
They're more Ruby on Rails.
Yeah, the toolchains get better.
I think the languages do try and work around some of the shortcomings of Java.
Like Kotlin, for example, is far more terse, and it's
got this whole co routine focus that Java doesn't
have.
Kotlin's like, Java got jealous of Python.
It was like, well, we're gonna do that too.
Okay, like I was kind of let down by Kotlin though.
And it's not that I don't like it, but I love Java and I love Python, right?
And I felt like Kotlin was going to be like, if Java and
Python had a baby and I got like readable, like easier Java.
And I did not.
I got a blob.
Like, it's, the structure is one of the best parts about
Java and I just felt like Kotlin, when you try to read,
like, a big repository of Kotlin, it's just a blob.
I unfortunately have a biased opinion of Kotlin, because I worked on
the Kotlin support in Android Studio, and it was, Nightmarishly hard.
Yeah, it was really challenging.
So, so, I have never used Kotlin in like a fun
context where like I'm playing with it for the joy.
It's always been, you know, incremental
compilation produces completely invalid bytecode.
What am I gonna do, you know?
It's, uh, it's unfortunate I can't have an unbiased opinion of it.
But from what I saw, like, the goals looked pretty noble.
I think at the time, at least, anyway, I remember Java 8 coming out,
and there was this, like, resurgence, I think, in the syntax of Java.
Like, they, they, I think Java 8 was the first time they
really started to try and make the syntax into more modern.
So I think they got the, they got lambdas for the first time, like
actual like nice usable lambdas in Java 8, if I remember correctly.
And then Java 9, they brought the module system in, which broke a whole
bunch of stuff, but it was in service of trying to make things nicer.
And since then, they seem to have really been putting a huge
amount of effort into making it a much nicer language to use.
And I think of a lot of people that tried Java back in like the 1.
7 days, tried it now, they'd probably find it a lot more pleasant.
I don't think it's got the same velocity problems that it used to have.
I think it probably is 2 OOP.
Like, it would be nice if, like, bare functions were a thing and stuff
like that, but, uh, broadly speaking, I think it's a very nice language.
We need Charlie Marsh to come along and build
UV for Java and just make everything nice.
I think it's gotten better, but I think sometimes Java is very much
part of like the old club of just like suffering, you know, and
people almost want to like, they feel like you have to suffer to hang.
The JVM is something I could say nasty things about.
It's a remarkable piece of technology, but care
and feeding of a JVM in a production environment.
At least way back,
it was really challenging.
From the Ops SRE side, I hate JVM languages because of the JVM, right?
It's just like, oh, I don't care how nice
that language is to write, I hate managing it.
It's a heavy beast.
It does a lot.
It's nicer now with G1.
Like the G1 garbage collector, they, I think, made a good set
of trade offs with it, where there's way fewer knobs to tune.
Tuning the concurrent mark and sweep garbage collector was very, very hard.
It was much more of an art than a science.
You had so many parameters to worry about.
Whereas they released G1.
I think the G1 is still the default garbage
collector, but you get essentially one parameter.
And it is how, how long is the maximum pause your application can, can do.
And you say like 200 milliseconds, and then
it tunes everything around that for you.
So it's a much nicer interface, much nicer
to care for and look after in production.
I just don't know why they don't teach more about
JVMs and garbage collectors in college, you know?
Because I feel like they teach you how to code and use Java, but
like, none of the stuff that you need to know for production.
And I'm like, what's the point of paying 40, 000 for I would
love a, a like College level debugging course, right?
Like I would, like, I would get a whole degree, like,
like, give me GDB, give me S trace, give me logs
and stuff.
Like, cause that's like, you pay like so much
money and they're like, this is the fun parts.
And that's like, not even the stuff that you do every day.
Majority of the time, like nobody's writing
a brand new application from scratch.
If anyone
is listening to this debugging course that I think that would be fantastic.
And that is actually something that.
People have asked me to do some things on like S Trace and various like tools.
I was like, it's not a course.
It's like, here's, here's 10 videos on like these
tools or something, but, um, I would love that.
The closest I can think of is
Brendan Gregg's book on debugging and stuff
like that, which is really like, cause he's
a world mentor.
I was, I was a reviewer for his second edition and let me tell you,
like it was, it was fantastic and way over my head in many ways.
It was like, like the bread is just like, like, I was
like, I know how to do some debugging and I don't know
how I got roped into like, Hey, we want to review this.
I'm like, I would love to review this.
I read the first one and the second one, like I'm
pointing things out and then it was just like, come back.
It's like, no, no, you're wrong because of X, Y, Z. I'm like, Oh, you're right.
Like I thought, I think of this in a different context or different use case.
And yeah, those tools are, are very specific and you
have to know exactly how to use them in the right way.
Onboarding educational material will split into like, you know, very
generic stuff that applies to every SRE then quite team specific stuff.
One of the generic courses that I love the most, and I
think like I've carried a lot from this into my life after
Google was a course given by a guy called Andrew Widdowson.
Andrew Widdowson, sorry.
And he did a How to Reverse Engineer a Production Service.
So it was for people who, it's kind of often that you've got a, an SRE
owned service that talks to a bunch of services, and not SRE owned,
it's just like the nature of things that will eventually happen.
And it's a dependent service that's causing you problems in some
way, like maybe it's not giving you the right responses back,
or maybe it's fully down and you have to try and figure out why.
So very often you'll be reverse engineering a service you've never seen
before in your life, and it'll just tell you, you know, Where do you start?
Like find the entry points, go and Google's got lots of like
Intel debugging tools that are not, not available to the public.
So it's like a masterclass on using those as well.
Really, really thoughtful and methodical.
And I've taken a whole bunch of the.
Lessons from it with me into, into the future.
Like I always now look for entry points.
Like that's the first thing I do.
It's like, how does the, the code path get into this section?
That's not doing the right thing.
Did you go from Google to build a mess, build a base?
Oh, buddy base.
No, no.
But this is something else we could probably talk about that I
have opinions on is my, I have a very fragmented career history.
Like I, I'm one of these people that, um, I think a lot
of tech influencers would say, I'd just been that CV.
He's been worked at too many places.
I did a little misadventure through FinTech
and FinTech just wasn't really my thing.
Thing, I think I, I got into it for like quite idealistic reasons.
I think ultimately day to day, you're not really helping users financially.
Like, I think there's a, I don't know, it's not that they're like
doing like horrible nefarious things, but it's not as focused
on helping users succeed financially as what I wanted it to be.
So I worked for a couple of banks in the
UK and then I came to BuddyBase after that.
So BuddyBase is a, like a low code app building platform and it's self hostable.
So the idea is.
If you want to build internal tools, like you say, you want to like
put a form in front of something or you want to connect a couple of
databases together and automate certain things in like a, almost like an
IFTTT style way, like thing happens over here, trigger thing over there.
BuddyBase has tools for doing all that stuff.
So you, you spin it up in your own VPC, connect it to your
internal databases without exposing anything to the internet.
You can build forms, you can build applications,
you can put graphs and stuff on it.
It's got like a, not quite like drag and drop, but fairly
close to like a, a WYSIWYG editor for, for patching things
together and setting up triggers and stuff like that.
I'm, I'm just still, uh, getting over the, if this than that
reference, because I don't know how many of our listeners even
know about, like, it's not a, it's not a big thing anymore.
Right.
Cause like you, there's like the really old school, like YouTube
pipes folks that are like, Oh, we did it first over here.
And, and there's a few things in, but like, if this and that, I. feel like
had a very big moment in this, like, Oh, look at all this cool stuff we did.
And then things like Zapier and other things kind
of came in and were like, we do it more of them.
And we give you more flexibility.
If that's not was very, I don't know, user friendly in a lot of ways.
It was very big.
I remember the, everything in the UI was big, it was very big buttons.
Yeah.
Back to BuddyBase.
Like.
You're doing back end there?
Mostly, yes.
So we are entirely TypeScript.
So TypeScript in the back end, TypeScript
in the front end, Svelte in the front end.
I mostly work on Any of the backend end points that the front end will talk to.
And then also kind of care and feeding of our Kubernetes cluster.
What does TypeScript on the backend look like?
A lot like TypeScript on the front end.
It's a, you know, there are certainly some differences
to be aware of, but broadly speaking, it's, it's
just TypeScript, you know, it's nothing tremendously
special.
I mean, the only times that I've really written
TypeScript was for like the AWS CDK and for front ends.
And so like, those are like.
Automation and then the front end stuff, is this like, do you run it with NPM?
Like, is this like a Node app that runs like a TypeScript?
Pretty much, yeah, it's,
it's, it's runs just Node.
js.
You spin up some, uh, we use Koa.
Imagine kind of like Flask, I guess, in Python, where you can
create endpoints and you can have middleware and things like that.
And then you define all of your endpoints as just bare
functions that take a request and return a response.
Is it as frustrating as Flask?
Because I was so excited to be able to use
Python to make like websites and web things.
And then it just felt like beating your head against the wall over and over.
It's one of those technologies, I think,
that it looks amazing on the homepage, right?
Like, oh, look how easy it is to get an endpoint up and running.
And then you spend more and more time with it and you realize, oh
no, I need to, I need to start like moving things in separate files.
And I need to start having middleware that attaches
to everything and no shade on Flask at all.
I think Flask is fantastic, but I think it does get to a certain
point where you start to have problems that You know, you may
not have initially anticipated having, and then you have to
start to create solutions that wrap around things in Flask.
And then you get to the point eventually,
it's like, damn, I should have used Django.
I love Python, and Flask is, I don't know, it's cool, but
it, the more complex it gets, you're just like, but why?
Yeah, Flask is in that weird middle ground
where it doesn't handle enough things.
It's not quite Ruby on Rails, right?
It's not very opinionated, but it's not low level enough like, like Django.
And so you're like, oh, I want to do this weird escape.
Like, yeah, you just got to go.
do your own thing.
I think it's the gluing it together part that is a pain.
You have to make all these different, like, functions
and then trying to make sure that it all works together
and then you put in the CSS and it's just, I don't know.
I guess that's all for now and I'm just a back end person also, but.
For me it's the feeling that there is almost certainly a
correct way the community has landed on for doing this, but
it's not very exposed to me and I have to either come up
with it or go and read other people's code to figure it out.
You, you mentioned briefly there that the backend
you have, you're managing Kubernetes clusters.
Yeah.
It's a self hosted thing, but you're like, here, go run it on Kubernetes.
So there's a cloud offering, right?
So, uh, you can either self host it if you want to.
And I think most of our customers who are using it kind of commercially do that.
And then if there is a cloud offering that if you
just want to try it, or you have a use case where.
You don't really need to self host it, then we can host it for you.
So we have like a multi tenant deployment of BuddyBase that runs in Kubernetes.
And that's in EKS.
So it's a reasonably like vanilla Kubernetes cluster.
We're not doing anything tremendously fancy with it.
It's just BuddyBase itself is a helm.
We have a helm chart for BuddyBase that you can
go and spin up in your own Kubernetes cluster.
We just run that in our production Kubernetes cluster.
As someone who came from SRE at Google, not necessarily on the
Borg team, but you had infrastructure, internal infrastructure,
probably dealing with a little bit of Borg here and there.
And then going on years later, dealing with Kubernetes.
What's that, what's the difference look like for you?
Using it's very
different.
So I, I, I remember how it felt to use Borg.
Uh, when I got to Google and I, you know, I'm in a very
established team, Android's been there for a long time.
You know, there are ways of doing things already, spinning up a new
service, like scaling things up and down, releasing new config changes.
These are all solved problems at this point, that there are kind
of push button solutions for all those things, or, you know,
make a commit with very well known, well structured changes.
I think I'm a Kubernetes apologist as well, I guess.
I think a lot of, it's very fun to hate on Kubernetes.
People love doing it.
Uh, I'm one of these people who I think Kubernetes
is actually really, really good and I like using it.
I started using it, uh, actually, oh man, I forgot, I, I wrote a
Nebula between the FinTech and the BuddyBase sections of my career.
So Nebula, the video streaming platform, it was my first exposure to Kubernetes.
So before I joined Nebula, I bought A couple of Kubernetes books.
Uh, I'm like trying to eyeball them on my shelf, but I can't see them.
So I, it's like Kubernetes best practices or something like that.
Or Kubernetes up and running, I think was one of them.
That was Kelsey's.
Yeah.
First book.
Yeah.
Kelsey's.
Yeah.
I, he was,
I think he was your first guest.
Yeah.
Fantastic.
Really great book.
Like I feel like had I not read that book and I think a lot of the
problems people that complain about Kubernetes might be having here is that
they've not got the grounding in how you're meant to think about and use
Kubernetes, like the importance of labels and selectors and things like that.
It's a very unusual concept.
If you've not come across it before, I think people can get
themselves tied in knots when they're not thinking about it correctly.
It does.
the educational journey in the way that I like to do it as
well, which is you start at the very basics and you build on
top and you build on top until you get to realistic scenarios.
Actually, I have a bare metal cluster running over in the corner
there in a little server rack, so just like a few Raspberry Pis
kind of wired together as a Kubernetes cluster and I I've been
kind of caring for that for the last like six or seven years now.
I keep it up to date.
I run it as like a high availability cluster, even though it doesn't need to be.
So I get the practice in doing those things.
It's been really fun, a really good learning journey.
All that kind of translates now to what I'm doing at BuddyBase.
So we have an EKS cluster, lots of things managed for us and things like that.
But generally I think companies of all sizes can benefit from Kubernetes.
Like the line people like to use is, you know, you are not Google.
You don't have Google size problems.
You don't need Google size solutions.
I think to missing the point a little bit, I think Kubernetes has
become this shared platform for deploying software where if you read
Kubernetes up, up and running, you spend the three or four days.
It might take an effort to read that and
go through it and go through the examples.
You can now benefit from a massive community of solutions on
top of kubernetes, like you'd get horizontal auto scaling for
free, you can use tools like lens to get this like incredible
dashboard into your cluster for essentially no effort whatsoever.
There's so much stuff out that people have built.
I think it is probably one of the highest velocity
choices you can make early on in your startup journey.
I feel the same way about like rails, right?
Because people are like, Oh, well, like, do you need rails?
Like, I don't know.
What are you trying to do?
It's like, it's a very opinionated tool set, but
the benefit usually comes from the ecosystem.
Not the tool itself, right?
And when I was at Amazon and ECS kept saying, like, we're going
to try to compete with Kubernetes, I'm like, no way you're
not like it's don't build the Kubernetes features into ECS.
And it doesn't matter if you have them or not,
because you don't have the ecosystem, right?
Like all of the tooling isn't there for ECS.
It's there for Kubernetes and you're not going to compete with the rest of
the internet's building for this single platform that can be run anywhere.
And so, yeah, I very much agree that like the benefits
of Kubernetes is not necessarily in Kubernetes itself.
It is in being able to learn and onboard people, being able to
hire people that's that work on it and are experienced with it.
And just like, oh, you know how Kubernetes works.
We might use it a little different.
We don't like Helm charts or we do some other like deployment,
whatever, like a lot of things that people do that can be different.
But the basic fundamentals of it, like, oh, well, there's a
controller, there's a CRD, there's some storage, whatever.
Like I can dig into it.
It's like.
The first time that people were going from BSD to Linux, right?
And they're like, Oh, well, like this Linux is
more of a standardized thing in the community.
But this Unix thing is, is enterprise.
It's the reliable thing.
Like everyone's going to do that.
But then like the tools work different and like the philosophies
underneath on those operating systems were different and.
The first time I logged into a BSD system, I
was like, I don't know anything that I'm doing.
Like, none of this makes any sense to me anymore.
Like, yeah, I'm in a terminal, but the
commands don't do what I thought they would do.
And just that, like, mindset of, like, I know how this fundamentally
should work, the philosophies behind it, even if it's used in the
wrong ways, still helps me accelerate being able to be effective.
I think that's also, like, Partially, like when you get something
in open source community and it's well used, like in multiple
places, the more people use things, they're going to create open
source projects to make their problems that they're having easier.
And I think Kubernetes really benefits from that, like the
whole community and the fact that people are using it at like
Google, but they're also using it at multiple other companies.
People are going to build, like they're going to write books, they're
going to do documentation, they're going to build tools to use that.
And that's.
Kind of like hopefully what we, why we continue to fund open source
and, you know, contribute to it because it makes it so much better.
And sometimes people see the surface of
Kubernetes, like this is so big, it's too complex.
And I'm like, do you know the flags for curl?
Have you done looked at the curl man page lately?
Like that is a crazy complex tool that you have
to know 10 percent of right to be effective.
Like you can do most of what you want with a little bit of that tool.
Go to FFM peg, right?
Like this.
I have to Google that every time.
I'm like, I don't know what, what I'm doing next.
Right?
Let me, let me find the.
Incantation for FFmpeg.
Those are incredibly complex tools that are also used at large companies
and they solve big problems, but I only need a little sliver of it.
And you don't need, you don't have to use the
entire surface area to be effective with it.
I feel like most of the time you only use, like, even with like Linux, there's,
I think I've taken the Linux exam twice and you only use like 10 commands,
go look at everyone's get history.
Like you can run that, like the top commands and they're like.
Eight outta 10 are the same.
50% get status
over and over again.
yeah.
Ls, cd, get status?
Yeah, pretty much.
So, so I, I have a, a degree in cs, um, from the
sixth West University in the UK doesn't exist anymore.
Um, , one of the things I love telling people, one of assignments where
school, one of the isli I got given was past the MAN page of LS and find out.
Every letter of the alphabet not used as a flag.
And I can't remember what the answer is now, but it's very few.
Like almost every single letter of the alphabet is used in LS.
That's a really good activity though, because
then you have to figure out what is used.
That's genius.
Once command lines have minus like dash letter and also
plus letter, like, you know, you're in some trouble, right?
Like once you're like, I'm in the plus flags, this is, this is a bad idea.
I'm pretty sure FFmpeg is, it's just a programming language now.
It's not fine.
I'm sure it's turning complete and you could do whatever you want.
When did you start doing animations?
What got you into that?
Beginning of 2023.
So, so I had been blogging for a long time before that.
I've written some moderately successful pieces here and
there, featured in some newsletters and things like that.
But broadly I'd not, I'd not been met with any success when I was blogging.
It's still doing it mostly for me, mostly for Solidifying
the things that I had learnt, uh, I find the process
of writing really does highlight things you don't know.
Like, things you thought you know and you don't know, very, very quickly.
So it's been something I've done for, uh, well over ten years.
And then I got really into the writing of, and I'm
gonna butcher his name, but Bartosz Cieszynowski.
Uh, the guy who does the mechanical watch post, he Recently released the
post on moons, he does these incredible, usually like physics based, so he
did the physics of bicycles and the, uh, the, uh, GPS, and these are all
beautiful, 3D visualized, you can interact with every piece of it, he did once
a mechanical watch one, and the internal combustion engine ones, like he's 3D
modeled an entire mechanical watch, and an entire internal combustion engine,
And you can see these exploded views and all the, the parts are color coded.
And I was immediately in love with this.
I thought it was brilliant.
I, you know, I went through all of the combustion engine
mechanical watch, his one on sound, his one on the
moon's one most recently, just couldn't get enough of it.
I thought the style was brilliant.
And I thought to myself, you know, what if that but programming?
Like, like what is that?
But.
Algorithms, uh, or data structures, or whatever.
And I thought to myself, I'm gonna have a go.
I know that I'm not gonna get anywhere near to the quality that he's done.
In my opinion, I think he is probably the most impressive
content creator in like the written form on the internet.
I don't think anyone comes close to him.
He's so, so good at what he does.
So yeah, I didn't want to do 3D, I didn't think I needed to do 3D, partly
because I didn't think I could, I didn't, I've not really done 3D visual
programming before, 2D is way easier to reason about, and I think, and I still
think this, it's not just a cop out because I don't want to do 3D, I think
you can probably Visualize most computer science ideas in two dimensions.
Unless the things that are just explicitly three dimensions.
So I'm sticking with it for the time being and getting better at it as I go.
And you'll notice, when you, if you look at like, uh, a piece by Bartosz
and a piece by me side by side, like it's, it's almost plagiarism.
Like just the style.
It's inspiration.
Yeah, I guess.
Right.
You know, good artist deal and all that stuff, but except for DeepSeek, right?
That's not, I'm sorry.
We
knew he was gonna throw shade in there somewhere.
Like,
we not allowed for some people, I guess.
I was listening to your episode of Crystal from from the last week.
Just before this.
Yeah.
But no, like I very similar, you know, you know, use similar use of colors,
but I essentially took all the parts of his work that worked well for me.
And I tried to apply them to something in computer science slash programming.
And I settled on load balancing for the first one.
It's a topic I knew fairly well.
I had a clear picture in my mind for how I would visualize it.
And this is a recurring theme.
I tend to think very visually about these things anyway.
And I never go as far as saying I'm a visual learner,
because some people get very upset with that.
Some people are like, there's no such thing as visual learning.
I am so a
visual learner.
That like, I feel like if you had like taught all of my CS classes, I would be
like, Even better of an engineer because when I you're sitting here saying this
other guy is like a better content creator And I've never seen his work so I
can't like say but like your visuals are beautiful like they speak to exactly
how my brain works and I think that the way that you did this is so rad because
There's so many kids that don't like I have very like Neurodivergent children.
I'm definitely neuro spacey and we go through our whole lives thinking that
we're dumb or not as smart as other people and then when you see the way
that you've broken these things down, they speak so like heavily to my brain.
And I just wonder like if more people discuss topics in this way, like how
many other people would we be able to bring into different just career forms?
You know, like people were like, well, math and engineering or they're so hard.
And like, I remember I was in a parent school.
class and it was for like helping your kids go to college and this
guy was like this woman was like oh my son wants to be go to school
to be a developer and he's like and this guy comes in he's like i'm
an engineer and if he's not good at math and he doesn't get good
grades tell him not to do it and i was like bro like who are you
like oh yeah i'll go do that right now yeah you
know what like who are you the nerve it cracks him because if you go
on an engineering floor we've all got like six different things of
caffeine and we're totally like all like just as neuro spicy, you know?
And I feel like if people, uh, learned how to kind of teach in
the manner that you are, and I hope more people do it, and I'm
really excited to find your stuff because it's like so many, like
math is instructions, programming are, is instructions, right?
And especially kids with neural spicy brains
who want to know how everything works.
It's amazing how their brains would actually be like amazing
for this job, but it's just not being taught in the right way.
And it's so exciting to see somebody teach in a way that
will help other people get these complex or, you know, I
guess dense information, but it's really not that dense.
It's really a cool puzzle that you're figuring out, but it depends
on who is giving you the information and the way that you do it.
It's just, it's so cool and impressive.
Thank you very much.
Everything you said just like hit me in my soul.
Like I completely agree.
I, so I have a neurodivergent kids as well.
So I've got two autistic little boys and I think about this a lot.
Like, uh,
Me too.
Ah, amazing!
So I have a five year old and a four year old, and I didn't know
anything about autism before having my two diagnosed, and it's
been a real journey, and I think I had an idea of what it meant,
and now I'm kind of learning and going through it for real.
It's, it's very much not what I imagined it
was, but I, I do worry a lot about education.
So we, we send them both to a special needs school that's
kind of specifically for autistic kids, and they're thriving.
My own experience of education wasn't wonderful.
I, I did good.
I did okay.
But I never really connected with it very well.
And there was something when I read some of Bartosz's work, similar to you,
I was like, this is how everything should be presented, at least to me.
Like, specifically me.
I would love everything to be presented in this way.
That'd be great.
And, but I construct these things in my head anyway, as I am learning.
Like, I'm learning about load balancers.
I am, in my head, constructing this idea of like, okay, Here's a set of five
servers and it's round robining between them and I can kind of see it happening.
Yes.
I have a mental map map of everything.
I have to be able and I need to, I need so much information in the front
part of learning something because I need to be able to build that.
Yeah.
It, it, it's, it's one of the most fun parts of the process.
It's getting, getting what's in here onto the screen in a way that's legible.
can be built up from the very basics.
And that's another thing I think is very important.
And you'll see it kind of repeated throughout my work is I try and
start with the absolute simplest, most stripped down example possible.
This is going to seem personal, but do you think that your experience
with your children has made you a better teacher and a better engineer?
And I'm asking this question because it has been such a journey.
I have three little boys.
I have a five year old, a seven year old and a 10
year old and getting my oldest, the proper education.
Like has been such a struggle and we've had to switch schools and I
was like determined that he would never feel like he was dumb or bad
at something because I went through my whole life feeling like I wasn't
like, good at school or smart, even though I did well, but it like caused
me so much anxiety because I had to like make myself focus, you know?
So like, Learning, and then I just was like, I always knew I had
ADHD, but I didn't know I had autism until like going through all
the stuff with him, you know, cause girls present in a different way.
So just like learning how to teach him for his brain and learning how
to break down concepts and kind of like learn more about how brains
work, I feel like has made me a better engineer and problem solver.
But it's like amazing to see the way that you break down all these problems.
Do you think that influenced like how you teach or like.
your empathy for how to break things down for people's brains?
I'm not sure.
I've never thought about it, and I was thinking
about it a little bit as you were talking.
I'd say it's probably yes, but I couldn't pinpoint
exactly how without thinking about it a bit more.
I think it's certainly given me an appreciation
for a wider variety of human experience, I suppose.
Like, uh, early on when I, we had just gotten our youngest diagnosed.
I remember reaching out to an autistic adult
who had autistic kids to have a conversation.
Because we were wondering, is special needs school the right choice?
Like, I think reasonable people can pitch it both ways.
Like, do you want your child to feel different?
Like, there's different ways of feeling different as well.
So either you go to a mainstream school and then you feel
different because you're different to the mainstream kids.
Or you go to a special needs school and then you're different
because you're in a different school to the other kids.
So.
I was confused about like, like I didn't want my kids to get to like
15 or whatever and hate the decisions that I made for them when they
were younger, like I wanted to try and make the best decisions that I
could and I thought that, you know, uh, the best way of doing that is
talking to people who have been through this and, and getting their
opinion and I remember saying to him, Oh, I don't really know any.
Autistic people.
And he's like, yeah, you do.
Yeah.
You know what?
It's amazing.
Like now, like all my friends are getting like themselves
diagnosed and we're like, dude, I was like attacked.
I was filling out that form for my kid doing his diagnosis.
I was like, what do you mean?
Do you mean this has been my whole life?
It's wild because my children are so different.
My oldest goes to, uh, I guess you say like a nor a regular
public school, but he is in like a res extended resource room.
But my two youngest are just in general ed, but the way that
their neurospaciness manifests are completely different.
One does great in school, and like, Loses all of his, I don't know,
containment at home because that's where he feels safe and the other
one is opposite and is totally chill, does his own thing and builds
things in his room at home and absolutely hates school, like, but it's,
it's wild how they can be in the same family and the same two parents
and they have completely different, like, manifestations of that.
So it's like, it just, It's made me really appreciate how our brains work and
how, and it's so funny, like, do you remember Justin Stanley from AWS, Justin?
No.
Oh, so like just you or him or like my friend, one
of my favorite engineers at work, my friend Ben.
Y'all will talk about stuff that you did as children, like, like, Ben
would, he built a, like, a Game Boy app thing on, like, a calculator
because his parents wouldn't let him have it, and Justin was saying
how he built a rocket, and it's like, these things that people
say that little boys were getting in trouble, they were so smart.
You know what I mean?
They were problem solving, and if you look back,
Einstein, um, Sebastian Bach, like, Beethoven, so many
of like, the revolutionary people in history had autism.
I brought my son like, a whole stack of books of like, people in history
who had had autism when he first got diagnosed, and it's like, If you
look at the amount of people that were like, just game changers and how
we interact with music and engineering and so many things, it's just,
and then you go and you work at a big tech company and so many, like,
it's just so funny that we've been there the whole time and nobody knew.
Have you ever read the book Life Animated?
It's about a parent who had an autistic kid.
And, and I, I read that one while I was working at Disney animation.
Cause it's a lot about how he loved 2d animation and he
related all of his like feelings through the characters.
And I remember I was audio booking that back and forth in my commutes to Disney.
And I was a new parent at the time.
And I'm just bawling like every day, I'm just like, just sobbing.
Just like, this is like so hard to voice like.
How things are important to different people in different ways.
And so I do recommend it, but, uh, just be prepared
that it might, uh, might cause some feelings.
Look, I'm like so tough, but when it comes to my kids, I'm warm and fuzzy.
And I'll like rip your hole.
Like I'm the mom that like, we'll go up there and like, they know me so well.
Like I know everybody at the department.
So like, I will totally be like, even the Disney shorts make me cry.
Like those like my favorite.
Circling back a little bit on, on just education side of things in the
animation, like you said, you were sitting on this load balancing thing.
Cause you're like, I know how to represent this.
I know how to visualize this, but then you had to like do it.
Uh, I remember back in the, the like early two
thousands nineties, like the internet is multimedia.
Like there was this huge, like we are going to do like video and text.
And it was like a big deal that you could do both.
And I feel like.
In this case, like this represents what's
possible on something on the internet, right?
Like you could have animations right alongside the text and it's
interactive and you can change how you're, you know, interacting with it.
And, and you've done what, seven animations now?
What would you have done different?
What did you learn through this process?
And what are you excited about doing next?
So
I've open sourced all of the visualization.
code behind these posts.
They live in a separate repo.
I don't, I, so I published them under an MIT license and
I very explicitly don't publish the, the words under an
MIT license just so I have something protected in it.
But I'm very happy for people to use the visualization
code and people have done, which is really cool.
If you go back and you look at the load balancing post though.
So I think it.
It really highlights the kind of Luddite I was when I went into
this, where it's all like, raw JS that there's absolutely no
like, package management, I don't use anything from NPM, I use a
couple of like, JSCDN links for like, Plotly and things like that.
But it's put together like someone that has no idea
what they're doing, and the reason is that I didn't.
I really didn't.
And I, I ended up using Pixie.
js, which is a, it pitches itself as like an HTML5 game making library?
So it's a lot of resources for drawing to canvases and
doing animations and things like that, handling input.
It seemed pretty good for my needs.
I don't use it anymore because it's, it's fairly heavy.
So it includes everything you'd need to make a game.
You know, you've got like audio and video handling stuff in there.
You've got input handling, lots of different things like
shaders and stuff like that, that I don't use and don't need.
So it was pretty heavy.
So I think it, it comes to like over half a Meg.
I think possibly even more than Plotly was like a megabyte of JS.
And I, I spoke to Scott Hanselman on his podcast, and
he was going through the network requests on my page.
He's like, that's like over a megabyte you've loaded there.
I'm so sorry.
I didn't mean to.
Scott calling you out.
Yeah, just getting shredded by Scott.
The moral of the story is I didn't know what I was doing.
I learned everything I needed to know to do some 2D
animations that kind of worked well enough on the page.
And I think there's actually some devices it doesn't work well enough on because
I didn't realize, for example, When you draw to a canvas, I'll be using WebGL.
Uh, you can only have so many WebGL contexts on one page, and
the context is the thing that you draw to the canvas through.
It's the thing you issue commands to.
And you can only have, on some devices,
like, eight of these things at any one time.
So I think on some of my devices, as you scroll
down, some of the things just don't load.
But it's, like, reasonably old mobile
devices, and I haven't gone back and fixed it.
It works fine on Modern phones, modern browsers and things like that.
So I thought it was probably, I don't want
to touch it cause it's, it's terrible.
I honestly, if you go back and you look at the code, you really
will think I'm absolutely unable to do front end development.
I don't see any of the stuff that you see.
Like I was geeking out.
About your load balancing one and like the animation seems so full even though
it's 2d That's the first one I clicked on and I thought that was your most
recent blog And then I went through the other ones So like I totally like I
think that you have like the artist You know, like when you're an artist and
you look at the things that you do you see all the problems in it But we don't
actually see the problems in it because I didn't see that at all Like all
the things you just described I did not see any of that when I looked at it.
I thought that was like It's such a full concept for 2D, and
like, I thought that was such an awesome way to present it,
and I didn't see any of the problems that you just stated.
I think it's a blessing and a curse.
I, I think, I, I think I, I, I care very
deeply about the quality of these things.
And for example, if you open the load balancing
post on mobile, And you rotate the screen.
None of the visualizations will adapt to the new screen size.
And it kills me that that doesn't work.
And you'll notice all my later ones, that does work because I was so,
like, ashamed of myself for not making it work in the log balancing one.
I like what you said about art.
I'm, you know, very flattered.
I want to lean into that way, way more, actually.
Thinking about it.
So if you were to open the Turing Machines post, for example.
Something I did for that one.
But I wasn't sure about the time, um, so I drew
the hero image in that, in that post by hand.
So like it's, it's uh, it was done in Procreate on my
iPad over the course of, you know, several evenings.
I wouldn't say I'm, I'm, I'm a good artist.
I think it is enough his likeness that it works well as a hero image.
Some mean people on Hacker News did say some stuff about it.
I think someone was like, He looks like a weird alien.
You should just stick to Shutterstock next time or something.
And I was like, Ow.
I will.
Can you just, can you tag me wherever that is?
And I will
do the
job
for
you.
Like, the lies you tell.
Like, you are definitely an artist.
And those people are just jelly.
And also, like, I am willing to fight on the internet for you.
Like, just tag me in.
Tag me in.
I have time.
I replied to it saying, oh, um, it's, it's a little bit of a, a
little bit of an anti AI statement, a little bit, as a, as more and
more people are using AI tools to generate worse and worse content,
unfortunately, um, I want my content to really stand out as It's
kind of human, really organic, and thought about and cared about.
So I tried this new idea of hand drawing a hero
image, and I probably will do it again, I suppose.
My wife is actually a significantly better artist than I am.
If you scroll down to the bottom of the Drawing Machine
page, there's three drawings at the bottom, and she did
those, and I think they look so much better than mine.
I hadn't told her about my plan to draw the hero
image, and it was like quite late into the process.
She was like, oh wow, what are you drawing?
And I was like, Oh, I'm drawing Alan Turing for my post.
And then she was like, Oh, I'll have a go as well.
And she drew a picture of him.
It's way better than mine.
I'm like, damn it.
I'm still going to use mine because it feels, it feels kind
of cheeky to use yours, but I want to lean more into that.
I would like, and I've not said this to anyone before.
I would love if.
Every single one of my posts, I worked with an artist
and I kind of paid them to create artwork for the page.
So, I'm writing a post at the moment, uh,
very early stages on multilayer perceptrons.
So very early kind of classifier neural networks.
And I would love to commission an artist to do kind of their own style.
So the important thing behind this is it would be their style.
Artificial neurons, like actual pictures of
neurons from the brain and things like that.
Just, just Artistic stuff around the writing, and then, you know,
feature them as an artist that has done the artwork and then for every
post I get somebody else and it's in their style, so every post has
this unique feeling to it, and at the same time Using my, you know,
modest platform to try and boost people, like, other artists profiles.
I have previously put out, like, Oh, hey, I want to
commission some artwork, uh, this, this kind of thing.
And the problem is you should get hundreds of DMs from scammers and
people that are not legitimate and people that are looking to rip you off.
Uh, so I, I don't know the best way to approach.
artists, and I would love advice if you have any,
because I think it would be so glad that you don't want
to use Fiverr, because like, they just down level, like, and just are terrible,
and they've ruined so much of graphic design and animation and so many, like,
other parts, that like, I love that you actually want to work with them.
It's not that you're getting a bunch of sinkhammers,
though, but I just think that's such a, like, rad idea.
To do that, and to really.
Shores away.
I think it would probably have to be, you know, through my own
network of people, like, you know, asking friends and stuff.
Even networking on social media on, like, Blue Sky, but, like, networking
with other artists, because there's, like, an artist feed, too.
Well, one of the things that I've found is just, like, finding
the right style and experience from an artist is really,
really difficult, where it's, like, there's a bunch of amazing
artists out there that just aren't the thing I'm looking for.
And that's like, cause like the, the vibe and the, the things
that I've, I've tried to do different animation projects in the
past that were mostly just like 2d hand drawn animation things.
And every time it's like, Oh, like this isn't, it's not a comic book.
I don't want that vibe.
It's not a, you know, a dark sort of like, uh, it's like, I want.
The specific thing, and I don't know how to tell someone that
I don't have necessarily the words to be able to say, like, I
have to go look through a bunch of it and then do the work to
like, Oh, these are the five that I think look about rights.
I'm sure there's plenty of artists that can do different styles,
and maybe they just don't show that, and it's hard to, like,
Have you ever tried just looking up a video or
grabbing a book and looking at the different styles?
It might be like, you know, like a book of just different styles.
Then that way you can kind of get the vocabulary that you're looking for.
Or even if you just Google different animation styles.
Yeah.
And I mean, but like.
Again, art and animation, there's a lot of
people that are desperate for work right now.
And it is also really hard because you will get hundreds of
people that even if you say, I want this specific thing, they're
still going to come to you and, and kind of waste your time.
And so like, Oh, here's all this stuff that I did.
I'm like, that's not what I asked for at all.
There was a lot of people that were just like, Oh, you know what?
Like, you're great, but it's not what I was asking for.
And.
Thanks.
And I, I stopped asking publicly for help for those things because of
the amount of time it was taking me to like filter out all the people
and then say like, okay, well, like what's your timeline look like?
What's the price?
And it just never lined up.
The funny story about how I originally had this
idea is, um, do you know who Andy Matuszak is?
He is an education researcher, fantastic, I'll
link him for the show notes, uh, after this.
He, his, his website is beautiful, he's obviously put a lot of time into it,
and, uh, I read a post by him with the title, Why Books Don't Work, like a
fairly incendiary title, but the, the, the point is that It's unrealistic
to, to expect people to read a book and then know everything in the book.
Like it's, you know, we have so many more tools now.
And it was just, it was just speaking to me.
I was like, I completely agree.
Like it's, it's a big part of what I'm doing at the moment.
And then I clicked on some of his other posts.
And every post I opened, like, had a different theme to it.
And I was like, is this?
Is he making like a bespoke theme for every single post?
That's an amazing idea.
And then I had this idea, oh, I'd love to like commission
different artists to do work in their style for each of my posts.
But it turned out actually, uh, every single link was, was
guest posts on other websites, so he wasn't actually doing that.
So I, he's given me this idea.
Even though he's never actually done it.
That's cool, but it also just seems like it would be very
time consuming, but that's pretty cool that he also, like,
is giving people, like, a platform, but at the same time.
So Sam, what, what do you think is next?
What's the next, uh, you said you're working
on, like, a neural, neural network sort of post?
What has been occupying a lot of my time recently is,
can I do this full time, is the question on my mind.
I don't think I can do the blog posts full time.
I think, uh, so I did recently land my
first sponsor, which is a huge step for me.
Very, very grateful that, so the company approached me and we
spoke and came to a good agreement, but it, but it's not like,
it's not the kind of money that I could live off, but it's,
you know, very, very generous for the, for the work it is.
So I'm thinking to myself, like, how would I, how
would I do this full time if I really wanted to?
And I'm beginning to kind of take steps in that direction, which is why.
I've not made a lot of progress on my next post yet, so I've got some visual
stuff working, but I haven't got everything that I need, because I'm just
doing a lot of logistical stuff, like I created a company, and I, you know, I'm
speaking to accountants and things like that, all the stuff that goes around it.
And you mentioned, I think, earlier on, you know, Oh,
you're doing this in your spare time, that's incredible.
Uh, so I do four days at work, and then I'd
have Fridays for working for my own company.
So it's not quite spare time, like, I have been able to carve out real time
for it now, which I'm incredibly grateful for, uh, But yeah, it's, what's next?
I, I want to keep making the posts, I want to keep a cadence of like
two or three a year, And I think with the sponsorship and with the
extra time that should be possible, into the foreseeable future.
But I'm also thinking about other ideas surrounding that, like writing these
blog posts is one way of reaching a wide audience with educational content,
but there's lots of other things out there, like everybody at the moment is
making courses, and Very inspired by the work of people like Aaron Francis
and Josh Camot and the stuff that, like, Matt Pocock is doing through Badass.
dev and things like that, and starting to wonder, you know, Is
there space in that market anywhere for the skills that I have?
And I think there probably is, and I've not fully fleshed
out my ideas in that space yet, but it would surprise me if
in the next three years I didn't do something in that space.
And then also the most unlikely thing, I can't remember who
I was talking to recently, but having this conversation about
physical products and talking about the question, like, what if
you could hold an algorithm, what if you could physically hold.
An algorithm.
Like, how would that look?
How would it work?
And, uh, it came about, uh, I have a friend who's got a 3D printer.
And I was like, could you 3D print a Turing machine?
Like, how would that work?
How would that look?
And we started looking at, there are physical example of Turing machines
out there that you cannot, you Turn a handle and it does something, some
sort of program, and it works in the same way a Turing machine does.
They're real complex, actually.
Like, probably way more complex than we could
achieve realistically with a 3D printer.
And plus, I wouldn't want to sell 3D printed products.
Like, a lot of thought has also recently been
put into, like, what is my brand, so to speak.
My personal brand.
And I really want to lean heavily into, like, quite high quality things.
So if I did Produce something like this.
In my head it would be a like a fully like metal Turing
machine that you know has a weight to it and it feels good
and like I'm thinking like a desk toy kind of accessory thing.
So I think those are the least likely.
Dealing with physical products has a whole
bunch of challenges that I know nothing about.
But I do love the idea of like, what if you could hold a bubble sort?
Like how would that look?
What would it, how would it work?
So kind of physical manifestations of algorithms is something that's on my mind.
That's such an interesting concept.
I hope you don't let all the hurdles stop you,
because that would be such a cool concept.
The biggest hurdle right now, and I'm sure you'll appreciate this,
I'm so time poor, you know, with the kids and with the Full time work.
Um, I haven't mentioned it, but I'm also a trustee for a
charity in the local area that works with special needs kids.
Time is like unbelievably precious to me.
Sam, do we have the same
life?
I think we have the same life.
Have you seen us in the same room?
I don't think we have.
We're, it's like You've never seen Batman and like, what's his name?
Batman and Bruce Wayne at the same time.
I think we're the same.
Which one of you is Batman?
Sam, definitely Sam.
Sam, thank you so much for coming on the show and explaining to us just your,
your career and all the things you've been through, but then also the things
you're looking forward to and the animations and everyone should go to Sam who.
dev because those animations are fantastic.
You'll learn a lot of things that even, even things you didn't.
really realize you didn't know, right?
There's a lot of things that just got exposed once
you put a visualization to some of this stuff.
And I think it's fantastic.
So
I hope you have a just long career in creative teaching because you
weren't, for one, a rad human on your own, but like the fact that just this
is so creative and I hope it's the future of how we teach people things.
Oh, I really appreciate it.
Thank you so much.
I thank you so much for inviting me on.
It's been a blast talking to you both.
I know you are semi active on blue sky.
I don't know if you're on other social networks or not.
Um, we do have the FAFO dot FM blue sky account has
a starter pack of guests and you're in there already.
We have, we have a couple of people in there, so it's been fun.
We've been just adding them as we're going.
There's still, we're always a little ahead on it too.
So anyone that's on blue sky, if you are looking for some of the past guests
or whatever show you're listening to check out the FAFO dot FM accounts.
And there's a startup back there that you can find anyone.
that has a blue sky, uh, isn't.
One more thing, what you said about websites having
downtime and how like multiplayer games, that's so real.
Like, it's so true.
Like you can't have like five seconds of downtime on a website, but
fortnight will like crash itself for like eight hours until you to come back.
And then your kids are sitting there at like 3 PM.
Like they just, just, I swear to God, you're
just like the way that they will wait.
Just wait for the next, like, what is it called?
Um, episode or whatever, and they get the new downloads and just, it's amazing,
like, they could just leave up a version of, like, the old game, but they
will completely take it down for eight hours and kids will sit there and wait.
It's like, you can tell all the kids in the
neighborhood are outside because Fortnite's down.
Yeah.
That's the thing, you can go outside, it's fine.
The thing that triggered it was, like, major database upgrades.
They'd be so much easier if I could just take the website NAND for an hour,
do it, and bring it back up, rather than have to set up a logical replica.
Yeah, much, much faster, much cheaper in general.
There's this weird, this weird obsession with 100
percent uptime that I don't think is well placed.
Yeah, cause like, there's a lot of websites
that we do not need 100 percent of the time.
Also, the IRS website was down for like three days last year
when we were trying to make like the business account for
this, so like, dude, it, nothing worked for three, and it was
even out, and it was like longer than the structured downtime.
It started 24 hours early and it was out longer.
And I was like, this is actually important, like.
Things are okay.
Right.
Like it's, it's not like, obviously some things probably
weren't okay, but generally speaking, like we're all still here.
And yeah.
One last thing, cause this episode is going out at the end of
February in the first week of March, March 6th through 9th,
Autumn and I will both be at the Southern California Linux Expo.
So if anyone's listening to this, wants to
come to scale, it is in Pasadena, which.
Traditionally has had fantastic weather.
I know in January, there have been a lot of fires around the area.
Um, but as far as I know, March is still going to be a great weather to attend.
It's a community events.
Uh, I help run the cloud native day.
Autumn, you have a talk there.
Y'all scale is going to be absolutely bananas.
Like did you, have you seen the list of people going?
There's,
there's some great people.
It
is going to be like.
Not just nerd summer camp like it is going to be unhinged nerd summer camp like
It's community run.
So if you are individual if you have education or like half
price tickets bring your kids I'm bringing mine on on Saturday.
I don't know you're coming with yours
if you see a horde of children That are just like absolutely ridiculous.
It's probably like me and Justin's kids, but like mostly mine.
Like
there's, there are game nights on Saturdays.
So feel free, please come.
Um, it's again, it's, it's a, it's community run events.
It is affordable for a lot of people.
I think the full price tickets are like 120 bucks or something like that.
So, and there is sponsorships for some of that if you need it,
but there's a lot of different tracks, a lot of different tech.
It is, it is a very fun, non.
Uh, enterprise, non single vendor sort of environment to be in.
You
are doing the best sales pitch that's like, uh, the best, like.
I've been going for.
Wholesome one.
And I'm just like, I'm going to do hood rat stuff with my friends.
And then like, my kids are going to like, you're just going to
see five little boys, like running around with like stickers on
their head and stuff and like scale shirts and like raspberry 22.
I think the first one I went to a scale seven or eight, maybe.
I can get banned from my kids.
Like, do you think there's be like, don't come back with those little,
like, uh, I have, if anyone's watching the show, like,
well, no one's watching it now, but we might have clips.
Uh, there's a bunch of raspberry pies that we're giving away for
the kids for the next generation track is, is by kids for kids.
My son gave a talk a few years ago, but there's kids in the audience
and they have like, uh, like a STEM room and stuff like that, but.
People in the community donated a bunch of raspberry pies.
Thank you so much for sending them.
Um, I have been setting them up 3d printing cases, making
sure they work, getting them ready to hand out at the event.
So we will have a couple of like scavenger hunt type things
that you can get a free raspberry pie if you, if you come.
So, uh, there's a lot of cool stuff.
So, uh, please check it out.
We'll have the link in the show notes and, um, Sam, thank you again for coming.
No, thank you.
Have a good one.
Thank you for listening to this episode of Fork Around and Find Out.
If you like this show, please consider sharing it with
a friend, a coworker, a family member, or even an enemy.
However we get the word out about this show
helps it to become sustainable for the long term.
If you want to sponsor this show, please go to fafo.
fm slash sponsor and reach out to us there about what
you're interested in sponsoring and how we can help.
We hope your systems stay available and your pagers stay quiet.
We'll see you again next time.