Animating the Stack with Sam Rose

This episode is stacked with information. You could even say “full stacked.” Sam has built and run some large scale systems as a SRE at Google, now building backend services at Budibase, and he spends his free time teaching others how systems work at understandable scale. We dive into what makes Google SRE different from other companies, what it’s like to be a parent, and how Sam got started with building animations for his blog. Don’t forget to visit and check out the easter eggs he’s hidden throughout.


Show Highlights

0:00 - Intro
2:00 - Sam’s background
6:00 - How Google did SRE
15:00 - Importance of docs
19:00 - The problems with Java
26:00 - Budibase
32:00 - Borg vs Kubernetes 
39:00 - Building animations
46:00 - Being a better teacher
56:00 - Art in the age of AI
1:00:00 - What’s next

Links Referenced

• SRE book https://sre.google/books/
• Sam’s Blog https://samwho.dev/
• Budibase - https://budibase.com/
• Bartosz Ciechanowski’s website - https://ciechanow.ski/
• Life Animated book - https://www.amazon.com/Life-Animated-Sidekicks-Heroes-Autism/dp/1484741234
• Andy Matuschak’s website - https://andymatuschak.org/
• Southern California Linux Expo (SCaLE) - https://www.socallinuxexpo.org/scale/22x

Sponsor

The sponsor for this week is YOU!

Please share and rate this episode!

https://fafo.fm/sponsor

Creators & Guests

Host

Autumn Nash

Host

Justin Garrison

What is Fork Around And Find Out?

Fork Around and Find Out is your downtime from uptime. Your break from the pager, and a chance to learn from expert’s successes and failures. We cover state-of-the-art, legacy practices for building, running, and maintaining software and systems.

Sam Rose: I think companies of all sizes can benefit from Kubernetes. Like the, the line people like to use is, you know, you are not Google. You don't have Google size problems. You don't need Google size solutions.

Justin Garrison: Welcome to fork around and find out the podcast about building, running, and maintaining software and systems.

Welcome to fork around and find out the most animated infrastructure podcast on the internet. I am Justin Garrison and with me is Autumn Nash. And today's guest is Sam Rose. Welcome to the show, Sam. Hey, thank you for having me.

Autumn Nash: I've been waiting for this episode because Sam's animation fire, like I love art and technical and when that like comes together, it makes me so happy.

Justin Garrison: And I want to get into that so bad, but I want to first talk to Sam about Where you've been before, what you're doing now, because I did not know that you were doing this crazy, awesome animation as in your back end engineer, like you're just like, Oh, no, this is on the side. I don't know how to do JavaScript.

I mean, you just did a lot of JavaScript.

Sam Rose: Yeah, this surprises a bunch of people. My background is almost entirely back end infrastructure, devopsy kind of thing. So I started. my career way back in like 2012. Oh, I feel so old. Don't say it. I feel old now and I'm getting like gray bits in my beard. Like the tides are turning, but like 2012 Ruby shop doing rails and stuff like that way back when, uh, I think Ruby was kind of hitting its peak, like 2010.

It's very on brand. Yeah. 2012 Ruby rails. Everyone was doing it. Then no JS came out of nowhere and things kind of shifted. But back then, yeah, working for an ad company mostly. And then. I went from there through like an acquisition. To another company I didn't stay long at. It was a bit of a, not like hostile acquisition, but it wasn't very nice for everybody involved.

I think it was fairly clear that we were required for technology, not for people. So over the course of a year, most of it left there. But then, I ended up getting what I thought at the time was my dream job. So I worked at Google for four years. Uh, I joined as a Site Reliability Engineer on the Android team.

Stayed in that for two years and then did two years of software engineering on the Android Studio team. Afterwards, and like at the time, this was probably a little bit after what people would call the golden years of Google. Like, I think, uh, it had already begun a bit of a decline in public perception, but it was still an incredibly good place to work.

And I learned a ton of things. I'd say probably the main thing I learned after the four years there was big companies aren't really my thing. I did the initial switch away from site reliability engineering to software engineering because I'd missed writing code a bit, like as a site reliability engineer.

At least on the team I was on, the things I was doing, not an enormous number of opportunities to get to get into code and write features that users will end up seeing. And I missed doing that.

Justin Garrison: What were you reliability in, uh, on the, on Android as a, as a site? Like, cause that's, those are, that's a phone operating system.

The phone

Sam Rose: talks to a whole bunch of stuff. Uh, so at the time, the Android SRE team was split into two halves. So there was the, the Play SRE team, which took care of the Play Store and everything related to that. And then there was what was called the Cloud SRE team. And this is all ancient history now.

These things have changed substantially. But we were responsible for The phone backup system, the authentication system. So the thing that maintains your Google login all times, we were on call for the push notification system. I think by far the biggest thing we were on call for, we were on call for a couple of like the device OTA system.

The thing that told your phone it needs to update a couple of the smaller things like that. Those are kind of the main hits. The two that I focus most of my time on was the backup system and the auth system. So the team. There were kind of six of us in London and then the six over in the U. S. and people kind of specialized on each of these services because these services.

It's Google, they're huge, like, they serve hundreds of thousands, if not millions of queries a second and it's hard to specialize in all of them, so I took backup and auth. My main, I would say, like, crowning achievement while I was there was taking both of those services from two nines to four nines of reliability, uh, through, like, lots of iterative change, working with dev teams, capacity planning, tweaking code paths to be more reliable, quite a lot of, Small nitty gritty things.

So that's broadly what I was responsible for when I was at Google.

Justin Garrison: Now, I mean, usually going from 2 to 3 nines is like, we can just write some better software, right? But going from 3 to 4 is a lot bigger lifts and usually involves more process, right? Because going from 4 to 5 is definitely like the people side of things.

It's not necessarily your code is just amazing. It's, we handle this better. But you said you weren't writing a lot of code. You said you wanted to get out of SRE. Because you weren't writing code. So, uh, kind of

Sam Rose: user end, like, uh, end user features is the stuff that I was kind of interested in, rather than, like, trawling through.

Tons of logs to find some specific error that's caused by some race condition somewhere like that's fun But it's not fun six months of the year. It's kind of interesting going this like four nines to five nines It's very challenging But the fun thing for us there was there's really no point in doing it because the the telcos themselves were not five nines So there was no point being more reliable than the network.

These phones were operating on So we we never saw five nines of the goal for most of the stuff on Android

Autumn Nash: That's interesting that the phones weren't as reliable. So you had to almost kind of walk back how much you were expected to be reliable.

Sam Rose: Yeah, it was nice. It was, it was almost like a freeing thing that we had these kind of slightly lower reliability targets than some of the other core Google services.

Justin Garrison: And just being able to calculate that out of just like, Oh, the thing I rely on is not reliable. So I'm not going to try to like beat it. And people often move to the cloud and they're like, I'm getting five nines. I'm like, the service you're using is three. Good luck. Right? Like, this is not, this is not going to add up.

Right.

Autumn Nash: I don't think people realize that there's different reliability for services versus like the cloud in general. Like, I don't think people actually realize that. Can you explain what being an SRE like at Google was? Because I think like being an engineer on a release team. and building language is so different than the traditional SCE job.

So I think, like, it would be interesting to hear what, like, your SRE job was, versus being, like, doing software development, you know?

Sam Rose: The SRE job at Google, as I experienced it, and it does differ between teams. So Google, for example, If you're working on some core service that everyone depends on, so say you're in like a networking SRE or something like that, your life is very different than if you're working on like product based things like Android, which is more of a product based thing.

But there's a lot of looking into the future, so a lot of kind of capacity planning, like can we service the expected growth over the next 6, 12, 24 months. There was quite a lot of communicating with development teams, say if, if something in our monitoring starts to like kind of break down. strange or unexpected because maybe some release or something unexpected has happened.

We will kind of work with dev teams to help them understand how that was affected by changes they may have made. There was an aspect of you were encouraged to, how do they phrase it? Something like automate yourself out of a job every 18 months or something like that was one of the taglines they had.

So it's like, whatever you're doing now, there should be a percentage of your time dedicated to making sure you don't have to be doing that in 18 months time. In my experience, that was more of a, a wishful kind of goal than something that happened regularly in practice. Some teams certainly achieved it occasionally, but I think most people were still doing the same stuff 24 months after that they were doing, 18 months prior to that.

It was a lot of almost babysitting, but I mean that like, I think that has some negative connotations associated with it, but it was, it was like, there's gotta be a better word for this here. Uh, not championing, not like, You are, you are the person like responsible for this being a reliable service, right?

You are like the vanguard, you know? Like anything, like anything getting in the way of the service being as reliable as it needs to be is on you. And the way you are babysitting them,

Autumn Nash: because you're keeping it alive.

Sam Rose: For sure. Yeah, yeah. Being on call is a big part of the job as well. So, uh, on call like roughly a third of the time.

And nobody that I have worked for before or since does on call quite as well as Google does. Just in terms of sort of emotional well being, I guess. So you're never on call 24 7. There's a limit to how much on call you can do in a given amount of time. You're always paid for on call. So it's a very fair system.

Every so many hours of on call that you do, you get an hour of time off in lieu. So I would End up racking up something like 35 to 40 days worth of time off in lieu over the course of a year and just take big blocks of time. Yeah. I'm And time off, which is really nice. Yeah. It's like, I'll, I'll see you

And you have to be, you know, mindful of the team and, and what everyone else's responsibilities are, but you don't want to kind of put like furloughs on subsets of people, but, but you wouldn't

Justin Garrison: be, you said not on call over 24. Twenty four seven. So is that like just waking hours? Like you're never on call when you're supposed to be asleep?

The, the way that we did it was

Sam Rose: in the UK it was 7:00 AM 7:00 PM. during the week. So actually, no, the weekends as well. It's, it's, um, and then four hours of that 12 hour, uh, is what you're being paid for. You didn't get paid for time where you were in work. So during the week, you're getting four hours a day of time off in lieu time.

It wasn't actually, it didn't directly correlate to four hours, but then the weekend was the full 12 hour block would count as paid time. So yeah, 12 hour shifts. And then you switch over to the U S.

Autumn Nash: That's wildly fair compared to most big companies on call. And I almost think that it's like a buy in, you know?

On call sucks, but when you get the time off and paid and all of that, and you're not being on for 24 seven, you'd probably have a better attitude towards your on call, but you also wouldn't be burnt out from on call. So you'd probably be a lot better at your job.

Sam Rose: Completely agree. And then the atmosphere in site reliability engineering, really good.

And I miss it a lot. I will be honest, I think people really cared for each other. People really cared about the workload you had, and they really cared about how much Because not all on call shifts are born equal. Like, you know, you could be dealing with a week of absolutely nothing, like nothing goes wrong, which would be great.

It's rare, but it's great. Or you could be dealing with like the worst outage you've ever seen. Right. And then maybe you had scheduled where you're covering somebody else's shift the week after that, and then you completely burn out. So you don't want to do that. People are generally really good at making sure everyone's okay.

And the education aspect of being an SRE at Google, seconds to none, like anything I've experienced this. There's so much material that's been produced to help you onboard as an SRE, and teams will have kind of personalized material for that team for, you know, we care about these things, here's the onboarding doc, here's some onboarding videos, you know, we'll, if you want to, we can sit through it together and whiteboard it out and things like that.

Really, really good, and I've not seen anything like it since.

Justin Garrison: Documentation and run books can make or break an on call like shift, right? It's like, Oh, if I don't have a run book for this alert, like I don't make me start from scratch. Like that is just so hard. And yeah, teams don't invest enough in that.

One of the things that I remember from the SRE book, cause this is, I mean, SRE book came out. Like 20 18, 20 19. And you were probably there during this like . I I was there. Yeah. Yeah. I have my

Sam Rose: free copy of it. .

Justin Garrison: One of the things that I always point out to people that are trying to start an SRE team is, is from the book.

I say the, the thing that Google, at least from the outside in the books did, was they gave the SRE team power. to say, we're not going to support your service. In the book, it says that like you are embedded on a product team. If the services are not reliable enough, SRE can pull out and say like, Hey, you're on your own now.

Dev team, go do your own thing. Is that how you experienced it? Is that actually whatever happened? It sounds like you were embedded full time with Android backups. It

Sam Rose: is true that SRE gets quite a lot of autonomy in what they do and don't support. And I was involved in a fair number of what they call production readiness reviews.

So you might imagine, um, there's a much smaller number of SREs than can reasonably look after all of Google's services. You know, Google runs an incomprehensible number of services and production. And at the time when I was there, there was like, you know, a few thousand SREs globally. So it was kind of rationed out, I guess.

Like, you couldn't come to SRE and ask for support unless you had something that was a pretty critical service. And the process for being onboarded into an SRE team Was this production readiness review where SREs would sit with you, you know, you'd whiteboard the architecture, you talk about how it works, you kind of, as an SRE, you'd read through the code of the service.

And generally speaking, Google services are fairly like homogenous. There are accepted ways of writing applications such that you know roughly your way around anything that comes your way. I'm not sure how it is now. I'm imagining like my ex colleagues at Google laughing at me now because things have really changed.

But back in the days, it's kind of how it was. And it wasn't perfect. There was certainly edge cases and weirdness and, you know, been on call for plenty of things that made no sense at the time. But the broad idea was there was a process you went through to get SRE support, and I did two or three of those while I was there.

They're very intense, like, they take quite a long time. You end up iterating with the team that you're working with. Like, it's not a yes no process. It's usually a It's almost like a car MOT where it's like, these things need fixing before we'll support you. But if you do fix them and we'll work with you, then we will onboard you as a, as a product we're on call for.

I never experienced the pager going back the other way. So this is referred to as taking the pager and giving back the pager. So I've never seen The page of being given back. I've, I've, I'm not sure I like the threats of it, to be honest. I know some teams threatened it more than others as a way to kind of get teams to pay attention and do things that they want to prioritize.

I didn't particularly like that practice. It felt way too adversarial to me. I think it's good to build up trust with the teams that you work with. One of the pain points on some of the products I was working on was they, they didn't get full time dev support. So backup and auth at the time. There were long periods of time where they were considered, you know, maintenance projects and the devs were working on something new and exciting.

So it was sometimes very difficult to get dev. Time and buy into to work on stuff that needed it from

Justin Garrison: reading the book, a lot of companies, the thing I saw that they did wrong was they're like, Oh, look, we renamed our sysadmins to DevOps. And now we renamed them from DevOps to SRE. And I'm like, you didn't change.

You didn't give them any power. You didn't give them any authority over what the devs were shipping over the products, what the services they were running. They were just on call for it and it was just another wall between dev and ops. And the thing that I always pointed out to people when they're like, we're doing SRE now.

I'm like, what power does SRE have? Can SRE say no? If they can't say no, it's not SRE. It's just sparkly dev ops. And that was always the point where I, I always push back on companies that were like, we have SRE now. I'm like, if you're doing this to pay them more money. Fine. That you should pay them more money.

They're on call. They're doing stuff. Or again, like give them trade offs, right? Like give them some time off at Disney plus. Like we, every on call shift we had, we get a day off. So it's like we would rotate on Thursdays and we'd get Friday off. And it was just like part of our rotation. We're like, Hey, you take Friday off.

If you were on call, I was an

Autumn Nash: engineer at the wrong company. In

Justin Garrison: that time off, like it's. It's not a complete, you know, one to one sort of like, we didn't, we didn't track it as well. It sounds like you did. And we were on call over overnights or whatever, but it was like, Hey, if I'm on call, I know that I get the Friday off.

I get a three day weekend. And that three day weekend is really helpful to get stuff done, to be able to go out and not carry my laptop with me and go do some stuff during the day or something. That stuff really does help.

Autumn Nash: Do you feel like you ever automated? yourself out of having good documentation or is Google really good at documentation?

Because I think like sometimes when you're building infrastructure or kind of in the, I won't say SRE because I wasn't an SRE, but kind of like in the releasing or just kind of infrastructure and keeping everything up and going, automation is so heavily focused on that people forget to actually document it and teach others well.

So like, do you think that with that push of automation, do you feel like they still balance it out really well with documentation?

Sam Rose: There was a meme at the time, and I'm sure it still exists, where some documentation would refer to as strawberries, and it harked back to a photograph from one of the canteens at Google.

I forget what it was now, but there was something labeled as strawberries, and they weren't. It was like pears or oranges or something like that. The meme was like, you know, this documentation is a strawberry because, you know, it's outdated now or things have changed. Google's not immune to these things.

Like the, there's plenty of outdated documentation and there's plenty of situations where. You'll be following some docs and something doesn't work because the world has shifted from under you. I'd say it's probably better than most companies I've worked at, but there was still, you probably wouldn't go like more than a couple of weeks without finding some bad documentation somewhere, but there really was.

It was considered important and if you found outdated documentation and you said to your boss or your team, I'm going to take a day and it should really go over this and make sure it's right, then people generally didn't argue. So I think the culture around documentation was really good, but the actual documentation itself.

Sometimes not great.

Autumn Nash: I think that's kind of like as good as you're going to get though, because the world literally does change underneath your feet, you know? So I think that's actually like really cool to hear that they incentivized almost documentation and gave you the space to do it. Because I think a lot of times when engineers are trying to get promoted or you come in new to the engineering role, there's no incentive to do.

The maintenance work or, you know, fixing something that's important for security or documenting things like there are a lot of important jobs that we need to do outside of writing code and none of those are incentivized. So trying to get people to, like, do, like, the hygiene of things is so hard because it's not incentivized towards promotion or getting leadership to look at you and but they're very important parts of being an engineer.

So that's pretty cool.

Sam Rose: Yeah, no, I agree. At the time I was there, there was this attempt in, like, shift of mindset on what work counts towards promotion and how it's weighted. And, uh, the internal line was, we need more of a focus on landing projects than launching projects. And I thought that was quite good, like, It's pithy and nice and it evokes a good, a good visual.

I don't know like how well that went. I, I, I think I left shortly after that was a big thing and becoming a thing. I've talked to friends I have still there probably a few years ago at this point and I'm led to believe things have really shifted for the better in that direction, so I'm optimistic.

Autumn Nash: I think it's so rad that you did SRE and you were so, like, back in, deeply technical, but your blog just reads as the most warm and fuzzy thing that I've ever read.

I felt like I almost got to know, like, a little bit just with, like, your little, like, one paragraph, and it's condensed in, like, what, four sentences? But it's, like, I just knew you were nice from this paragraph. Like, I just felt warm and fuzzy from four sentences. And usually like, I think we get this kind of like rep that engineers are like not warm and fuzzy and that they don't like people, but that's like the warmest fuzziest thing I've ever read.

I also don't like wasps and I buy way too many more books than I can read. I was like, me and Sam are gonna be friends. Like .

Sam Rose: I, um, no thank you very much. I, I also, the

Autumn Nash: buttons work on your like , sorry, I had to fan girl over that.

Sam Rose: type in Moo, MOO.

Autumn Nash: Oh

Sam Rose: my gosh. There's a whole bunch of Easter eggs

Justin Garrison: in there. Oh boy, that is absolutely ridiculous.

Obviously it's a terrible audio show. If you go to samhu. dev, Sam's websites, he has a little keyboard on top, a little like six key keypad, and in the buttons you can hover over them, and then you can also click on them. And if you type M O O, I'm going to let you go to his website and figure out what happens.

This is the

Autumn Nash: coolest personal website. Like you do this in your spare time, dude? Like what do you mean? Yeah. Like mine's is so ugly. Like this is so great.

Justin Garrison: Wait, before, before we dive into that completely, you went from SRE at Google, then you went to the Android team. It sounds like you're writing a lot of Java.

Like in general, was this, yeah, it sounds like a lot of Java here.

Sam Rose: I'm a Java apologist. I think I, I still think Java is a nice language let down by a bad tool set. That's what I normally say. Ooh, I like

Justin Garrison: that take. That is, that is the, that is like the mid take on Java. It's not really spicy. It's like, yeah, no, it sounds about right.

All the other JVM type languages try to improve the tools more than the language, right? Like the Kotlins, the Scalas, the other things around it seem like the toolchains are better, or at least more opinionated. They're more Ruby on Rails.

Sam Rose: Yeah, the toolchains get better. I think the languages do try and work around some of the shortcomings of Java.

Like Kotlin, for example, is far more terse, and it's got this whole co routine focus that Java doesn't

Justin Garrison: have. Kotlin's like, Java got jealous of Python. It was like, well, we're gonna do that too.

Autumn Nash: Okay, like I was kind of let down by Kotlin though. And it's not that I don't like it, but I love Java and I love Python, right?

And I felt like Kotlin was going to be like, if Java and Python had a baby and I got like readable, like easier Java. And I did not. I got a blob. Like, it's, the structure is one of the best parts about Java and I just felt like Kotlin, when you try to read, like, a big repository of Kotlin, it's just a blob.

Sam Rose: I unfortunately have a biased opinion of Kotlin, because I worked on the Kotlin support in Android Studio, and it was, Nightmarishly hard. Yeah, it was really challenging. So, so, I have never used Kotlin in like a fun context where like I'm playing with it for the joy. It's always been, you know, incremental compilation produces completely invalid bytecode.

What am I gonna do, you know? It's, uh, it's unfortunate I can't have an unbiased opinion of it. But from what I saw, like, the goals looked pretty noble. I think at the time, at least, anyway, I remember Java 8 coming out, and there was this, like, resurgence, I think, in the syntax of Java. Like, they, they, I think Java 8 was the first time they really started to try and make the syntax into more modern.

So I think they got the, they got lambdas for the first time, like actual like nice usable lambdas in Java 8, if I remember correctly. And then Java 9, they brought the module system in, which broke a whole bunch of stuff, but it was in service of trying to make things nicer. And since then, they seem to have really been putting a huge amount of effort into making it a much nicer language to use.

And I think of a lot of people that tried Java back in like the 1. 7 days, tried it now, they'd probably find it a lot more pleasant. I don't think it's got the same velocity problems that it used to have. I think it probably is 2 OOP. Like, it would be nice if, like, bare functions were a thing and stuff like that, but, uh, broadly speaking, I think it's a very nice language.

We need Charlie Marsh to come along and build UV for Java and just make everything nice.

Autumn Nash: I think it's gotten better, but I think sometimes Java is very much part of like the old club of just like suffering, you know, and people almost want to like, they feel like you have to suffer to hang.

Sam Rose: The JVM is something I could say nasty things about. It's a remarkable piece of technology, but care and feeding of a JVM in a production environment.

At least way back,

Justin Garrison: it was really challenging. From the Ops SRE side, I hate JVM languages because of the JVM, right? It's just like, oh, I don't care how nice that language is to write, I hate managing it.

Sam Rose: It's a heavy beast. It does a lot. It's nicer now with G1. Like the G1 garbage collector, they, I think, made a good set of trade offs with it, where there's way fewer knobs to tune.

Tuning the concurrent mark and sweep garbage collector was very, very hard. It was much more of an art than a science. You had so many parameters to worry about. Whereas they released G1. I think the G1 is still the default garbage collector, but you get essentially one parameter. And it is how, how long is the maximum pause your application can, can do.

And you say like 200 milliseconds, and then it tunes everything around that for you. So it's a much nicer interface, much nicer to care for and look after in production.

Autumn Nash: I just don't know why they don't teach more about JVMs and garbage collectors in college, you know? Because I feel like they teach you how to code and use Java, but like, none of the stuff that you need to know for production.

And I'm like, what's the point of paying 40, 000 for I would

Justin Garrison: love a, a like College level debugging course, right? Like I would, like, I would get a whole degree, like, like, give me GDB, give me S trace, give me logs

Autumn Nash: and stuff. Like, cause that's like, you pay like so much money and they're like, this is the fun parts.

And that's like, not even the stuff that you do every day. Majority of the time, like nobody's writing a brand new application from scratch. If anyone

Justin Garrison: is listening to this debugging course that I think that would be fantastic. And that is actually something that. People have asked me to do some things on like S Trace and various like tools.

I was like, it's not a course. It's like, here's, here's 10 videos on like these tools or something, but, um, I would love that. The closest I can think of is

Sam Rose: Brendan Gregg's book on debugging and stuff like that, which is really like, cause he's

Justin Garrison: a world mentor. I was, I was a reviewer for his second edition and let me tell you, like it was, it was fantastic and way over my head in many ways.

It was like, like the bread is just like, like, I was like, I know how to do some debugging and I don't know how I got roped into like, Hey, we want to review this. I'm like, I would love to review this. I read the first one and the second one, like I'm pointing things out and then it was just like, come back.

It's like, no, no, you're wrong because of X, Y, Z. I'm like, Oh, you're right. Like I thought, I think of this in a different context or different use case. And yeah, those tools are, are very specific and you have to know exactly how to use them in the right way.

Sam Rose: Onboarding educational material will split into like, you know, very generic stuff that applies to every SRE then quite team specific stuff.

One of the generic courses that I love the most, and I think like I've carried a lot from this into my life after Google was a course given by a guy called Andrew Widdowson. Andrew Widdowson, sorry. And he did a How to Reverse Engineer a Production Service. So it was for people who, it's kind of often that you've got a, an SRE owned service that talks to a bunch of services, and not SRE owned, it's just like the nature of things that will eventually happen.

And it's a dependent service that's causing you problems in some way, like maybe it's not giving you the right responses back, or maybe it's fully down and you have to try and figure out why. So very often you'll be reverse engineering a service you've never seen before in your life, and it'll just tell you, you know, Where do you start?

Like find the entry points, go and Google's got lots of like Intel debugging tools that are not, not available to the public. So it's like a masterclass on using those as well. Really, really thoughtful and methodical. And I've taken a whole bunch of the. Lessons from it with me into, into the future. Like I always now look for entry points.

Like that's the first thing I do. It's like, how does the, the code path get into this section? That's not doing the right thing.

Justin Garrison: Did you go from Google to build a mess, build a base?

Sam Rose: Oh, buddy base. No, no. But this is something else we could probably talk about that I have opinions on is my, I have a very fragmented career history.

Like I, I'm one of these people that, um, I think a lot of tech influencers would say, I'd just been that CV. He's been worked at too many places. I did a little misadventure through FinTech and FinTech just wasn't really my thing. Thing, I think I, I got into it for like quite idealistic reasons. I think ultimately day to day, you're not really helping users financially.

Like, I think there's a, I don't know, it's not that they're like doing like horrible nefarious things, but it's not as focused on helping users succeed financially as what I wanted it to be. So I worked for a couple of banks in the UK and then I came to BuddyBase after that. So BuddyBase is a, like a low code app building platform and it's self hostable.

So the idea is. If you want to build internal tools, like you say, you want to like put a form in front of something or you want to connect a couple of databases together and automate certain things in like a, almost like an IFTTT style way, like thing happens over here, trigger thing over there.

BuddyBase has tools for doing all that stuff. So you, you spin it up in your own VPC, connect it to your internal databases without exposing anything to the internet. You can build forms, you can build applications, you can put graphs and stuff on it. It's got like a, not quite like drag and drop, but fairly close to like a, a WYSIWYG editor for, for patching things together and setting up triggers and stuff like that.

Justin Garrison: I'm, I'm just still, uh, getting over the, if this than that reference, because I don't know how many of our listeners even know about, like, it's not a, it's not a big thing anymore. Right. Cause like you, there's like the really old school, like YouTube pipes folks that are like, Oh, we did it first over here.

And, and there's a few things in, but like, if this and that, I. feel like had a very big moment in this, like, Oh, look at all this cool stuff we did. And then things like Zapier and other things kind of came in and were like, we do it more of them. And we give you more flexibility. If that's not was very, I don't know, user friendly in a lot of ways.

It was very big. I remember the, everything in the UI was big, it was very big buttons. Yeah.

Sam Rose: Yeah.

Justin Garrison: Back to BuddyBase. Like. You're doing back end there?

Sam Rose: Mostly, yes. So we are entirely TypeScript. So TypeScript in the back end, TypeScript in the front end, Svelte in the front end. I mostly work on Any of the backend end points that the front end will talk to.

And then also kind of care and feeding of our Kubernetes cluster. What does TypeScript on the backend look like? A lot like TypeScript on the front end. It's a, you know, there are certainly some differences to be aware of, but broadly speaking, it's, it's just TypeScript, you know, it's nothing tremendously

Justin Garrison: special.

I mean, the only times that I've really written TypeScript was for like the AWS CDK and for front ends. And so like, those are like. Automation and then the front end stuff, is this like, do you run it with NPM? Like, is this like a Node app that runs like a TypeScript? Pretty much, yeah, it's,

Sam Rose: it's, it's runs just Node.

js. You spin up some, uh, we use Koa. Imagine kind of like Flask, I guess, in Python, where you can create endpoints and you can have middleware and things like that. And then you define all of your endpoints as just bare functions that take a request and return a response.

Autumn Nash: Is it as frustrating as Flask?

Because I was so excited to be able to use Python to make like websites and web things. And then it just felt like beating your head against the wall over and over.

Sam Rose: It's one of those technologies, I think, that it looks amazing on the homepage, right? Like, oh, look how easy it is to get an endpoint up and running.

And then you spend more and more time with it and you realize, oh no, I need to, I need to start like moving things in separate files. And I need to start having middleware that attaches to everything and no shade on Flask at all. I think Flask is fantastic, but I think it does get to a certain point where you start to have problems that You know, you may not have initially anticipated having, and then you have to start to create solutions that wrap around things in Flask.

And then you get to the point eventually, it's like, damn, I should have used Django.

Autumn Nash: I love Python, and Flask is, I don't know, it's cool, but it, the more complex it gets, you're just like, but why?

Justin Garrison: Yeah, Flask is in that weird middle ground where it doesn't handle enough things. It's not quite Ruby on Rails, right?

It's not very opinionated, but it's not low level enough like, like Django. And so you're like, oh, I want to do this weird escape. Like, yeah, you just got to go. do your own thing.

Autumn Nash: I think it's the gluing it together part that is a pain. You have to make all these different, like, functions and then trying to make sure that it all works together and then you put in the CSS and it's just, I don't know.

I guess that's all for now and I'm just a back end person also, but.

Sam Rose: For me it's the feeling that there is almost certainly a correct way the community has landed on for doing this, but it's not very exposed to me and I have to either come up with it or go and read other people's code to figure it out.

Justin Garrison: You, you mentioned briefly there that the backend you have, you're managing Kubernetes clusters. Yeah. It's a self hosted thing, but you're like, here, go run it on Kubernetes.

Sam Rose: So there's a cloud offering, right? So, uh, you can either self host it if you want to. And I think most of our customers who are using it kind of commercially do that.

And then if there is a cloud offering that if you just want to try it, or you have a use case where. You don't really need to self host it, then we can host it for you. So we have like a multi tenant deployment of BuddyBase that runs in Kubernetes. And that's in EKS. So it's a reasonably like vanilla Kubernetes cluster.

We're not doing anything tremendously fancy with it. It's just BuddyBase itself is a helm. We have a helm chart for BuddyBase that you can go and spin up in your own Kubernetes cluster. We just run that in our production Kubernetes cluster.

Justin Garrison: As someone who came from SRE at Google, not necessarily on the Borg team, but you had infrastructure, internal infrastructure, probably dealing with a little bit of Borg here and there.

And then going on years later, dealing with Kubernetes. What's that, what's the difference look like for you? Using it's very

Sam Rose: different. So I, I, I remember how it felt to use Borg. Uh, when I got to Google and I, you know, I'm in a very established team, Android's been there for a long time. You know, there are ways of doing things already, spinning up a new service, like scaling things up and down, releasing new config changes.

These are all solved problems at this point, that there are kind of push button solutions for all those things, or, you know, make a commit with very well known, well structured changes. I think I'm a Kubernetes apologist as well, I guess. I think a lot of, it's very fun to hate on Kubernetes. People love doing it.

Uh, I'm one of these people who I think Kubernetes is actually really, really good and I like using it. I started using it, uh, actually, oh man, I forgot, I, I wrote a Nebula between the FinTech and the BuddyBase sections of my career. So Nebula, the video streaming platform, it was my first exposure to Kubernetes.

So before I joined Nebula, I bought A couple of Kubernetes books. Uh, I'm like trying to eyeball them on my shelf, but I can't see them. So I, it's like Kubernetes best practices or something like that. Or Kubernetes up and running, I think was one of them.

Justin Garrison: That was Kelsey's. Yeah. First book.

Sam Rose: Yeah. Kelsey's.

Yeah.

Justin Garrison: I, he was,

Sam Rose: I think he was your first guest. Yeah. Fantastic. Really great book. Like I feel like had I not read that book and I think a lot of the problems people that complain about Kubernetes might be having here is that they've not got the grounding in how you're meant to think about and use Kubernetes, like the importance of labels and selectors and things like that.

It's a very unusual concept. If you've not come across it before, I think people can get themselves tied in knots when they're not thinking about it correctly. It does. the educational journey in the way that I like to do it as well, which is you start at the very basics and you build on top and you build on top until you get to realistic scenarios.

Actually, I have a bare metal cluster running over in the corner there in a little server rack, so just like a few Raspberry Pis kind of wired together as a Kubernetes cluster and I I've been kind of caring for that for the last like six or seven years now. I keep it up to date. I run it as like a high availability cluster, even though it doesn't need to be.

So I get the practice in doing those things. It's been really fun, a really good learning journey. All that kind of translates now to what I'm doing at BuddyBase. So we have an EKS cluster, lots of things managed for us and things like that. But generally I think companies of all sizes can benefit from Kubernetes.

Like the line people like to use is, you know, you are not Google. You don't have Google size problems. You don't need Google size solutions. I think to missing the point a little bit, I think Kubernetes has become this shared platform for deploying software where if you read Kubernetes up, up and running, you spend the three or four days.

It might take an effort to read that and go through it and go through the examples. You can now benefit from a massive community of solutions on top of kubernetes, like you'd get horizontal auto scaling for free, you can use tools like lens to get this like incredible dashboard into your cluster for essentially no effort whatsoever.

There's so much stuff out that people have built. I think it is probably one of the highest velocity choices you can make early on in your startup journey.

Justin Garrison: I feel the same way about like rails, right? Because people are like, Oh, well, like, do you need rails? Like, I don't know. What are you trying to do?

It's like, it's a very opinionated tool set, but the benefit usually comes from the ecosystem. Not the tool itself, right? And when I was at Amazon and ECS kept saying, like, we're going to try to compete with Kubernetes, I'm like, no way you're not like it's don't build the Kubernetes features into ECS. And it doesn't matter if you have them or not, because you don't have the ecosystem, right?

Like all of the tooling isn't there for ECS. It's there for Kubernetes and you're not going to compete with the rest of the internet's building for this single platform that can be run anywhere. And so, yeah, I very much agree that like the benefits of Kubernetes is not necessarily in Kubernetes itself.

It is in being able to learn and onboard people, being able to hire people that's that work on it and are experienced with it. And just like, oh, you know how Kubernetes works. We might use it a little different. We don't like Helm charts or we do some other like deployment, whatever, like a lot of things that people do that can be different.

But the basic fundamentals of it, like, oh, well, there's a controller, there's a CRD, there's some storage, whatever. Like I can dig into it. It's like. The first time that people were going from BSD to Linux, right? And they're like, Oh, well, like this Linux is more of a standardized thing in the community. But this Unix thing is, is enterprise.

It's the reliable thing. Like everyone's going to do that. But then like the tools work different and like the philosophies underneath on those operating systems were different and. The first time I logged into a BSD system, I was like, I don't know anything that I'm doing. Like, none of this makes any sense to me anymore.

Like, yeah, I'm in a terminal, but the commands don't do what I thought they would do. And just that, like, mindset of, like, I know how this fundamentally should work, the philosophies behind it, even if it's used in the wrong ways, still helps me accelerate being able to be effective.

Autumn Nash: I think that's also, like, Partially, like when you get something in open source community and it's well used, like in multiple places, the more people use things, they're going to create open source projects to make their problems that they're having easier. And I think Kubernetes really benefits from that, like the whole community and the fact that people are using it at like Google, but they're also using it at multiple other companies.

People are going to build, like they're going to write books, they're going to do documentation, they're going to build tools to use that. And that's. Kind of like hopefully what we, why we continue to fund open source and, you know, contribute to it because it makes it so much better.

Justin Garrison: And sometimes people see the surface of Kubernetes, like this is so big, it's too complex.

And I'm like, do you know the flags for curl? Have you done looked at the curl man page lately? Like that is a crazy complex tool that you have to know 10 percent of right to be effective. Like you can do most of what you want with a little bit of that tool. Go to FFM peg, right? Like this. I have to Google that every time.

I'm like, I don't know what, what I'm doing next. Right? Let me, let me find the. Incantation for FFmpeg. Those are incredibly complex tools that are also used at large companies and they solve big problems, but I only need a little sliver of it. And you don't need, you don't have to use the entire surface area to be effective with it.

Autumn Nash: I feel like most of the time you only use, like, even with like Linux, there's, I think I've taken the Linux exam twice and you only use like 10 commands,

Justin Garrison: go look at everyone's get history. Like you can run that, like the top commands and they're like. Eight outta 10 are the same. 50% get status

Sam Rose: over and over again.

yeah.

Justin Garrison: Ls, cd, get status? Yeah, pretty much.

Sam Rose: So, so I, I have a, a degree in cs, um, from the sixth West University in the UK doesn't exist anymore. Um, , one of the things I love telling people, one of assignments where school, one of the isli I got given was past the MAN page of LS and find out. Every letter of the alphabet not used as a flag.

And I can't remember what the answer is now, but it's very few. Like almost every single letter of the alphabet is used in LS.

Autumn Nash: That's a really good activity though, because then you have to figure out what is used. That's genius.

Justin Garrison: Once command lines have minus like dash letter and also plus letter, like, you know, you're in some trouble, right?

Like once you're like, I'm in the plus flags, this is, this is a bad idea. I'm pretty sure FFmpeg is, it's just a programming language now. It's not fine. I'm sure it's turning complete and you could do whatever you want. When did you start doing animations?

Sam Rose: What got you into that? Beginning of 2023. So, so I had been blogging for a long time before that.

I've written some moderately successful pieces here and there, featured in some newsletters and things like that. But broadly I'd not, I'd not been met with any success when I was blogging. It's still doing it mostly for me, mostly for Solidifying the things that I had learnt, uh, I find the process of writing really does highlight things you don't know.

Like, things you thought you know and you don't know, very, very quickly. So it's been something I've done for, uh, well over ten years. And then I got really into the writing of, and I'm gonna butcher his name, but Bartosz Cieszynowski. Uh, the guy who does the mechanical watch post, he Recently released the post on moons, he does these incredible, usually like physics based, so he did the physics of bicycles and the, uh, the, uh, GPS, and these are all beautiful, 3D visualized, you can interact with every piece of it, he did once a mechanical watch one, and the internal combustion engine ones, like he's 3D modeled an entire mechanical watch, and an entire internal combustion engine, And you can see these exploded views and all the, the parts are color coded.

And I was immediately in love with this. I thought it was brilliant. I, you know, I went through all of the combustion engine mechanical watch, his one on sound, his one on the moon's one most recently, just couldn't get enough of it. I thought the style was brilliant. And I thought to myself, you know, what if that but programming?

Like, like what is that? But. Algorithms, uh, or data structures, or whatever. And I thought to myself, I'm gonna have a go. I know that I'm not gonna get anywhere near to the quality that he's done. In my opinion, I think he is probably the most impressive content creator in like the written form on the internet.

I don't think anyone comes close to him. He's so, so good at what he does. So yeah, I didn't want to do 3D, I didn't think I needed to do 3D, partly because I didn't think I could, I didn't, I've not really done 3D visual programming before, 2D is way easier to reason about, and I think, and I still think this, it's not just a cop out because I don't want to do 3D, I think you can probably Visualize most computer science ideas in two dimensions.

Unless the things that are just explicitly three dimensions. So I'm sticking with it for the time being and getting better at it as I go. And you'll notice, when you, if you look at like, uh, a piece by Bartosz and a piece by me side by side, like it's, it's almost plagiarism. Like just the style.

Justin Garrison: It's inspiration.

Sam Rose: Yeah, I guess. Right. You know, good artist deal and all that stuff, but except for DeepSeek, right? That's not, I'm sorry.

Justin Garrison: We

Autumn Nash: knew he was gonna throw shade in there somewhere. Like,

Sam Rose: we not allowed for some people, I guess. I was listening to your episode of Crystal from from the last week. Just before this. Yeah.

But no, like I very similar, you know, you know, use similar use of colors, but I essentially took all the parts of his work that worked well for me. And I tried to apply them to something in computer science slash programming. And I settled on load balancing for the first one. It's a topic I knew fairly well.

I had a clear picture in my mind for how I would visualize it. And this is a recurring theme. I tend to think very visually about these things anyway. And I never go as far as saying I'm a visual learner, because some people get very upset with that. Some people are like, there's no such thing as visual learning.

I am so a

Autumn Nash: visual learner. That like, I feel like if you had like taught all of my CS classes, I would be like, Even better of an engineer because when I you're sitting here saying this other guy is like a better content creator And I've never seen his work so I can't like say but like your visuals are beautiful like they speak to exactly how my brain works and I think that the way that you did this is so rad because There's so many kids that don't like I have very like Neurodivergent children.

I'm definitely neuro spacey and we go through our whole lives thinking that we're dumb or not as smart as other people and then when you see the way that you've broken these things down, they speak so like heavily to my brain. And I just wonder like if more people discuss topics in this way, like how many other people would we be able to bring into different just career forms?

You know, like people were like, well, math and engineering or they're so hard. And like, I remember I was in a parent school. class and it was for like helping your kids go to college and this guy was like this woman was like oh my son wants to be go to school to be a developer and he's like and this guy comes in he's like i'm an engineer and if he's not good at math and he doesn't get good grades tell him not to do it and i was like bro like who are you

Sam Rose: like oh yeah i'll go do that right now yeah you

Autumn Nash: know what like who are you the nerve it cracks him because if you go on an engineering floor we've all got like six different things of caffeine and we're totally like all like just as neuro spicy, you know?

And I feel like if people, uh, learned how to kind of teach in the manner that you are, and I hope more people do it, and I'm really excited to find your stuff because it's like so many, like math is instructions, programming are, is instructions, right? And especially kids with neural spicy brains who want to know how everything works.

It's amazing how their brains would actually be like amazing for this job, but it's just not being taught in the right way. And it's so exciting to see somebody teach in a way that will help other people get these complex or, you know, I guess dense information, but it's really not that dense. It's really a cool puzzle that you're figuring out, but it depends on who is giving you the information and the way that you do it.

It's just, it's so cool and impressive.

Sam Rose: Thank you very much. Everything you said just like hit me in my soul. Like I completely agree. I, so I have a neurodivergent kids as well. So I've got two autistic little boys and I think about this a lot. Like, uh,

Autumn Nash: Me too.

Sam Rose: Ah, amazing! So I have a five year old and a four year old, and I didn't know anything about autism before having my two diagnosed, and it's been a real journey, and I think I had an idea of what it meant, and now I'm kind of learning and going through it for real.

It's, it's very much not what I imagined it was, but I, I do worry a lot about education. So we, we send them both to a special needs school that's kind of specifically for autistic kids, and they're thriving. My own experience of education wasn't wonderful. I, I did good. I did okay. But I never really connected with it very well.

And there was something when I read some of Bartosz's work, similar to you, I was like, this is how everything should be presented, at least to me. Like, specifically me. I would love everything to be presented in this way. That'd be great. And, but I construct these things in my head anyway, as I am learning.

Like, I'm learning about load balancers. I am, in my head, constructing this idea of like, okay, Here's a set of five servers and it's round robining between them and I can kind of see it happening.

Autumn Nash: Yes. I have a mental map map of everything. I have to be able and I need to, I need so much information in the front part of learning something because I need to be able to build that.

Sam Rose: Yeah. It, it, it's, it's one of the most fun parts of the process. It's getting, getting what's in here onto the screen in a way that's legible. can be built up from the very basics. And that's another thing I think is very important. And you'll see it kind of repeated throughout my work is I try and start with the absolute simplest, most stripped down example possible.

Autumn Nash: This is going to seem personal, but do you think that your experience with your children has made you a better teacher and a better engineer? And I'm asking this question because it has been such a journey. I have three little boys. I have a five year old, a seven year old and a 10 year old and getting my oldest, the proper education.

Like has been such a struggle and we've had to switch schools and I was like determined that he would never feel like he was dumb or bad at something because I went through my whole life feeling like I wasn't like, good at school or smart, even though I did well, but it like caused me so much anxiety because I had to like make myself focus, you know?

So like, Learning, and then I just was like, I always knew I had ADHD, but I didn't know I had autism until like going through all the stuff with him, you know, cause girls present in a different way. So just like learning how to teach him for his brain and learning how to break down concepts and kind of like learn more about how brains work, I feel like has made me a better engineer and problem solver.

But it's like amazing to see the way that you break down all these problems. Do you think that influenced like how you teach or like. your empathy for how to break things down for people's brains?

Sam Rose: I'm not sure. I've never thought about it, and I was thinking about it a little bit as you were talking. I'd say it's probably yes, but I couldn't pinpoint exactly how without thinking about it a bit more.

I think it's certainly given me an appreciation for a wider variety of human experience, I suppose. Like, uh, early on when I, we had just gotten our youngest diagnosed. I remember reaching out to an autistic adult who had autistic kids to have a conversation. Because we were wondering, is special needs school the right choice?

Like, I think reasonable people can pitch it both ways. Like, do you want your child to feel different? Like, there's different ways of feeling different as well. So either you go to a mainstream school and then you feel different because you're different to the mainstream kids. Or you go to a special needs school and then you're different because you're in a different school to the other kids.

So. I was confused about like, like I didn't want my kids to get to like 15 or whatever and hate the decisions that I made for them when they were younger, like I wanted to try and make the best decisions that I could and I thought that, you know, uh, the best way of doing that is talking to people who have been through this and, and getting their opinion and I remember saying to him, Oh, I don't really know any.

Autistic people. And he's like, yeah, you do. Yeah. You know what? It's amazing.

Autumn Nash: Like now, like all my friends are getting like themselves diagnosed and we're like, dude, I was like attacked. I was filling out that form for my kid doing his diagnosis. I was like, what do you mean? Do you mean this has been my whole life?

It's wild because my children are so different. My oldest goes to, uh, I guess you say like a nor a regular public school, but he is in like a res extended resource room. But my two youngest are just in general ed, but the way that their neurospaciness manifests are completely different. One does great in school, and like, Loses all of his, I don't know, containment at home because that's where he feels safe and the other one is opposite and is totally chill, does his own thing and builds things in his room at home and absolutely hates school, like, but it's, it's wild how they can be in the same family and the same two parents and they have completely different, like, manifestations of that.

So it's like, it just, It's made me really appreciate how our brains work and how, and it's so funny, like, do you remember Justin Stanley from AWS, Justin? No. Oh, so like just you or him or like my friend, one of my favorite engineers at work, my friend Ben. Y'all will talk about stuff that you did as children, like, like, Ben would, he built a, like, a Game Boy app thing on, like, a calculator because his parents wouldn't let him have it, and Justin was saying how he built a rocket, and it's like, these things that people say that little boys were getting in trouble, they were so smart.

You know what I mean? They were problem solving, and if you look back, Einstein, um, Sebastian Bach, like, Beethoven, so many of like, the revolutionary people in history had autism. I brought my son like, a whole stack of books of like, people in history who had had autism when he first got diagnosed, and it's like, If you look at the amount of people that were like, just game changers and how we interact with music and engineering and so many things, it's just, and then you go and you work at a big tech company and so many, like, it's just so funny that we've been there the whole time and nobody knew.

Justin Garrison: Have you ever read the book Life Animated? It's about a parent who had an autistic kid. And, and I, I read that one while I was working at Disney animation. Cause it's a lot about how he loved 2d animation and he related all of his like feelings through the characters. And I remember I was audio booking that back and forth in my commutes to Disney.

And I was a new parent at the time. And I'm just bawling like every day, I'm just like, just sobbing. Just like, this is like so hard to voice like. How things are important to different people in different ways. And so I do recommend it, but, uh, just be prepared that it might, uh, might cause some feelings.

Autumn Nash: Look, I'm like so tough, but when it comes to my kids, I'm warm and fuzzy. And I'll like rip your hole. Like I'm the mom that like, we'll go up there and like, they know me so well. Like I know everybody at the department. So like, I will totally be like, even the Disney shorts make me cry. Like those like my favorite.

Justin Garrison: Circling back a little bit on, on just education side of things in the animation, like you said, you were sitting on this load balancing thing. Cause you're like, I know how to represent this. I know how to visualize this, but then you had to like do it. Uh, I remember back in the, the like early two thousands nineties, like the internet is multimedia.

Like there was this huge, like we are going to do like video and text. And it was like a big deal that you could do both. And I feel like. In this case, like this represents what's possible on something on the internet, right? Like you could have animations right alongside the text and it's interactive and you can change how you're, you know, interacting with it.

And, and you've done what, seven animations now? What would you have done different? What did you learn through this process? And what are you excited about doing next? So

Sam Rose: I've open sourced all of the visualization. code behind these posts. They live in a separate repo. I don't, I, so I published them under an MIT license and I very explicitly don't publish the, the words under an MIT license just so I have something protected in it.

But I'm very happy for people to use the visualization code and people have done, which is really cool. If you go back and you look at the load balancing post though. So I think it. It really highlights the kind of Luddite I was when I went into this, where it's all like, raw JS that there's absolutely no like, package management, I don't use anything from NPM, I use a couple of like, JSCDN links for like, Plotly and things like that.

But it's put together like someone that has no idea what they're doing, and the reason is that I didn't. I really didn't. And I, I ended up using Pixie. js, which is a, it pitches itself as like an HTML5 game making library? So it's a lot of resources for drawing to canvases and doing animations and things like that, handling input.

It seemed pretty good for my needs. I don't use it anymore because it's, it's fairly heavy. So it includes everything you'd need to make a game. You know, you've got like audio and video handling stuff in there. You've got input handling, lots of different things like shaders and stuff like that, that I don't use and don't need.

So it was pretty heavy. So I think it, it comes to like over half a Meg. I think possibly even more than Plotly was like a megabyte of JS. And I, I spoke to Scott Hanselman on his podcast, and he was going through the network requests on my page. He's like, that's like over a megabyte you've loaded there.

I'm so sorry. I didn't mean to. Scott calling you out. Yeah, just getting shredded by Scott. The moral of the story is I didn't know what I was doing. I learned everything I needed to know to do some 2D animations that kind of worked well enough on the page. And I think there's actually some devices it doesn't work well enough on because I didn't realize, for example, When you draw to a canvas, I'll be using WebGL.

Uh, you can only have so many WebGL contexts on one page, and the context is the thing that you draw to the canvas through. It's the thing you issue commands to. And you can only have, on some devices, like, eight of these things at any one time. So I think on some of my devices, as you scroll down, some of the things just don't load.

But it's, like, reasonably old mobile devices, and I haven't gone back and fixed it. It works fine on Modern phones, modern browsers and things like that. So I thought it was probably, I don't want to touch it cause it's, it's terrible. I honestly, if you go back and you look at the code, you really will think I'm absolutely unable to do front end development.

Autumn Nash: I don't see any of the stuff that you see. Like I was geeking out. About your load balancing one and like the animation seems so full even though it's 2d That's the first one I clicked on and I thought that was your most recent blog And then I went through the other ones So like I totally like I think that you have like the artist You know, like when you're an artist and you look at the things that you do you see all the problems in it But we don't actually see the problems in it because I didn't see that at all Like all the things you just described I did not see any of that when I looked at it.

I thought that was like It's such a full concept for 2D, and like, I thought that was such an awesome way to present it, and I didn't see any of the problems that you just stated.

Sam Rose: I think it's a blessing and a curse. I, I think, I, I think I, I, I care very deeply about the quality of these things. And for example, if you open the load balancing post on mobile, And you rotate the screen.

None of the visualizations will adapt to the new screen size. And it kills me that that doesn't work. And you'll notice all my later ones, that does work because I was so, like, ashamed of myself for not making it work in the log balancing one. I like what you said about art. I'm, you know, very flattered. I want to lean into that way, way more, actually.

Thinking about it. So if you were to open the Turing Machines post, for example. Something I did for that one. But I wasn't sure about the time, um, so I drew the hero image in that, in that post by hand. So like it's, it's uh, it was done in Procreate on my iPad over the course of, you know, several evenings.

I wouldn't say I'm, I'm, I'm a good artist. I think it is enough his likeness that it works well as a hero image. Some mean people on Hacker News did say some stuff about it. I think someone was like, He looks like a weird alien. You should just stick to Shutterstock next time or something. And I was like, Ow.

Autumn Nash: I will. Can you just, can you tag me wherever that is? And I will

Sam Rose: do the

Autumn Nash: job

Sam Rose: for

Autumn Nash: you. Like, the lies you tell. Like, you are definitely an artist. And those people are just jelly. And also, like, I am willing to fight on the internet for you. Like, just tag me in. Tag me in. I have time.

Sam Rose: I replied to it saying, oh, um, it's, it's a little bit of a, a little bit of an anti AI statement, a little bit, as a, as more and more people are using AI tools to generate worse and worse content, unfortunately, um, I want my content to really stand out as It's kind of human, really organic, and thought about and cared about.

So I tried this new idea of hand drawing a hero image, and I probably will do it again, I suppose. My wife is actually a significantly better artist than I am. If you scroll down to the bottom of the Drawing Machine page, there's three drawings at the bottom, and she did those, and I think they look so much better than mine.

I hadn't told her about my plan to draw the hero image, and it was like quite late into the process. She was like, oh wow, what are you drawing? And I was like, Oh, I'm drawing Alan Turing for my post. And then she was like, Oh, I'll have a go as well. And she drew a picture of him. It's way better than mine.

I'm like, damn it. I'm still going to use mine because it feels, it feels kind of cheeky to use yours, but I want to lean more into that. I would like, and I've not said this to anyone before. I would love if. Every single one of my posts, I worked with an artist and I kind of paid them to create artwork for the page.

So, I'm writing a post at the moment, uh, very early stages on multilayer perceptrons. So very early kind of classifier neural networks. And I would love to commission an artist to do kind of their own style. So the important thing behind this is it would be their style. Artificial neurons, like actual pictures of neurons from the brain and things like that.

Just, just Artistic stuff around the writing, and then, you know, feature them as an artist that has done the artwork and then for every post I get somebody else and it's in their style, so every post has this unique feeling to it, and at the same time Using my, you know, modest platform to try and boost people, like, other artists profiles.

I have previously put out, like, Oh, hey, I want to commission some artwork, uh, this, this kind of thing. And the problem is you should get hundreds of DMs from scammers and people that are not legitimate and people that are looking to rip you off. Uh, so I, I don't know the best way to approach. artists, and I would love advice if you have any, because I think it would be so glad that you don't want

Autumn Nash: to use Fiverr, because like, they just down level, like, and just are terrible, and they've ruined so much of graphic design and animation and so many, like, other parts, that like, I love that you actually want to work with them.

It's not that you're getting a bunch of sinkhammers, though, but I just think that's such a, like, rad idea. To do that, and to really.

Sam Rose: Shores away. I think it would probably have to be, you know, through my own network of people, like, you know, asking friends and stuff.

Autumn Nash: Even networking on social media on, like, Blue Sky, but, like, networking with other artists, because there's, like, an artist feed, too.

Justin Garrison: Well, one of the things that I've found is just, like, finding the right style and experience from an artist is really, really difficult, where it's, like, there's a bunch of amazing artists out there that just aren't the thing I'm looking for. And that's like, cause like the, the vibe and the, the things that I've, I've tried to do different animation projects in the past that were mostly just like 2d hand drawn animation things.

And every time it's like, Oh, like this isn't, it's not a comic book. I don't want that vibe. It's not a, you know, a dark sort of like, uh, it's like, I want. The specific thing, and I don't know how to tell someone that I don't have necessarily the words to be able to say, like, I have to go look through a bunch of it and then do the work to like, Oh, these are the five that I think look about rights.

I'm sure there's plenty of artists that can do different styles, and maybe they just don't show that, and it's hard to, like,

Autumn Nash: Have you ever tried just looking up a video or grabbing a book and looking at the different styles? It might be like, you know, like a book of just different styles. Then that way you can kind of get the vocabulary that you're looking for.

Or even if you just Google different animation styles.

Justin Garrison: Yeah. And I mean, but like. Again, art and animation, there's a lot of people that are desperate for work right now. And it is also really hard because you will get hundreds of people that even if you say, I want this specific thing, they're still going to come to you and, and kind of waste your time.

And so like, Oh, here's all this stuff that I did. I'm like, that's not what I asked for at all. There was a lot of people that were just like, Oh, you know what? Like, you're great, but it's not what I was asking for. And. Thanks. And I, I stopped asking publicly for help for those things because of the amount of time it was taking me to like filter out all the people and then say like, okay, well, like what's your timeline look like?

What's the price? And it just never lined up.

Sam Rose: The funny story about how I originally had this idea is, um, do you know who Andy Matuszak is? He is an education researcher, fantastic, I'll link him for the show notes, uh, after this. He, his, his website is beautiful, he's obviously put a lot of time into it, and, uh, I read a post by him with the title, Why Books Don't Work, like a fairly incendiary title, but the, the, the point is that It's unrealistic to, to expect people to read a book and then know everything in the book.

Like it's, you know, we have so many more tools now. And it was just, it was just speaking to me. I was like, I completely agree. Like it's, it's a big part of what I'm doing at the moment. And then I clicked on some of his other posts. And every post I opened, like, had a different theme to it. And I was like, is this?

Is he making like a bespoke theme for every single post? That's an amazing idea. And then I had this idea, oh, I'd love to like commission different artists to do work in their style for each of my posts. But it turned out actually, uh, every single link was, was guest posts on other websites, so he wasn't actually doing that.

So I, he's given me this idea. Even though he's never actually done it.

Autumn Nash: That's cool, but it also just seems like it would be very time consuming, but that's pretty cool that he also, like, is giving people, like, a platform, but at the same time.

Justin Garrison: So Sam, what, what do you think is next? What's the next, uh, you said you're working on, like, a neural, neural network sort of post?

Sam Rose: What has been occupying a lot of my time recently is, can I do this full time, is the question on my mind. I don't think I can do the blog posts full time. I think, uh, so I did recently land my first sponsor, which is a huge step for me. Very, very grateful that, so the company approached me and we spoke and came to a good agreement, but it, but it's not like, it's not the kind of money that I could live off, but it's, you know, very, very generous for the, for the work it is.

So I'm thinking to myself, like, how would I, how would I do this full time if I really wanted to? And I'm beginning to kind of take steps in that direction, which is why. I've not made a lot of progress on my next post yet, so I've got some visual stuff working, but I haven't got everything that I need, because I'm just doing a lot of logistical stuff, like I created a company, and I, you know, I'm speaking to accountants and things like that, all the stuff that goes around it.

And you mentioned, I think, earlier on, you know, Oh, you're doing this in your spare time, that's incredible. Uh, so I do four days at work, and then I'd have Fridays for working for my own company. So it's not quite spare time, like, I have been able to carve out real time for it now, which I'm incredibly grateful for, uh, But yeah, it's, what's next?

I, I want to keep making the posts, I want to keep a cadence of like two or three a year, And I think with the sponsorship and with the extra time that should be possible, into the foreseeable future. But I'm also thinking about other ideas surrounding that, like writing these blog posts is one way of reaching a wide audience with educational content, but there's lots of other things out there, like everybody at the moment is making courses, and Very inspired by the work of people like Aaron Francis and Josh Camot and the stuff that, like, Matt Pocock is doing through Badass.

dev and things like that, and starting to wonder, you know, Is there space in that market anywhere for the skills that I have? And I think there probably is, and I've not fully fleshed out my ideas in that space yet, but it would surprise me if in the next three years I didn't do something in that space.

And then also the most unlikely thing, I can't remember who I was talking to recently, but having this conversation about physical products and talking about the question, like, what if you could hold an algorithm, what if you could physically hold. An algorithm. Like, how would that look? How would it work?

And, uh, it came about, uh, I have a friend who's got a 3D printer. And I was like, could you 3D print a Turing machine? Like, how would that work? How would that look? And we started looking at, there are physical example of Turing machines out there that you cannot, you Turn a handle and it does something, some sort of program, and it works in the same way a Turing machine does.

They're real complex, actually. Like, probably way more complex than we could achieve realistically with a 3D printer. And plus, I wouldn't want to sell 3D printed products. Like, a lot of thought has also recently been put into, like, what is my brand, so to speak. My personal brand. And I really want to lean heavily into, like, quite high quality things.

So if I did Produce something like this. In my head it would be a like a fully like metal Turing machine that you know has a weight to it and it feels good and like I'm thinking like a desk toy kind of accessory thing. So I think those are the least likely. Dealing with physical products has a whole bunch of challenges that I know nothing about.

But I do love the idea of like, what if you could hold a bubble sort? Like how would that look? What would it, how would it work? So kind of physical manifestations of algorithms is something that's on my mind.

Autumn Nash: That's such an interesting concept. I hope you don't let all the hurdles stop you, because that would be such a cool concept.

Sam Rose: The biggest hurdle right now, and I'm sure you'll appreciate this, I'm so time poor, you know, with the kids and with the Full time work. Um, I haven't mentioned it, but I'm also a trustee for a charity in the local area that works with special needs kids. Time is like unbelievably precious to me. Sam, do we have the same

Autumn Nash: life?

I think we have the same life.

Sam Rose: Have you seen us in the same room? I don't think we have.

Autumn Nash: We're, it's like You've never seen Batman and like, what's his name? Batman and Bruce Wayne at the same time. I think we're the same.

Justin Garrison: Which one of you is Batman?

Autumn Nash: Sam, definitely Sam.

Justin Garrison: Sam, thank you so much for coming on the show and explaining to us just your, your career and all the things you've been through, but then also the things you're looking forward to and the animations and everyone should go to Sam who. dev because those animations are fantastic. You'll learn a lot of things that even, even things you didn't.

really realize you didn't know, right? There's a lot of things that just got exposed once you put a visualization to some of this stuff. And I think it's fantastic. So

Autumn Nash: I hope you have a just long career in creative teaching because you weren't, for one, a rad human on your own, but like the fact that just this is so creative and I hope it's the future of how we teach people things.

Sam Rose: Oh, I really appreciate it. Thank you so much. I thank you so much for inviting me on. It's been a blast talking to you both.

Justin Garrison: I know you are semi active on blue sky. I don't know if you're on other social networks or not. Um, we do have the FAFO dot FM blue sky account has a starter pack of guests and you're in there already.

We have, we have a couple of people in there, so it's been fun. We've been just adding them as we're going. There's still, we're always a little ahead on it too. So anyone that's on blue sky, if you are looking for some of the past guests or whatever show you're listening to check out the FAFO dot FM accounts.

And there's a startup back there that you can find anyone. that has a blue sky, uh, isn't.

Autumn Nash: One more thing, what you said about websites having downtime and how like multiplayer games, that's so real. Like, it's so true. Like you can't have like five seconds of downtime on a website, but fortnight will like crash itself for like eight hours until you to come back.

And then your kids are sitting there at like 3 PM. Like they just, just, I swear to God, you're just like the way that they will wait. Just wait for the next, like, what is it called? Um, episode or whatever, and they get the new downloads and just, it's amazing, like, they could just leave up a version of, like, the old game, but they will completely take it down for eight hours and kids will sit there and wait.

It's like, you can tell all the kids in the neighborhood are outside because Fortnite's down.

Sam Rose: Yeah. That's the thing, you can go outside, it's fine. The thing that triggered it was, like, major database upgrades. They'd be so much easier if I could just take the website NAND for an hour, do it, and bring it back up, rather than have to set up a logical replica.

Yeah, much, much faster, much cheaper in general. There's this weird, this weird obsession with 100 percent uptime that I don't think is well placed.

Autumn Nash: Yeah, cause like, there's a lot of websites that we do not need 100 percent of the time. Also, the IRS website was down for like three days last year when we were trying to make like the business account for this, so like, dude, it, nothing worked for three, and it was even out, and it was like longer than the structured downtime.

It started 24 hours early and it was out longer. And I was like, this is actually important, like.

Sam Rose: Things are okay. Right. Like it's, it's not like, obviously some things probably weren't okay, but generally speaking, like we're all still here. And yeah.

Justin Garrison: One last thing, cause this episode is going out at the end of February in the first week of March, March 6th through 9th, Autumn and I will both be at the Southern California Linux Expo.

So if anyone's listening to this, wants to come to scale, it is in Pasadena, which. Traditionally has had fantastic weather. I know in January, there have been a lot of fires around the area. Um, but as far as I know, March is still going to be a great weather to attend. It's a community events. Uh, I help run the cloud native day.

Autumn, you have a talk there.

Autumn Nash: Y'all scale is going to be absolutely bananas. Like did you, have you seen the list of people going? There's,

Justin Garrison: there's some great people. It

Autumn Nash: is going to be like. Not just nerd summer camp like it is going to be unhinged nerd summer camp like

Justin Garrison: It's community run. So if you are individual if you have education or like half price tickets bring your kids I'm bringing mine on on Saturday.

I don't know you're coming with yours

Autumn Nash: if you see a horde of children That are just like absolutely ridiculous. It's probably like me and Justin's kids, but like mostly mine. Like

Justin Garrison: there's, there are game nights on Saturdays. So feel free, please come. Um, it's again, it's, it's a, it's community run events. It is affordable for a lot of people.

I think the full price tickets are like 120 bucks or something like that. So, and there is sponsorships for some of that if you need it, but there's a lot of different tracks, a lot of different tech. It is, it is a very fun, non. Uh, enterprise, non single vendor sort of environment to be in. You

Autumn Nash: are doing the best sales pitch that's like, uh, the best, like.

I've been going for. Wholesome one. And I'm just like, I'm going to do hood rat stuff with my friends. And then like, my kids are going to like, you're just going to see five little boys, like running around with like stickers on their head and stuff and like scale shirts and like raspberry 22.

Justin Garrison: I think the first one I went to a scale seven or eight, maybe.

Autumn Nash: I can get banned from my kids. Like, do you think there's be like, don't come back with those little,

Justin Garrison: like, uh, I have, if anyone's watching the show, like, well, no one's watching it now, but we might have clips. Uh, there's a bunch of raspberry pies that we're giving away for the kids for the next generation track is, is by kids for kids.

My son gave a talk a few years ago, but there's kids in the audience and they have like, uh, like a STEM room and stuff like that, but. People in the community donated a bunch of raspberry pies. Thank you so much for sending them. Um, I have been setting them up 3d printing cases, making sure they work, getting them ready to hand out at the event.

So we will have a couple of like scavenger hunt type things that you can get a free raspberry pie if you, if you come. So, uh, there's a lot of cool stuff. So, uh, please check it out. We'll have the link in the show notes and, um, Sam, thank you again for coming. No, thank you. Have a good one.

Thank you for listening to this episode of Fork Around and Find Out. If you like this show, please consider sharing it with a friend, a coworker, a family member, or even an enemy. However we get the word out about this show helps it to become sustainable for the long term. If you want to sponsor this show, please go to fafo.

fm slash sponsor and reach out to us there about what you're interested in sponsoring and how we can help. We hope your systems stay available and your pagers stay quiet. We'll see you again next time.