WEBVTT

00:00:10.349 --> 00:00:10.679
All right.

00:00:10.679 --> 00:00:12.800
Welcome to Ship Talk, the SRE edition.

00:00:13.289 --> 00:00:20.010
I'm Jim Hirschauer, host of the SRE Edition, and a recovering techie who likes to talk to other people about technology.

00:00:20.010 --> 00:00:28.750
Now, Ship Talk is a podcast sponsored by Harness, the software delivery platform, and the SRE Edition is all about reliability topics.

00:00:29.195 --> 00:00:33.335
My guest today is Matt Schillerstrom who also happens to be my coworker.

00:00:33.905 --> 00:00:47.015
The other day, matt and I, we were talking about the impact that all of the current corporate efficiency initiatives might have on the reliability of applications, and Matt had some really great insights that I thought he could share with us.

00:00:47.015 --> 00:00:48.755
So Matt, welcome to the show.

00:00:49.924 --> 00:00:50.164
Yeah.

00:00:50.164 --> 00:00:50.765
Hey Jim.

00:00:50.765 --> 00:00:52.698
Glad to be here for the first chat here.

00:00:52.707 --> 00:00:54.476
Yeah, we've been working together for what now?

00:00:54.476 --> 00:00:55.646
A little less than a year.

00:00:56.402 --> 00:00:57.271
We've had a lot of fun.

00:00:57.332 --> 00:00:57.962
So yeah.

00:00:57.962 --> 00:01:01.741
And we get to spend a ton of time together and have some really interesting conversations.

00:01:02.072 --> 00:01:07.802
So I know a lot about you, Matt, but I'd love for you to take a minute to go ahead and share your tech background with our listeners.

00:01:08.581 --> 00:01:09.602
Yeah, for sure.

00:01:09.646 --> 00:01:12.176
I've been in tech for about 20 plus years.

00:01:12.185 --> 00:01:15.480
I started out my career actually at a nuclear power plant.

00:01:15.546 --> 00:01:22.466
When I graduated college, I thought I was gonna get into the leading edge technology, but I found myself working on like 1980.

00:01:23.337 --> 00:01:31.016
General Electric, like old computers, like with backs and VMs, operating systems, like the true green screen old school.

00:01:31.016 --> 00:01:31.049
Yeah.

00:01:31.049 --> 00:01:33.302
But come on, I I wouldn't take that back.

00:01:33.408 --> 00:01:38.207
Yeah, I was at the power plant for 13 years, just doing like IT system work.

00:01:38.537 --> 00:01:39.737
Administration.

00:01:40.037 --> 00:01:47.117
I had four or five years where I was actually in engineering doing reliability testing on pumps and motors.

00:01:47.159 --> 00:01:55.429
But then I got back into the power plant control system and I was an administrator of the actual controls that the operators would use to run the power plant.

00:01:56.058 --> 00:01:57.198
So that was cool.

00:01:57.259 --> 00:01:57.528
Yeah.

00:01:57.528 --> 00:01:59.638
Really important reliability at the power plant.

00:02:00.739 --> 00:02:01.338
Life or death.

00:02:01.388 --> 00:02:01.948
Yeah, exactly.

00:02:01.948 --> 00:02:04.849
You know, is that gonna ra, is it gonna leak radio activity or not?

00:02:04.909 --> 00:02:05.989
Right, right, right.

00:02:06.090 --> 00:02:07.031
But yeah, then that was it.

00:02:07.031 --> 00:02:15.550
Target a big retailer out of Minneapolis, Minnesota for five years as a lead engineer building out their chaos engineering program.

00:02:15.556 --> 00:02:24.841
So it was like an old school IT disaster recovery team, and then we revamped it into doing real-time disaster recovery, utilizing chaos engineering.

00:02:25.450 --> 00:02:25.931
Awesome.

00:02:25.931 --> 00:02:34.950
That's quite a background you have, so some serious life or death, reliability work that you've done and then maybe not quite life and death, but serious business that you're involved with.

00:02:34.955 --> 00:02:36.270
So, great background.

00:02:36.366 --> 00:02:36.575
Yeah.

00:02:36.756 --> 00:02:43.626
So listen, before we get to our main topic, I'd like to just play a little, little game here to start out and have some fun.

00:02:43.836 --> 00:02:49.175
I'd love for you to, to tell me one truth and one lie, and I'll try and guess which one is which, based on what I know.

00:02:50.225 --> 00:02:51.855
Okay, So truth and a lie.

00:02:51.855 --> 00:02:53.366
Let's see if I can fool you.

00:02:53.388 --> 00:02:56.057
I'm an avid saxophone player.

00:02:56.057 --> 00:03:09.967
Played all my life, marching band, jazz band, wind ensembles, or I've actually never had to do Kubernetes production rollout by myself.

00:03:11.957 --> 00:03:12.497
Okay.

00:03:13.608 --> 00:03:16.698
For some reason, I think, I think you're a musician.

00:03:16.698 --> 00:03:20.358
I don't know if, if so, what's jogging my memory about that.

00:03:20.358 --> 00:03:24.737
But remember, I know you, this is gonna be a little bit easier for me, but, but that's my guess.

00:03:24.798 --> 00:03:27.348
My guess is you're, you're a musician, you're a saxophone player.

00:03:28.312 --> 00:03:28.802
Yeah.

00:03:28.802 --> 00:03:29.163
Yeah.

00:03:29.643 --> 00:03:30.002
It's nice.

00:03:30.008 --> 00:03:32.133
That's one quirky thing about me, I guess.

00:03:32.133 --> 00:03:32.612
So.

00:03:32.763 --> 00:03:33.182
Cool.

00:03:33.242 --> 00:03:35.582
Well, it's great to have hobbies outside of it, right?

00:03:36.393 --> 00:03:36.752
Yeah.

00:03:36.872 --> 00:03:37.052
Yeah.

00:03:37.682 --> 00:03:38.193
All right, cool.

00:03:38.193 --> 00:03:40.022
Well, let's, let's jump into our main topic now.

00:03:40.022 --> 00:03:46.487
So the reason that I wanted to have this conversation, Is because companies are cutting back today, right?

00:03:46.487 --> 00:03:54.579
Right now with the current economic environment companies are looking for efficiencies and those efficiencies can potentially come with compromises.

00:03:54.579 --> 00:03:59.554
And we were having a conversation about this and you shared really interesting story with me.

00:03:59.558 --> 00:04:01.757
So I think this was back to your retail days.

00:04:01.757 --> 00:04:04.356
Why don't you go ahead and, and and share that story one more time.

00:04:05.497 --> 00:04:05.947
Yeah.

00:04:06.366 --> 00:04:17.557
I think what's interesting just with this story in retail, Like it relates a lot to my power plant days because at a nuclear power plant, you have defense in depth.

00:04:17.617 --> 00:04:19.887
You have multiple redundancies built in.

00:04:20.398 --> 00:04:25.588
And as an example, like if I were to take you on a tour of a power plant and you were to point, what is this?

00:04:25.593 --> 00:04:26.458
What is this, what is that?

00:04:26.458 --> 00:04:32.187
It'd all be for like failover or safety, like hardly anything in the power plant.

00:04:33.132 --> 00:04:35.622
Like big turbine itself is what makes money.

00:04:35.653 --> 00:04:37.872
Everything else is there to safely shut it down.

00:04:37.963 --> 00:04:54.043
And when I look at my experience, like at the retail stores, right, and environments we would often like over buy like over capacity, what we really needed leading up to peak seasons with like Cyber Monday or you know, the holiday season.

00:04:54.463 --> 00:05:01.603
And what that meant was like a month before, like we expected, you know, thousand x of traffic.

00:05:02.247 --> 00:05:13.221
We would actually just be willingly paying millions of dollars in extra infrastructure just to have that capacity and confidence that our system would be scaled appropriately to handle.

00:05:13.221 --> 00:05:13.461
Right.

00:05:13.466 --> 00:05:24.112
And if you think back to like 20 18, 20 19, like, you know, the economy was really good and, you know, spending that, you know, millions of dollars extra on infrastructure.

00:05:24.531 --> 00:05:26.062
Didn't really matter, right?

00:05:26.062 --> 00:05:26.271
Yeah.

00:05:26.302 --> 00:05:30.891
Or it maybe it mattered, but like people didn't know how to look at the granularity of it.

00:05:30.891 --> 00:05:36.351
Cuz bin ops wasn't as like, you know, popular or well known back then either.

00:05:36.922 --> 00:05:37.341
Right?

00:05:37.367 --> 00:05:37.533
Yeah.

00:05:37.533 --> 00:05:42.723
So how, so how many times a year would you say that your company was over provisioning like this?

00:05:43.487 --> 00:05:43.728
Yeah.

00:05:43.728 --> 00:05:50.838
So at my retail like company, we would spin up, you know, multiple times a year over provision, over capacity.

00:05:50.838 --> 00:06:05.494
But again, we had that confidence that we would be able to perform and withstand, you know, disruption in event because like if we had network loss or you know, peak unexpected spike in traffic we bought assets that it should be resilient through.

00:06:06.259 --> 00:06:07.399
Chaotic Storm.

00:06:07.428 --> 00:06:07.999
Right.

00:06:08.062 --> 00:06:11.600
And it, and it was, you know, we were, we were rock solid generally speaking.

00:06:11.660 --> 00:06:26.209
And what was interesting, like leading into the pandemic and covid, all, a lot of these retailers, especially, you know, like grocery stores, they were running peak load almost every day with, you know, delivery and on demand shopping.

00:06:26.240 --> 00:06:26.360
Mm-hmm.

00:06:27.620 --> 00:06:38.779
And what was interesting about that too is that because they were running this well-oiled machine at peak capacity, like over provision, like they were just rock solid and stable.

00:06:38.810 --> 00:06:39.139
Right?

00:06:39.139 --> 00:06:42.589
Reliability was there like a hundred percent.

00:06:42.589 --> 00:06:47.240
And again, they were paying more infrastructure, but they were also making a lot more money at the time.

00:06:47.420 --> 00:06:47.629
Right.

00:06:47.634 --> 00:06:51.302
And it, they didn't really have to think about like that efficiency as much.

00:06:51.307 --> 00:06:52.057
Just because.

00:06:52.447 --> 00:06:53.437
It was working.

00:06:53.947 --> 00:06:54.098
Yeah.

00:06:54.098 --> 00:06:58.627
And you can do, you can do that when business is good, but what happens when times get tough and lean?

00:06:59.228 --> 00:06:59.718
Yeah.

00:06:59.908 --> 00:07:00.398
Yeah.

00:07:00.398 --> 00:07:11.718
And I think, you know, as you saw, you know, after the pandemic and more people getting out in the open and not doing as much online shopping, like retailers in general had to scale back their infrastructure, right.

00:07:11.978 --> 00:07:15.367
To like normal capacity, like pre covid times.

00:07:15.367 --> 00:07:20.408
And what you saw, like, and you can see this in the like internet if you search outages too.

00:07:22.012 --> 00:07:33.653
The, the infrastructure that they had, that they provisioned it, it didn't meet, you know, that reliability standard that they had throughout, like, you know, these high provisioned time periods over Covid, right?

00:07:33.862 --> 00:07:47.242
Because they scaled back from an EC economic perspective, but also they hadn't really tested, you know, everything that they probably should have going back to the right sized infrastructure during that time period.

00:07:47.242 --> 00:07:47.923
So then you.

00:07:48.517 --> 00:07:52.237
Ma, you know, massive outages that we saw even from aws, right.

00:07:52.898 --> 00:07:55.208
And Google and other cloud providers.

00:07:56.017 --> 00:07:56.288
Yeah.

00:07:56.288 --> 00:08:02.617
So ultimately, what is the, you know, what do you recommend as a, as a good way to fix that, to address that and to deal with

00:08:02.617 --> 00:08:02.858
that?

00:08:03.367 --> 00:08:11.528
Yeah, I think, I mean, when you look at 2023 here, you have to have the right amount of testing and like, what does that even look like?

00:08:12.577 --> 00:08:21.757
You besides having the right tools, like you need the right people with the right like checklist of how you're gonna like, handle like your business logic.

00:08:22.298 --> 00:08:33.496
So taking a step back, like what that looks like is working with your business, like your product leaders that are driving like those business initiatives, just to understand the service level objectives, right?

00:08:33.496 --> 00:08:43.486
That lead up to the service level agreements because at the end of the day, It's a business decision really, on how much you wanna spend for that, like customer transaction, right?

00:08:43.876 --> 00:08:53.537
I would always joke back in the day about selling bananas, and you could spend a lot of money on infrastructure to support selling bananas online, right?

00:08:53.596 --> 00:09:05.596
But at the end of the day, like the business can decide, you know, like, well, if, if x happens, if X disruption happens either in the supply chain or from a technology perspective, let's, let's do this right?

00:09:06.032 --> 00:09:11.672
and an engineer might overscale that infrastructure or business process.

00:09:11.881 --> 00:09:29.221
But if, if, again, if you talk to the business and kind of learn about the consequences and then you can just make those logical decisions to kind of like, you know, provision just enough to get through through that transaction, but then fail, you know, otherwise.

00:09:30.841 --> 00:09:31.231
Yeah.

00:09:31.292 --> 00:09:34.052
So it's really, it's really about understanding those failure modes, right?

00:09:34.081 --> 00:09:40.652
Like setting up the right targets, or you mentioned SLOs, so I'll just say reliability targets in general.

00:09:40.841 --> 00:09:41.331
Yeah.

00:09:41.336 --> 00:09:44.441
And then you know, how do you test for all those different modes of failure?

00:09:44.441 --> 00:09:49.270
And it's, that's, that's kind of the whole purpose of the ca practice of chaos engineering, right?

00:09:50.681 --> 00:09:51.191
Yeah.

00:09:51.250 --> 00:09:51.490
Yeah.

00:09:51.490 --> 00:09:54.600
So where it ends up, you know, like, so like chaos engineer.

00:09:55.076 --> 00:10:05.456
Today it sounds like a daunting task or a cultural movement, but really like where we've shaped it to be at Harness is a simple task that you can run in your deployments, right?

00:10:05.515 --> 00:10:12.206
So you can run a series of tasks against specific use cases that you want to prove that you're like resilient through.

00:10:12.821 --> 00:10:13.120
Right.

00:10:13.120 --> 00:10:27.373
And if companies out there are provisioning new workloads to meet certain capacities, they can simply execute some of these tests to see how their system would, you know, handle that failure, but also how it would perform for their customer.

00:10:27.403 --> 00:10:27.702
Right.

00:10:27.702 --> 00:10:30.552
What, at the end of the day, what's that customer experience look like?

00:10:31.243 --> 00:10:34.783
and then they can choose, you know, is that good enough or not?

00:10:35.173 --> 00:10:35.472
Yeah.

00:10:35.472 --> 00:10:42.942
So, you know, SLOs, reliability targets, those are supposed to be a good measure of customer experience.

00:10:43.030 --> 00:10:50.350
So is that what you would use, is that a good measure for determining if your chaos experiments or are, you know, passing or failing?

00:10:50.951 --> 00:10:51.280
Yeah.

00:10:51.280 --> 00:10:58.870
You know, like an ideal situation if you have your service level objectives set up for your service.

00:10:58.965 --> 00:10:59.504
Correctly.

00:10:59.595 --> 00:11:04.544
And that's, that's a daunting task sometimes cuz sometimes you don't know what all those should be.

00:11:05.205 --> 00:11:18.735
But if you run a series of chaos experiments to understand like, okay, this is my steady state of my customer experience, and then during these types of disruptions, this is how I expect the system to behave.

00:11:19.725 --> 00:11:30.524
you can basically just ensure that your service level objectives maintain that performance throughout that whole experience, through the steady state disruption, the recovery, the failure.

00:11:30.914 --> 00:11:36.615
And then you, you can decide if that was good enough because not every transaction would fail.

00:11:36.674 --> 00:11:38.774
There's always like a percentage, right?

00:11:39.195 --> 00:11:47.065
And again, as a business leader, maybe it's okay if like 5% of people air out when trying to buy bananas, right?

00:11:47.078 --> 00:11:47.649
Because it.

00:11:48.443 --> 00:11:54.063
low cost item and they likely won't get it at a competitor and they'll just wait a couple minutes.

00:11:54.489 --> 00:11:56.678
You know, for the system to recover, to buy that.

00:11:57.339 --> 00:11:57.818
Yeah.

00:11:57.849 --> 00:11:58.418
Yeah.

00:11:58.418 --> 00:12:00.875
I love your banana analogy here.

00:12:00.936 --> 00:12:07.115
I think, I think you had mentioned to me once that bananas were like the top selling product at, at your retail store.

00:12:07.115 --> 00:12:08.225
Wasn't that right?

00:12:09.306 --> 00:12:09.635
Yeah.

00:12:09.635 --> 00:12:13.686
Bananas were the most popular and when they ran out of stock, everybody freaked out.

00:12:13.686 --> 00:12:15.306
So that's insane.

00:12:15.306 --> 00:12:15.966
That's so funny.

00:12:17.066 --> 00:12:17.515
All right.

00:12:17.515 --> 00:12:19.534
Well let's transition to a little more fun.

00:12:19.534 --> 00:12:21.875
So we covered the main topic that we wanted to talk about.

00:12:21.967 --> 00:12:26.044
Let's, before, before we go, let's, let's play one more game.

00:12:26.052 --> 00:12:28.332
And it's not necessarily a game, but it's just a little bit of fun.

00:12:28.363 --> 00:12:34.212
Every person who's worked in it for any meaningful length of time, they've messed something up somewhere along the way.

00:12:34.243 --> 00:12:35.232
I know I've done it.

00:12:35.248 --> 00:12:40.783
You know, I'll, I'll admit right here that I actually rebooted a production system once during the middle of the production.

00:12:41.423 --> 00:12:42.024
Oops.

00:12:42.024 --> 00:12:44.404
That's, that's a painful lesson to learn.

00:12:44.705 --> 00:12:45.754
But it happens, right?

00:12:45.754 --> 00:12:47.164
It happens to almost everyone.

00:12:47.585 --> 00:12:50.105
So Matt, what's your worst it mess up?

00:12:50.855 --> 00:12:54.455
Yeah, I think, I mean, we've always had those production issues.

00:12:54.460 --> 00:13:05.044
I mean, I've had a countless database issues that I ran into that I didn't understand what I was doing, and I was overconfident and blew up a database, but the one that haunts me still.

00:13:05.080 --> 00:13:06.773
And it was a safe failure.

00:13:06.832 --> 00:13:09.082
But at the nuclear power plant we had.

00:13:09.712 --> 00:13:18.743
Rubidium time servers so that we can basically have exact measurement on anything that happened in the plant, just as far as timing.

00:13:19.163 --> 00:13:31.774
And when we were doing an upgrade of it, I accidentally like pulled out one of the cables so that it lost network connectivity and it all the other servers kind of like just went into this panic mode.

00:13:32.195 --> 00:13:33.534
And I had to reboot.

00:13:34.644 --> 00:13:39.445
and it like kind of had a disruption for like five minutes, but it was okay.

00:13:39.445 --> 00:13:41.095
Like everything kind of failed safe.

00:13:41.095 --> 00:13:43.105
So it was a good like disaster recovery.

00:13:43.434 --> 00:13:45.024
It was a great test, what happened?

00:13:45.565 --> 00:13:51.089
But it was unplanned and you know, at a nuclear power plant, you have to be completely honest all the time.

00:13:51.089 --> 00:13:58.499
So you had to report it, you know, talk through the, you know, the root cause of what the issue happened and like how you could learn from it.

00:13:59.413 --> 00:14:06.104
But it was a good way to validate like the procedures, processes and see how like people panicked or didn't panic.

00:14:06.583 --> 00:14:06.913
Wow.

00:14:06.933 --> 00:14:08.443
That's, that's intense.

00:14:09.134 --> 00:14:09.703
All right.

00:14:09.823 --> 00:14:10.094
All right.

00:14:10.094 --> 00:14:14.759
Well Matt, listen, thank you so much for playing along and sharing all of your wisdom with us.

00:14:14.759 --> 00:14:15.808
Really appreciate it.

00:14:16.109 --> 00:14:18.119
It was a pleasure getting to speak with you today.

00:14:18.418 --> 00:14:26.099
And to all of our listeners, if you're an SRE or if you're in a related role and you wanna be a guest speaker on Ship Talk, please go ahead and send us an email.

00:14:26.349 --> 00:14:30.438
podcast@shiptalk.io and we'll get back to you.

00:14:30.948 --> 00:14:31.729
So that's all for now.

00:14:31.879 --> 00:14:32.479
Until next time.