WEBVTT

00:00:10.919 --> 00:00:11.339
<v Jim Hirschauer>Hey everybody.

00:00:11.339 --> 00:00:13.039
<v Jim Hirschauer>Welcome to Ship Talk, the SRE edition.

00:00:13.500 --> 00:00:15.240
<v Jim Hirschauer>I'm Jim Hirschauer, your host for today.

00:00:15.307 --> 00:00:23.342
<v Jim Hirschauer>Ship Talk is a DevOps podcast brought to you by Harness, the software delivery platform, and the SRE edition focuses on reliability topics.

00:00:23.943 --> 00:00:25.983
<v Jim Hirschauer>My guest today is Brian Finster.

00:00:26.132 --> 00:00:27.408
<v Jim Hirschauer>Thanks for joining us.

00:00:27.542 --> 00:00:30.573
<v Jim Hirschauer>Brian, you're from Defense Unicorns and welcome to the show.

00:00:31.233 --> 00:00:32.442
<v Bryan Finster>Yeah, thanks so much for having me, Jim.

00:00:32.442 --> 00:00:35.506
<v Jim Hirschauer>Brian, why don't you take a minute to share your background.

00:00:35.506 --> 00:00:40.106
<v Jim Hirschauer>Tell us a little bit about defense unicorns, and talk about any other projects you might wanna fill us in on.

00:00:40.616 --> 00:00:40.896
<v Bryan Finster>Sure.

00:00:40.921 --> 00:00:44.072
<v Bryan Finster>So I've been a software engineer for nearly 30 years now.

00:00:44.072 --> 00:00:55.832
<v Bryan Finster>I, I still do that most of the time in supply chain software, which really kind of informs my attitudes towards delivering software because it's, you know, it's, it's a supply chain problem, just a different kind of thing.

00:00:55.832 --> 00:01:03.979
<v Bryan Finster>We're, we're moving for the last several years I've been working on how do we improve the flow of software delivery, focusing on implementing continuous delivery as a workflow.

00:01:04.474 --> 00:01:06.334
<v Bryan Finster>You know, what are the challenges around that?

00:01:06.385 --> 00:01:06.825
<v Bryan Finster>Yeah.

00:01:06.965 --> 00:01:14.837
<v Bryan Finster>And you know, I spent a long time at Walmart doing that, and then I, I got stolen away by defense unicorns and.

00:01:15.796 --> 00:01:20.147
<v Bryan Finster>Defense unicorns, you know, we're trying to solve some really hard software supply chain problems.

00:01:20.147 --> 00:01:32.197
<v Bryan Finster>You know, how do you deliver you know, a hardened secure solution to top secret environments, you know, that are air gap where you can't, you don't have access to normal tools, that sort of thing.

00:01:32.406 --> 00:01:32.557
<v Bryan Finster>Yeah.

00:01:32.617 --> 00:01:34.036
<v Bryan Finster>You know, just really trying to solve.

00:01:34.740 --> 00:01:39.210
<v Bryan Finster>That, that problem space, if you can solve that delivery problem, everything else is easy.

00:01:39.301 --> 00:01:41.191
<v Jim Hirschauer>Yeah, that's a very difficult problem for sure.

00:01:41.850 --> 00:01:45.150
<v Bryan Finster>But that's, I love working really hard problems and this is a real fun one.

00:01:45.510 --> 00:01:45.870
<v Jim Hirschauer>Great.

00:01:45.870 --> 00:01:46.561
<v Jim Hirschauer>Sounds exciting.

00:01:46.650 --> 00:01:48.510
<v Jim Hirschauer>All right, so look, I appreciate the background.

00:01:48.786 --> 00:01:50.531
<v Jim Hirschauer>I feel like you're gonna drop some knowledge on us today.

00:01:51.340 --> 00:01:51.850
<v Jim Hirschauer>I'll try.

00:01:51.911 --> 00:01:53.667
<v Jim Hirschauer>Before we get there, though yeah.

00:01:53.816 --> 00:01:56.037
<v Jim Hirschauer>On this podcast, we'd like to have a little bit of fun first.

00:01:56.037 --> 00:02:00.165
<v Jim Hirschauer>And so what I'd love to do is to hear about your hobbies.

00:02:00.165 --> 00:02:02.025
<v Jim Hirschauer>Everybody has hobbies outside of work.

00:02:02.025 --> 00:02:05.094
<v Jim Hirschauer>And why don't you pick one of your favorite hobbies and, and fill us in.

00:02:05.105 --> 00:02:10.566
<v Bryan Finster>Well, motorcycling I think is, is one of these things that I really geek out about.

00:02:10.566 --> 00:02:18.936
<v Bryan Finster>I mean, I'm like a, you're not gonna catch me saying this brand is the best, cuz I'm like a nerd about motorcycles, new bikes come out and I'll go and try to test ride'em.

00:02:18.936 --> 00:02:21.786
<v Bryan Finster>Not because I wanna buy one just cause like, Hey, what's this new bike like right now?

00:02:21.790 --> 00:02:22.175
<v Bryan Finster>Nice.

00:02:22.175 --> 00:02:22.235
<v Bryan Finster>Yeah.

00:02:22.415 --> 00:02:23.705
<v Bryan Finster>Love motorcycles.

00:02:23.721 --> 00:02:26.259
<v Bryan Finster>And I, I love distance riding.

00:02:26.287 --> 00:02:28.657
<v Bryan Finster>I, I, I ride a BMW GS adventure.

00:02:28.723 --> 00:02:32.936
<v Bryan Finster>And I've taken that thing on thousand mile trips in a day.

00:02:32.936 --> 00:02:34.258
<v Bryan Finster>You know, it's, it's fun.

00:02:34.258 --> 00:02:39.009
<v Bryan Finster>It still kind of gets the adrenaline junkie part of me going as well.

00:02:39.014 --> 00:02:40.509
<v Bryan Finster>I love diving into corners.

00:02:40.960 --> 00:02:41.439
<v Jim Hirschauer>Nice.

00:02:41.439 --> 00:02:41.620
<v Jim Hirschauer>Yeah.

00:02:41.620 --> 00:02:43.419
<v Jim Hirschauer>Well, I don't know a whole lot about motorcycles.

00:02:43.419 --> 00:02:46.780
<v Jim Hirschauer>I definitely know that BMW makes some fantastic motorcycles for sure.

00:02:46.780 --> 00:02:48.340
<v Jim Hirschauer>And it sounds like you have an adventure bike.

00:02:48.759 --> 00:02:50.990
<v Jim Hirschauer>Those are the ones that you can kind of take on road offroad, right?

00:02:51.759 --> 00:02:52.210
<v Bryan Finster>yes.

00:02:52.479 --> 00:02:53.349
<v Bryan Finster>But I'm realistic.

00:02:53.349 --> 00:02:58.090
<v Bryan Finster>They make bikes that are slightly more reliable than Harley Davidson's and much less reliable than Honda's.

00:02:58.090 --> 00:02:59.500
<v Bryan Finster>You just need to know where you're sitting.

00:02:59.740 --> 00:03:00.069
<v Bryan Finster>Okay.

00:03:01.120 --> 00:03:02.680
<v Bryan Finster>In a reliability space.

00:03:03.094 --> 00:03:03.250
<v Bryan Finster>Yeah.

00:03:03.789 --> 00:03:04.449
<v Jim Hirschauer>Good to know.

00:03:04.526 --> 00:03:08.944
<v Jim Hirschauer>So any, any fun stories to share about riding your motorcycle on these long trips?

00:03:08.959 --> 00:03:13.429
<v Bryan Finster>Just, you know, I love getting there and I love going places where there, where I can be challenged.

00:03:13.579 --> 00:03:19.319
<v Bryan Finster>You know, there's a road in north North Carolina, a Tennessee border called the Dragon Us 1 29.

00:03:19.409 --> 00:03:19.889
<v Bryan Finster>I've heard of it.

00:03:19.930 --> 00:03:20.009
<v Bryan Finster>Yeah.

00:03:20.580 --> 00:03:23.340
<v Bryan Finster>318 corners and 11 miles.

00:03:23.520 --> 00:03:23.789
<v Bryan Finster>Yeah.

00:03:23.847 --> 00:03:29.007
<v Bryan Finster>And taking a big adventure bike like that, and a big guy like me, I'm six six, I'm not a small person.

00:03:29.074 --> 00:03:33.516
<v Bryan Finster>And diving into those corners and, and just challenging how do I do this corner better?

00:03:33.516 --> 00:03:34.627
<v Bryan Finster>How do I do this corner better?

00:03:34.632 --> 00:03:36.667
<v Bryan Finster>I mean, it's a lot like how I try to deliver software.

00:03:36.667 --> 00:03:39.347
<v Bryan Finster>It's like, you know, small incremental improvement all the time.

00:03:39.347 --> 00:03:39.826
<v Bryan Finster>You know.

00:03:40.486 --> 00:03:41.512
<v Jim Hirschauer>Awesome, awesome.

00:03:41.512 --> 00:03:42.592
<v Jim Hirschauer>Diving into corners.

00:03:42.711 --> 00:03:43.102
<v Bryan Finster>Yeah.

00:03:43.102 --> 00:03:51.125
<v Bryan Finster>You know, and, this is something that I sometimes try to use this as an analogy because there's habits that you have to develop.

00:03:51.305 --> 00:03:53.194
<v Bryan Finster>Motorcycles are very counterintuitive.

00:03:53.960 --> 00:03:58.069
<v Bryan Finster>In a car, sometimes the safest thing to do is to hit the brakes.

00:03:58.069 --> 00:04:06.080
<v Bryan Finster>You're coming into a corner too fast in a car, you, you wanna slow down and reduce your energy coming into that corner so you don't run off the corner.

00:04:06.560 --> 00:04:13.460
<v Bryan Finster>The problem is, is that if you apply habits you learned in a car to a motorcycle, it doesn't work the same way.

00:04:13.460 --> 00:04:15.620
<v Bryan Finster>You need to understand the physics of the motorcycle.

00:04:15.620 --> 00:04:18.050
<v Bryan Finster>I mean, motorcycles turn by leaning.

00:04:18.860 --> 00:04:19.610
<v Bryan Finster>Yeah, right?

00:04:19.699 --> 00:04:19.910
<v Bryan Finster>Yep.

00:04:20.000 --> 00:04:25.579
<v Bryan Finster>And if you hit the brakes on a motorcycle, the bike stops leaning, it stands up.

00:04:25.579 --> 00:04:32.569
<v Bryan Finster>And so if you're coming into a corner too fast in a motorcycle, you need to just be smooth and just lean harder.

00:04:33.185 --> 00:04:39.935
<v Bryan Finster>And go through the corner because the most dangerous thing you can do is just slam on the brakes, stand the bike up, and they just run right off the corner.

00:04:40.595 --> 00:04:56.367
<v Bryan Finster>And this is a lot like the behaviors I see when people are trying to transition to continuous delivery workflows is, is that they, you know, they'll hit a bump along the way and they'll start slowing down delivery, which means that the delivered batches get bigger.

00:04:56.742 --> 00:05:04.663
<v Bryan Finster>But the safety from CD comes from delivering smaller batches and getting faster feedback and it, it's the same sort of mindset.

00:05:04.663 --> 00:05:06.523
<v Bryan Finster>It's counterintuitive, but you need to.

00:05:07.557 --> 00:05:14.187
<v Bryan Finster>Deliver more frequently in smaller chunks rather than slow everything down and panic because you hit a roadblock.

00:05:14.877 --> 00:05:15.987
<v Jim Hirschauer>Yeah, I love that analogy.

00:05:15.992 --> 00:05:23.067
<v Jim Hirschauer>So, leaning into your, your software delivery practices, that's a great segue into the main topic for today's show.

00:05:23.067 --> 00:05:28.362
<v Jim Hirschauer>You know, the SRE edition of Ship Talk, we like to focus on reliability and resiliency.

00:05:28.377 --> 00:05:44.535
<v Jim Hirschauer>Right now we're having some conversations industry wide, about efficiency and on the show we did last week, I spoke to Matt Schillerstrom about efficiency and I wanna have that similar conversation with you today and maybe expand into the, the software delivery portions of this.

00:05:44.535 --> 00:05:52.305
<v Jim Hirschauer>So, you know, Overall efficiency is a really hot topic right now, and especially given the current economic environment and as companies, yeah, yeah.

00:05:52.305 --> 00:05:58.814
<v Jim Hirschauer>As companies are looking for different ways to reduce their costs and increase productivity, automation, reliability, resiliency.

00:05:58.819 --> 00:06:00.584
<v Jim Hirschauer>These are becoming top priorities.

00:06:01.125 --> 00:06:10.285
<v Jim Hirschauer>So I'd love to hear from you, what's your take on how companies can be successful given the fact that efficiency and reliability, they're often working against each.

00:06:11.470 --> 00:06:18.170
<v Bryan Finster>Well, I think that a mistake a lot of companies make is they say, we're gonna become more efficient so that we can be more productive.

00:06:18.439 --> 00:06:21.139
<v Bryan Finster>And by that they mean generate more output.

00:06:21.350 --> 00:06:21.439
<v Bryan Finster>Mm-hmm.

00:06:21.949 --> 00:06:36.154
<v Bryan Finster>But you know, the perspective I have, like I said, I've been a developer for a long, long time, and except for a small well, It's been a long career except for a portion of time when operational responsibility was stripped away from me.

00:06:36.221 --> 00:06:42.290
<v Bryan Finster>Where for, to improve developer productivity, make support of my application, go to another team.

00:06:42.560 --> 00:06:45.651
<v Bryan Finster>I have had operational responsibility for what I build.

00:06:45.696 --> 00:06:49.295
<v Bryan Finster>Right, and I don't mean I own the infrastructure and all the tools and everything.

00:06:49.295 --> 00:06:55.925
<v Bryan Finster>I mean that I'm building the application and if it breaks, I'm the one getting woken up in the middle of the night to fix the application.

00:06:56.555 --> 00:07:00.096
<v Bryan Finster>And ops is the reason for all of this.

00:07:00.096 --> 00:07:03.516
<v Bryan Finster>We want to become more efficient about how we deliver software.

00:07:04.321 --> 00:07:12.336
<v Bryan Finster>so that we can, you know, make smaller batch sizes of of work so that we can fix production faster.

00:07:12.365 --> 00:07:12.665
<v Bryan Finster>Right.

00:07:12.665 --> 00:07:19.475
<v Bryan Finster>So the smaller batches of work allow us to uncover the pain points preventing us from delivering high quality change.

00:07:19.480 --> 00:07:19.995
<v Bryan Finster>Right?

00:07:20.165 --> 00:07:20.466
<v Bryan Finster>Right.

00:07:20.471 --> 00:07:27.605
<v Bryan Finster>It could be processed, it could be how we test whatever, but shoving down those batch sizes, uncovers all that pain.

00:07:28.206 --> 00:07:29.946
<v Bryan Finster>But to do that, we have to.

00:07:30.425 --> 00:07:54.202
<v Bryan Finster>Anytime we find waste in our flow of delivery, we need to remove that waste handoffs to other teams and lack of access to product information we need, whatever that is, how do we improve the supply chain of communication and optimize our processes so that we can keep shrinking that change set size down so that we can every single day build the muscle memory.

00:07:54.833 --> 00:07:56.663
<v Bryan Finster>Of fixing production.

00:07:56.692 --> 00:07:58.103
<v Bryan Finster>I mean, this is the thing about cd.

00:07:58.103 --> 00:07:59.153
<v Bryan Finster>It's not about speed.

00:07:59.153 --> 00:08:00.833
<v Bryan Finster>It's about building that muscle memory.

00:08:00.833 --> 00:08:02.242
<v Bryan Finster>We are delivering daily.

00:08:02.708 --> 00:08:14.708
<v Bryan Finster>Yeah, hopefully multiple times a day if we're on a high performing or a team, so that when we wake up at three o'clock in the morning, we're Oh, and, and I left this out, and we're only ever using our emergency flow.

00:08:14.767 --> 00:08:23.858
<v Bryan Finster>We're using our hot fix process to deliver every single change because we are validating that our hot fix process works very, very well.

00:08:24.158 --> 00:08:28.658
<v Bryan Finster>We're stressing that system of delivery to uncover.

00:08:29.017 --> 00:08:37.658
<v Bryan Finster>Broke breaks during the normal business hours, so we don't hit those breaks in the middle of the night when something breaks because something will break when we're asleep.

00:08:38.048 --> 00:08:38.437
<v Bryan Finster>Yeah, right.

00:08:38.442 --> 00:08:44.798
<v Bryan Finster>And I focus on CD because I wanna go to sleep at 3 0 5 when I woke up at three o'clock with a system that's down.

00:08:44.908 --> 00:08:45.467
<v Bryan Finster>Right.

00:08:45.638 --> 00:08:48.427
<v Bryan Finster>I've spent too many years on pager.

00:08:48.982 --> 00:08:52.493
<v Bryan Finster>It was too many sleepless nights not to do continuous delivery.

00:08:53.363 --> 00:08:55.043
<v Jim Hirschauer>So you said something really interesting.

00:08:55.043 --> 00:08:56.962
<v Jim Hirschauer>You said that it's about ops.

00:08:56.962 --> 00:09:00.173
<v Jim Hirschauer>It's about being able to recover your systems as quickly as possible.

00:09:00.173 --> 00:09:00.263
<v Jim Hirschauer>Yes.

00:09:00.623 --> 00:09:02.302
<v Jim Hirschauer>But what does that mean for Dev?

00:09:02.302 --> 00:09:08.842
<v Jim Hirschauer>Because at the end of the day, you also said in your time in Dev, you were responsible for what you deployed.

00:09:08.842 --> 00:09:09.143
<v Jim Hirschauer>Right.

00:09:09.148 --> 00:09:10.043
<v Jim Hirschauer>That software that you deploy.

00:09:10.048 --> 00:09:10.302
<v Bryan Finster>I am.

00:09:10.493 --> 00:09:11.692
<v Bryan Finster>I am still a dev.

00:09:11.722 --> 00:09:11.932
<v Bryan Finster>Yeah.

00:09:11.932 --> 00:09:13.942
<v Bryan Finster>And here's just the hard truth.

00:09:14.062 --> 00:09:16.432
<v Bryan Finster>I was talking to Andrew Clay Shafer about this yesterday.

00:09:16.957 --> 00:09:28.538
<v Bryan Finster>This whole thing about coddling developers to take away responsibility for the things that they build so they don't feel the impact of their decisions drives poor quality.

00:09:28.748 --> 00:09:33.817
<v Bryan Finster>If an organization wants high quality software, they will give ownership to teams.

00:09:34.457 --> 00:09:40.727
<v Bryan Finster>And ownership means taking responsibility for the, your architectural and, and tech stack decisions.

00:09:40.758 --> 00:09:40.847
<v Bryan Finster>Mm-hmm.

00:09:41.148 --> 00:09:49.097
<v Bryan Finster>you don't wind up with a team voluntarily building something with five different languages if they also have to fix it in the middle of the night.

00:09:49.217 --> 00:09:49.457
<v Bryan Finster>Right.

00:09:49.607 --> 00:09:53.988
<v Bryan Finster>They're gonna optimize for clean code, clean architecture, easy to understand.

00:09:54.457 --> 00:09:56.283
<v Bryan Finster>Simple and effective.

00:09:56.363 --> 00:09:56.923
<v Bryan Finster>Right?

00:09:56.923 --> 00:09:57.243
<v Bryan Finster>Right.

00:09:57.243 --> 00:10:02.942
<v Bryan Finster>They're gonna optimize for not being woken up and that's what drives quality.

00:10:02.942 --> 00:10:07.472
<v Bryan Finster>And when you separate those two things, you kill that quality feedback loop.

00:10:07.472 --> 00:10:13.113
<v Bryan Finster>And developers don't know that they're breaking stuff and a some will just start playing.

00:10:14.048 --> 00:10:22.388
<v Bryan Finster>But the, I'm a developer, I don't, I have no problem holding my peers accountable for being professionals and taking responsibility for the work that we do.

00:10:22.957 --> 00:10:26.347
<v Jim Hirschauer>And how does that translate into the software delivery life cycle as a whole?

00:10:26.352 --> 00:10:32.798
<v Jim Hirschauer>You mentioned continuous delivery, but when we talk about like reliability and resiliency, that shouldn't start in production, right?

00:10:32.798 --> 00:10:34.028
<v Jim Hirschauer>By the time you get to production.

00:10:34.373 --> 00:10:35.302
<v Jim Hirschauer>It's there, right?

00:10:35.302 --> 00:10:38.903
<v Jim Hirschauer>And so whatever's there is there, and it's gonna be as reliable as you've made it.

00:10:38.903 --> 00:10:40.883
<v Jim Hirschauer>And whatever users are gonna do to it, they'll do.

00:10:40.883 --> 00:10:45.052
<v Jim Hirschauer>So where does it start and how do you make that part of your software delivery life cycle?

00:10:45.442 --> 00:10:47.572
<v Bryan Finster>Well, it starts with your pipeline.

00:10:47.722 --> 00:10:51.503
<v Bryan Finster>The, the purpose of a CD pipeline is not to deliver software.

00:10:51.562 --> 00:10:55.432
<v Bryan Finster>The purpose of a CD pipeline is to prevent bad software from being delivered.

00:10:55.552 --> 00:10:55.763
<v Bryan Finster>Mm-hmm.

00:10:55.883 --> 00:11:00.082
<v Bryan Finster>And so we need to identify what, number one, what is our definition of deliverable.

00:11:01.503 --> 00:11:02.807
<v Bryan Finster>Can we even describe it?

00:11:02.807 --> 00:11:06.857
<v Bryan Finster>If we can't, we already have a problem If we need to have a definition of deliverable.

00:11:07.128 --> 00:11:11.118
<v Bryan Finster>It's a certain level of security, a certain level of performance for what it is we're delivering on.

00:11:11.118 --> 00:11:16.121
<v Bryan Finster>You know, this particular use the problem we're trying to solve, and then codify that in the pipeline.

00:11:16.682 --> 00:11:23.292
<v Bryan Finster>And then we start shipping down the pipeline to find out how wrong we are about defining our definitions in the pipeline and hardening our pipeline.

00:11:23.298 --> 00:11:34.912
<v Bryan Finster>When I, when I've worked with teams in the past on this, I tell'em that, look, you're a product team, you own this business problem and solving this business problem, but your primary product is the pipeline delivering it.

00:11:34.942 --> 00:11:47.126
<v Bryan Finster>Your job is to, whenever something breaks, because it will, your response is, Where do we put something in the pipeline to quickly make that not happen again?

00:11:47.606 --> 00:11:58.346
<v Bryan Finster>You know, and this comes to, we have to design efficient tests that are capable of giving us a signal when things broke, about where it broke so we can fix it quickly before it went to production.

00:11:58.346 --> 00:12:08.047
<v Bryan Finster>And when things break in production, putting a another test in place that will keep efficiently doing that, we need to keep track of how long that takes to run that pipeline.

00:12:08.386 --> 00:12:21.017
<v Bryan Finster>Because if it takes you four hours to run your pipeline, you're not going to, when there's an emergency you're gonna come up with some hot fix process that's not well tested and pour gasoline on a dumpster fire.

00:12:21.856 --> 00:12:23.447
<v Bryan Finster>I know I've done it, right?

00:12:23.451 --> 00:12:23.626
<v Bryan Finster>Yeah.

00:12:23.626 --> 00:12:24.376
<v Bryan Finster>So, yeah.

00:12:24.736 --> 00:12:32.017
<v Bryan Finster>You know, and, and so it's, you're constantly going, okay, we're gonna ship, we're gonna find out what breaks, we're gonna just continue improving the things that break.

00:12:32.456 --> 00:12:38.020
<v Jim Hirschauer>When you put all that together, it sounded like you were saying our pipelines are not just about delivering new features.

00:12:38.020 --> 00:12:38.171
<v Jim Hirschauer>Right.

00:12:38.181 --> 00:12:41.421
<v Jim Hirschauer>They're there to make sure that we're delivering quality software to our end users.

00:12:41.426 --> 00:12:42.471
<v Bryan Finster>They're our safety net.

00:12:42.620 --> 00:12:43.130
<v Bryan Finster>Yeah.

00:12:43.135 --> 00:12:43.140
<v Bryan Finster>Yeah.

00:12:43.581 --> 00:12:45.831
<v Bryan Finster>And, and we are, we own our safety net.

00:12:46.041 --> 00:12:47.691
<v Bryan Finster>We will make mistakes.

00:12:47.961 --> 00:12:50.110
<v Bryan Finster>The automation is there to help us.

00:12:50.571 --> 00:12:52.385
<v Bryan Finster>Avoid making them twice.

00:12:52.446 --> 00:13:04.635
<v Jim Hirschauer>So, from that perspective, it sounds like you're saying that resiliency engineering should be part of this overall process, and not only in production, but baked into your software delivery pipelines as well.

00:13:04.635 --> 00:13:05.176
<v Bryan Finster>Everything.

00:13:05.176 --> 00:13:12.225
<v Bryan Finster>If you're starting a new team, working on a new product, Or even if you're just starting a new service for a product that exists.

00:13:12.225 --> 00:13:12.316
<v Bryan Finster>Mm-hmm.

00:13:12.885 --> 00:13:13.186
<v Bryan Finster>right?

00:13:13.905 --> 00:13:15.515
<v Bryan Finster>Feature zero is your pipeline.

00:13:15.735 --> 00:13:15.826
<v Bryan Finster>Yeah.

00:13:15.855 --> 00:13:27.285
<v Bryan Finster>If you don't have a way to get to production, to invalidate your assumptions about what you're building, everything else you do is waste, and you're just piling defect on top of defect without knowing it.

00:13:28.426 --> 00:13:29.355
<v Jim Hirschauer>You know, it's interesting.

00:13:29.424 --> 00:13:33.184
<v Jim Hirschauer>We were just working with a company out in Australia that Yeah.

00:13:33.190 --> 00:13:35.764
<v Jim Hirschauer>Was releasing a new product, right?

00:13:35.764 --> 00:13:45.514
<v Jim Hirschauer>This is an established company, but they're releasing new product and their take on resiliency engineering was that they needed to get it implemented before they released their new product.

00:13:45.514 --> 00:13:49.985
<v Jim Hirschauer>And that was honestly for me, that was the first time I had ever heard a company make that statement.

00:13:50.284 --> 00:13:55.485
<v Jim Hirschauer>Their opinion was if we launched this new product, and it provides a poor customer experience.

00:13:55.794 --> 00:13:58.044
<v Jim Hirschauer>It's worse than not launching the product at all for them.

00:13:58.284 --> 00:14:01.105
<v Jim Hirschauer>They didn't want to to create that bad first impression.

00:14:01.764 --> 00:14:02.695
<v Bryan Finster>I totally agree.

00:14:02.965 --> 00:14:03.264
<v Bryan Finster>Yeah.

00:14:03.294 --> 00:14:12.461
<v Bryan Finster>But you also need to figure out how I can get this into production to validate some of our assumptions about it running in a production environment without also exposing it to people.

00:14:13.221 --> 00:14:14.721
<v Bryan Finster>In a way that's not good.

00:14:14.721 --> 00:14:15.022
<v Bryan Finster>Right?

00:14:15.381 --> 00:14:18.471
<v Bryan Finster>I mean, you want, you, you, you, you want to deploy.

00:14:18.532 --> 00:14:30.562
<v Bryan Finster>There's, there's many, many times when I'm deploying the production things that nobody, but I know, I mean, it's, it's being tested by the fact that it's not blowing up and dying in that production environment.

00:14:30.611 --> 00:14:36.461
<v Bryan Finster>One of the things I work on is I'm a contributor to minimum cd.org and we have just a list of problems to solve.

00:14:36.986 --> 00:14:37.256
<v Bryan Finster>Right.

00:14:37.256 --> 00:14:39.086
<v Bryan Finster>It's not implementation do CD this way.

00:14:39.086 --> 00:14:44.006
<v Bryan Finster>It's, here's problems to solve that if you solve these problems, now you're doing continuous delivery.

00:14:44.246 --> 00:14:44.336
<v Bryan Finster>Right.

00:14:44.456 --> 00:14:48.506
<v Bryan Finster>One of those problems is production-like test environments, but it's production-like.

00:14:48.527 --> 00:14:49.006
<v Bryan Finster>Yeah.

00:14:49.777 --> 00:14:53.486
<v Bryan Finster>the odds of you having a test environment that matches production are zero.

00:14:53.697 --> 00:14:53.907
<v Bryan Finster>Yeah.

00:14:54.331 --> 00:14:55.177
<v Bryan Finster>Absolutely right.

00:14:55.256 --> 00:14:55.437
<v Bryan Finster>Yeah.

00:14:55.596 --> 00:15:05.996
<v Bryan Finster>And so, you know, getting something out there in a way where you can do some validation that it will actually run and doesn't break anything without hurting anything while you do it, is critical.

00:15:06.417 --> 00:15:11.067
<v Jim Hirschauer>So is this validation, is this strictly the realm of chaos engineering type of solutions?

00:15:11.067 --> 00:15:14.826
<v Jim Hirschauer>Or is there more than that that goes into it?

00:15:14.846 --> 00:15:15.956
<v Bryan Finster>I mean, it's far more than that.

00:15:15.956 --> 00:15:18.927
<v Bryan Finster>Not only is it resilient, but does it perform it, is it functional?

00:15:19.346 --> 00:15:19.586
<v Bryan Finster>Right?

00:15:19.586 --> 00:15:22.167
<v Bryan Finster>I mean, what tests can we throw at it in production?

00:15:22.677 --> 00:15:22.797
<v Bryan Finster>Yeah.

00:15:22.856 --> 00:15:26.126
<v Bryan Finster>To validate its does in production what it should do.

00:15:26.126 --> 00:15:28.437
<v Bryan Finster>I mean, we, there's all sorts of mistakes we can make.

00:15:28.437 --> 00:15:31.136
<v Bryan Finster>We could have configurations set incorrectly.

00:15:31.136 --> 00:15:32.636
<v Bryan Finster>We only find out when we deliver.

00:15:33.626 --> 00:15:34.076
<v Jim Hirschauer>Yeah.

00:15:34.256 --> 00:15:36.356
<v Jim Hirschauer>Are you, so you're advocating testing and production?

00:15:37.226 --> 00:15:37.947
<v Bryan Finster>Absolutely.

00:15:37.947 --> 00:15:41.667
<v Bryan Finster>But testing production doesn't, doesn't mean we only test in production.

00:15:41.672 --> 00:15:43.527
<v Bryan Finster>It means we also test in production.

00:15:44.726 --> 00:15:48.596
<v Bryan Finster>I, I think people really need to understand that all of this is an integrated system.

00:15:48.596 --> 00:15:49.256
<v Bryan Finster>You know, there's this.

00:15:50.067 --> 00:15:52.917
<v Bryan Finster>People see these different words being used.

00:15:52.917 --> 00:15:59.006
<v Bryan Finster>They'll hear agile and DevOps and continuous delivery, and that they'll think that these are all different things and they're not.

00:15:59.636 --> 00:16:03.667
<v Bryan Finster>What we're trying to do is we're trying to implement a system of delivery that gives us.

00:16:04.447 --> 00:16:08.562
<v Bryan Finster>Rapid feedback from idea all the way to delivery, right?

00:16:08.741 --> 00:16:11.922
<v Bryan Finster>Every single step in between rapid feedback that we have a problem.

00:16:12.552 --> 00:16:23.032
<v Jim Hirschauer>Brian, I love your, viewpoint on this holistic way of thinking about software delivery because in reality, this is what our end users, it's what they experience.

00:16:23.032 --> 00:16:30.461
<v Jim Hirschauer>At the end of the day, everything that led up to delivering that software to an end user, it is what their customer experience is.

00:16:31.072 --> 00:16:36.291
<v Bryan Finster>You know, and, and, and by the way support documentation, all of those are part of, those are all features.

00:16:36.500 --> 00:16:37.791
<v Bryan Finster>Those are all part of the product.

00:16:38.211 --> 00:16:39.140
<v Bryan Finster>Make sure they're good.

00:16:39.350 --> 00:16:40.010
<v Jim Hirschauer>Absolutely.

00:16:40.610 --> 00:16:41.660
<v Jim Hirschauer>Yeah, I couldn't agree more.

00:16:41.660 --> 00:16:46.370
<v Jim Hirschauer>There's so many times where I've personally been looking at docs to try and figure out how do I use this thing, right?

00:16:46.370 --> 00:16:50.706
<v Jim Hirschauer>And what I'm looking for isn't documented All right, great.

00:16:50.706 --> 00:16:58.495
<v Jim Hirschauer>So listen, let's take a quick transition here because on this podcast, we also do another fun segment at the end.

00:16:58.495 --> 00:17:07.227
<v Jim Hirschauer>So, one of the things I love to ask and, and I found that people really do like talking about, because at the end of the day, we're all humans and we, we mess up.

00:17:07.227 --> 00:17:10.450
<v Jim Hirschauer>And when we work in IT, sometimes we have some pretty interesting mess up.

00:17:10.450 --> 00:17:13.994
<v Jim Hirschauer>So, Brian, I'd love to know, what's your worst it mess up?

00:17:15.345 --> 00:17:20.255
<v Bryan Finster>Well, I had to write down a list because I've been doing this for a while and I've, I've messed up right.

00:17:20.255 --> 00:17:30.095
<v Bryan Finster>And this, this particular mess up is one of the reasons why I say that nobody should ever have access to production at all, except through a pipeline.

00:17:30.095 --> 00:17:31.144
<v Bryan Finster>Mm-hmm.

00:17:31.194 --> 00:17:31.684
<v Bryan Finster>Okay.

00:17:31.684 --> 00:17:32.005
<v Bryan Finster>Okay.

00:17:32.164 --> 00:17:37.404
<v Bryan Finster>Because I was on a production system in the wrong path.

00:17:38.025 --> 00:17:39.105
<v Bryan Finster>I was in bin.

00:17:39.214 --> 00:17:40.055
<v Bryan Finster>Yeah, right.

00:17:40.084 --> 00:17:44.195
<v Bryan Finster>I mistyped a command cause I was trying to clear a log file, oh.

00:17:44.826 --> 00:17:47.976
<v Bryan Finster>And I was like, why is it taking so long to delete that log file?

00:17:50.711 --> 00:17:53.010
<v Jim Hirschauer>Did, did you delete your whole operating system

00:17:53.015 --> 00:17:57.815
<v Bryan Finster>Well it was, it was the primary application that consisted about 800 different discreet programs.

00:17:57.815 --> 00:17:58.174
<v Bryan Finster>Right.

00:17:58.234 --> 00:18:00.335
<v Bryan Finster>So it was the, the warehouse management system?

00:18:00.365 --> 00:18:00.545
<v Bryan Finster>Yeah.

00:18:00.545 --> 00:18:00.815
<v Bryan Finster>Right.

00:18:01.085 --> 00:18:03.575
<v Bryan Finster>So I was deleting the warehouse management system.

00:18:03.634 --> 00:18:03.964
<v Bryan Finster>Oh, wow.

00:18:04.063 --> 00:18:06.553
<v Bryan Finster>From that, from that box, it was out on the edge.

00:18:06.553 --> 00:18:14.833
<v Bryan Finster>Now I rapidly went to another box, you know, another DC and inserted FTPing applications across and hoping that we had no version.

00:18:16.073 --> 00:18:20.472
<v Bryan Finster>We were gonna kill and then crossed my fingers and then just didn't tell anybody.

00:18:20.833 --> 00:18:23.472
<v Bryan Finster>Never never touch production

00:18:24.643 --> 00:18:25.482
<v Jim Hirschauer>So did it work out?

00:18:25.482 --> 00:18:26.262
<v Jim Hirschauer>Did you get away with it?

00:18:26.772 --> 00:18:27.643
<v Bryan Finster>I got away with it, yeah.

00:18:27.775 --> 00:18:39.535
<v Bryan Finster>You know, there was another time when I didn't get away with it, but my tech lead covered for me where I, I deleted an application that only existed in one distribution center that calculated tax for Argentina.

00:18:39.535 --> 00:18:44.799
<v Bryan Finster>So for 30 days, Argentina didn't have any tax calculation going on, and that was, yeah.

00:18:44.950 --> 00:18:45.339
<v Bryan Finster>Oh my.

00:18:45.339 --> 00:18:46.660
<v Bryan Finster>Never touch production.

00:18:48.250 --> 00:18:48.940
<v Jim Hirschauer>Yeah, I agree.

00:18:48.940 --> 00:19:03.420
<v Jim Hirschauer>So I used to be a systems administrator a long time ago, and we worked in those production systems all the time and it, it made me nervous even though I was really comfortable in there and I had, you know, multiple factor authentication to get the root user on Unix.

00:19:04.410 --> 00:19:04.920
<v Jim Hirschauer>Wow.

00:19:04.920 --> 00:19:06.089
<v Jim Hirschauer>Did you have to be careful in there?

00:19:06.345 --> 00:19:14.554
<v Bryan Finster>My office mate at a previous company he was tech lead for public stores and he thought he was deleting the inventory table from a development system.

00:19:14.554 --> 00:19:23.375
<v Bryan Finster>He deleted a production inventory table and from that point on, I always had a different color screen on my terminal emulator for production versus development.

00:19:23.434 --> 00:19:24.484
<v Jim Hirschauer>Oh, that is a brilliant.

00:19:25.164 --> 00:19:25.914
<v Jim Hirschauer>Absolutely.

00:19:25.914 --> 00:19:26.365
<v Jim Hirschauer>So, yeah.

00:19:26.395 --> 00:19:28.046
<v Jim Hirschauer>Lesson for everyone listening.

00:19:28.526 --> 00:19:28.615
<v Jim Hirschauer>Yeah.

00:19:28.645 --> 00:19:36.476
<v Jim Hirschauer>If you have any access to production systems and non-production systems, like Triple Verify which one you're on before running that command.

00:19:36.506 --> 00:19:38.185
<v Jim Hirschauer>Especially if, if it's a destructive command.

00:19:38.191 --> 00:19:45.536
<v Bryan Finster>Absolutely visual verification somewhere PROD, you know, giant letters or screen colored something cuz you will make a mistake.

00:19:46.411 --> 00:19:47.161
<v Jim Hirschauer>Absolutely.

00:19:47.461 --> 00:19:47.881
<v Jim Hirschauer>All right.

00:19:47.881 --> 00:19:50.045
<v Jim Hirschauer>Well, Brian it's been wonderful talking to you.

00:19:50.075 --> 00:19:52.934
<v Jim Hirschauer>Thank you so much for being our guest on the show today.

00:19:52.940 --> 00:19:57.942
<v Jim Hirschauer>Again, love your idea of looking at software delivery from a holistic perspective.

00:19:57.942 --> 00:19:58.873
<v Jim Hirschauer>Completely agree with it.

00:19:59.113 --> 00:20:06.727
<v Jim Hirschauer>Thank you for sharing your humanity with us and, and, you know, the, the mess up that you've had.

00:20:06.727 --> 00:20:07.429
<v Jim Hirschauer>We've all done it.

00:20:07.429 --> 00:20:08.148
<v Jim Hirschauer>So yeah.

00:20:08.148 --> 00:20:10.432
<v Jim Hirschauer>We're all, we're all in the same boat with you.

00:20:10.623 --> 00:20:12.113
<v Jim Hirschauer>Yeah, yeah, for sure.

00:20:12.292 --> 00:20:23.342
<v Jim Hirschauer>And to everyone listening if you're interested in being a guest on Ship Talk, if you're an SRE or if you're in a DevOps related role, Just feel free to send us an email.

00:20:23.419 --> 00:20:27.048
<v Jim Hirschauer>Send that email to podcast at ship talk.io and we'll get back to you.

00:20:27.588 --> 00:20:28.249
<v Jim Hirschauer>That's all for now.

00:20:28.308 --> 00:20:28.909
<v Jim Hirschauer>Until next time.

00:20:29.179 --> 00:20:29.628
<v Bryan Finster>Thanks, Jim.

00:20:30.019 --> 00:20:30.499
<v Jim Hirschauer>Thanks Brian.