WEBVTT

00:00:11.269 --> 00:00:11.630
<v Jim Hirschauer>Alright.

00:00:11.644 --> 00:00:12.480
<v Jim Hirschauer>Welcome to ShipTalk.

00:00:12.900 --> 00:00:14.880
<v Jim Hirschauer>I'm Jim Hirschauer, your host for today.

00:00:15.460 --> 00:00:19.839
<v Jim Hirschauer>ShipTalk is a DevOps podcast brought to you by Harness, the software delivery platform.

00:00:20.399 --> 00:00:26.699
<v Jim Hirschauer>And we usually talk about reliability topics, but today we're gonna be talking about developer experience.

00:00:27.315 --> 00:00:30.914
<v Jim Hirschauer>So my guest today is Kyle Shelton, who's a consultant for Toyota Racing.

00:00:31.274 --> 00:00:32.295
<v Jim Hirschauer>Kyle, welcome to the show.

00:00:33.284 --> 00:00:34.545
<v Kyle Shelton>Hey, thanks for having me.

00:00:34.664 --> 00:00:35.457
<v Jim Hirschauer>Yeah, absolutely.

00:00:35.523 --> 00:00:39.063
<v Jim Hirschauer>Kyle, could you please take a minute to share your background and what you're up to these days?

00:00:40.262 --> 00:00:40.533
<v Kyle Shelton>Yeah.

00:00:40.533 --> 00:00:44.103
<v Kyle Shelton>So I, I got started in the technical field early on.

00:00:44.103 --> 00:00:49.954
<v Kyle Shelton>My, my father owned a telephone company, so I've been pulling Cat three, cat five cable since I was about 10, 11 years old.

00:00:49.954 --> 00:00:52.823
<v Kyle Shelton>And my first real job was with Verizon Wireless.

00:00:52.918 --> 00:01:01.531
<v Kyle Shelton>I was a telecom technical specialists, and I also helped build the nation's first voiceover LTE network with multi engineering.

00:01:01.545 --> 00:01:05.471
<v Kyle Shelton>I got, My career started in the cloud as a network engineer.

00:01:05.471 --> 00:01:08.625
<v Kyle Shelton>I got a degree in network engineering from Westwood.

00:01:08.655 --> 00:01:22.644
<v Kyle Shelton>And you know, I had to shift to the cloud when I was working with a startup and they decided to, you know, do what a lot of companies are doing now, and that's get rid of all of their on-prem and co-location data centers and go 100% cloud native.

00:01:22.644 --> 00:01:31.503
<v Kyle Shelton>So basically my VP of engineering said either get an AWS cert and have a job or, or go find another job if you wanna be a network engineer.

00:01:31.503 --> 00:01:38.677
<v Kyle Shelton>So from there that's, that's how I got my start in, in cloud and DevOps and that was my first DevOps engineer job.

00:01:38.677 --> 00:01:52.841
<v Kyle Shelton>And I, I, I went from being strictly on the operations systems admin, network engineering side of the house to now getting more involved with the software developers and, and you know improving the developer experience.

00:01:52.841 --> 00:01:57.296
<v Kyle Shelton>And so currently I am supporting the developers for Toyota racing.

00:01:57.343 --> 00:02:01.209
<v Kyle Shelton>I spent some time with AWS and Splunk, as well as a SRE and DevOps engineer.

00:02:01.879 --> 00:02:02.629
<v Jim Hirschauer>Fantastic.

00:02:02.629 --> 00:02:03.620
<v Jim Hirschauer>Thanks for sharing that.

00:02:03.688 --> 00:02:10.288
<v Jim Hirschauer>Kyle, I think you're familiar with the format of the show, and we don't like to jump straight into the, the beefy tech topics here.

00:02:10.288 --> 00:02:12.748
<v Jim Hirschauer>We like to start off with something I like to call just for fun.

00:02:13.209 --> 00:02:17.318
<v Jim Hirschauer>So for this segment, Kyle, I'd love for you to share what's your favorite hobby outside of tech?

00:02:18.399 --> 00:02:20.287
<v Kyle Shelton>Yeah, so great question.

00:02:20.307 --> 00:02:22.621
<v Kyle Shelton>I actually love ice fishing.

00:02:22.860 --> 00:02:24.901
<v Kyle Shelton>And you say, well, don't you live in Texas?

00:02:24.901 --> 00:02:25.980
<v Kyle Shelton>Where can you go ice fishing?

00:02:25.985 --> 00:02:26.221
<v Kyle Shelton>Right.

00:02:26.221 --> 00:02:34.354
<v Kyle Shelton>Well actually, when I was working at Verizon, I got shipped out to Colorado Springs, and that's where I experienced ice fishing up there in the high Alpine Lakes.

00:02:34.354 --> 00:02:37.200
<v Kyle Shelton>And so I'll be honest, my favorite thing to do is fishing.

00:02:37.231 --> 00:03:05.640
<v Kyle Shelton>Being up in the mountains and yeah sitting over a ice hole at negative 10 degrees you know, freezing your butt off and, and the work it takes with the l with the higher elevation too, there's not a lot of oxygen, so, you know, there's, on some of the lakes you can't drive a four-wheeler or anything, so you gotta trek all your gear out into the middle of the ice and hope that they'll come and bite your bait, but, You know, the actual fishing itself is kind of like what I, I think of when, when I fished as a kid on the lake dock, right?

00:03:05.640 --> 00:03:10.110
<v Kyle Shelton>With a hotdog and a little brim, and you just throw the hotdog down there and the fish comes up and get it.

00:03:10.116 --> 00:03:17.256
<v Kyle Shelton>And that's kind of like what it is in this format because you're on a 30, 40 foot lake and it's crystal clear water.

00:03:17.276 --> 00:03:20.336
<v Kyle Shelton>These are the lakes that sit at the bottom of the continental divide, right?

00:03:20.336 --> 00:03:23.486
<v Kyle Shelton>And the scenery alone is what kind of mesmerizes me.

00:03:23.486 --> 00:03:25.366
<v Kyle Shelton>But yeah, you're in a hut.

00:03:25.366 --> 00:03:34.061
<v Kyle Shelton>You drill a hole and you watch down and you'll see, you know, Big old trout come and you got a little bitty pole with a little bitty line, and you have to be really finesse.

00:03:34.061 --> 00:03:39.700
<v Kyle Shelton>You can't, you know, jerk'em too hard or, or pull'em out of the hole too fast, right?

00:03:39.700 --> 00:03:40.751
<v Kyle Shelton>Cuz they'll just break off.

00:03:40.751 --> 00:03:42.951
<v Kyle Shelton>So it's it's, it's, it's a lot of fun.

00:03:42.978 --> 00:03:43.888
<v Kyle Shelton>It's kind of crazy.

00:03:43.888 --> 00:03:52.442
<v Kyle Shelton>Your body kind of gets shocked after you're done and you're real tired because of all the Like, I guess exposure to the, to the frigid temperatures.

00:03:52.442 --> 00:03:59.110
<v Kyle Shelton>I do have all the gear, we have heaters and stuff, but still, I mean, it's, it's pretty intense and it's just it's one of the things me and my wife like to do.

00:03:59.469 --> 00:03:59.830
<v Kyle Shelton>Cool.

00:03:59.830 --> 00:04:00.460
<v Kyle Shelton>Go up there.

00:04:00.460 --> 00:04:06.304
<v Kyle Shelton>We, we, we didn't get to go up this year because we got a little one coming here in a couple weeks, but we'll definitely be out there next year.

00:04:06.818 --> 00:04:07.239
<v Jim Hirschauer>Awesome.

00:04:07.239 --> 00:04:09.389
<v Jim Hirschauer>Well, I've never gone ice fishing before.

00:04:09.389 --> 00:04:11.038
<v Jim Hirschauer>I've done other types of fishing.

00:04:11.109 --> 00:04:16.588
<v Jim Hirschauer>For somebody who's never done it before, what are like your top three piece of advice if they want to get started with ice fishing?

00:04:17.439 --> 00:04:23.019
<v Kyle Shelton>Yeah, so I would research where you are and always look at safety measures.

00:04:23.069 --> 00:04:23.939
<v Kyle Shelton>It's very dangerous.

00:04:23.944 --> 00:04:27.538
<v Kyle Shelton>Obviously, you can die through hypothermia if you were to follow through.

00:04:27.538 --> 00:04:32.778
<v Kyle Shelton>So make sure you're familiar with the safety and what, what is safe and what is not safe for the ice.

00:04:32.846 --> 00:04:38.372
<v Kyle Shelton>Number two, I would go with someone that knows what they're doing first, because like I said, there's a lot to it.

00:04:38.384 --> 00:04:48.997
<v Kyle Shelton>Whether it's, you know what type of auger to buy what type of gear to use, you know, definitely don't just buy a bunch of stuff and go out there before going out with a guide or something just to kind of learn the basics.

00:04:49.052 --> 00:04:53.283
<v Kyle Shelton>And then the third thing, which was my biggest mistake was, you don't need a cooler.

00:04:53.283 --> 00:05:00.641
<v Kyle Shelton>So funny story the first time I went I'm up there and I, I went out for a weekend with some of the guys I work with up there.

00:05:00.641 --> 00:05:02.411
<v Kyle Shelton>These are all native Coloradans.

00:05:02.411 --> 00:05:07.000
<v Kyle Shelton>And you know, they pull up in the RV and I roll out and I'm thinking, all right, it's fishing.

00:05:07.031 --> 00:05:08.680
<v Kyle Shelton>We gotta have a cooler full of beer, right?

00:05:08.680 --> 00:05:14.050
<v Kyle Shelton>And so I literally rolled the cooler on top of the ice and, and met'em out there.

00:05:14.050 --> 00:05:18.190
<v Kyle Shelton>And they looked at me and they kind of looked at each other and they were like, what are you doing, dude?

00:05:18.370 --> 00:05:20.411
<v Kyle Shelton>And I was like, well, we're, we're fishing, right?

00:05:20.411 --> 00:05:21.821
<v Kyle Shelton>We gotta have a cooler full of beer.

00:05:21.821 --> 00:05:24.190
<v Kyle Shelton>And they're like, you know, you're like standing on a cooler, right?

00:05:24.196 --> 00:05:28.471
<v Kyle Shelton>You don't need a cooler to keep the beer cold because you're on an ice block buddy.

00:05:28.800 --> 00:05:32.584
<v Kyle Shelton>And so the biggest thing is that if you do go ice fishing, do not bring a cooler.

00:05:32.915 --> 00:05:33.754
<v Kyle Shelton>You do not need it.

00:05:33.814 --> 00:05:34.834
<v Jim Hirschauer>That's great advice.

00:05:35.285 --> 00:05:36.725
<v Jim Hirschauer>You know, the safety advice as well.

00:05:37.745 --> 00:05:37.954
<v Kyle Shelton>Yeah.

00:05:37.954 --> 00:05:38.165
<v Kyle Shelton>Yeah.

00:05:38.165 --> 00:05:39.725
<v Kyle Shelton>The safety advice is big, right?

00:05:39.725 --> 00:05:39.814
<v Kyle Shelton>Yeah.

00:05:39.819 --> 00:05:45.214
<v Kyle Shelton>You can get little ice pick things and, and understand like if you do fall through, you know, don't panic.

00:05:45.440 --> 00:05:52.014
<v Kyle Shelton>It is crazy because although you might think it, it's thick throughout you can have spots that are not safe.

00:05:52.014 --> 00:05:56.817
<v Kyle Shelton>And so I would definitely be aware of all the safety hazards and conditions before you go out there.

00:05:56.817 --> 00:05:57.927
<v Jim Hirschauer>For sure, for sure.

00:05:58.007 --> 00:06:02.807
<v Jim Hirschauer>So as a fisherman, every fisherman has their, their fisherman stories about the one that got away.

00:06:03.048 --> 00:06:06.108
<v Jim Hirschauer>Do you have any great fisherman tales about your one that got away?

00:06:07.367 --> 00:06:08.088
<v Kyle Shelton>Yeah.

00:06:08.214 --> 00:06:09.504
<v Kyle Shelton>Yeah, it's actually my wife.

00:06:09.504 --> 00:06:13.910
<v Kyle Shelton>So the last trip we went on she had what I would call a master angler.

00:06:13.910 --> 00:06:23.093
<v Kyle Shelton>So I've caught one, I've got it hanging on the wall over there, and it was you know, the day before I was moving back to Texas, from Colorado, I went ice fishing one last time.

00:06:23.093 --> 00:06:27.012
<v Kyle Shelton>And I caught that fish basically 10 minutes before sundown.

00:06:27.012 --> 00:06:28.812
<v Kyle Shelton>I was out there all day, caught nothing.

00:06:29.548 --> 00:06:35.307
<v Kyle Shelton>Didn't get a bite was like, well, this is how I'm gonna leave Colorado, you know, with the worst fishing day of my life.

00:06:35.307 --> 00:06:41.367
<v Kyle Shelton>And, and sure enough, you know, it, it only took one bite and I caught a pretty decent size one that night.

00:06:41.367 --> 00:06:47.995
<v Kyle Shelton>And then I caught the one that I, that I have mounted on my wall which was a, it was about eight and a half pounds, 25 inches.

00:06:47.995 --> 00:06:50.935
<v Kyle Shelton>It's a master angler, certified master angler rainbow trout.

00:06:50.935 --> 00:06:56.177
<v Kyle Shelton>And so last year my wife she had one about that same size all the way up to the hole and she.

00:06:56.987 --> 00:06:58.547
<v Kyle Shelton>You know, we got really excited.

00:06:58.547 --> 00:07:00.646
<v Kyle Shelton>I got more excited than she was right.

00:07:00.646 --> 00:07:03.557
<v Kyle Shelton>And I flipped the ice hut up and I was like, oh.

00:07:03.557 --> 00:07:06.406
<v Kyle Shelton>And then she got right to the top and it, and it broke off.

00:07:06.406 --> 00:07:10.009
<v Kyle Shelton>But after that she looked at me and she's like, okay, I get it.

00:07:10.470 --> 00:07:11.819
<v Kyle Shelton>I see why you're so obsessed now.

00:07:12.425 --> 00:07:20.699
<v Kyle Shelton>She's like, it's not, it's not the catching, it is literally the one that gets away and that like, you're gonna catch that fish one day no matter what.

00:07:20.699 --> 00:07:22.745
<v Kyle Shelton>Even if you spend your whole life trying to get it.

00:07:22.745 --> 00:07:28.928
<v Kyle Shelton>But you know, I can relate on the same note with that too, is like when I was a kid, the largest bass I've ever.

00:07:29.213 --> 00:07:29.814
<v Kyle Shelton>Scene.

00:07:29.853 --> 00:07:33.033
<v Kyle Shelton>I pulled it right up to the boat and I did what you're not supposed to do.

00:07:33.033 --> 00:07:36.963
<v Kyle Shelton>I hoed it and tried to get it up in the boat with a little, you know, with the wrong tackle.

00:07:36.963 --> 00:07:39.812
<v Kyle Shelton>And I, I broke it off and I will never do that again.

00:07:39.903 --> 00:07:40.302
<v Kyle Shelton>Yeah.

00:07:40.312 --> 00:07:40.742
<v Kyle Shelton>So, yeah.

00:07:40.747 --> 00:07:40.802
<v Kyle Shelton>Yeah.

00:07:40.862 --> 00:07:41.523
<v Kyle Shelton>Lessons learned.

00:07:41.523 --> 00:07:42.122
<v Kyle Shelton>A few of them.

00:07:42.273 --> 00:07:42.543
<v Jim Hirschauer>Yeah.

00:07:42.622 --> 00:07:42.942
<v Jim Hirschauer>Yeah.

00:07:43.682 --> 00:07:44.103
<v Jim Hirschauer>All right.

00:07:44.108 --> 00:07:45.423
<v Jim Hirschauer>Well, thanks for sharing that story.

00:07:45.632 --> 00:07:46.473
<v Jim Hirschauer>Interesting stuff.

00:07:46.473 --> 00:07:46.742
<v Jim Hirschauer>Yeah.

00:07:47.163 --> 00:07:49.232
<v Jim Hirschauer>So let's, let's talk tech a little bit.

00:07:49.247 --> 00:07:57.749
<v Jim Hirschauer>You mentioned, you mentioned in your intro that you were into developer experience, and you know, that's one of those topics today that's just so important.

00:07:57.798 --> 00:08:01.678
<v Jim Hirschauer>We're we're asking developers to do so much and mm-hmm.

00:08:01.759 --> 00:08:05.119
<v Jim Hirschauer>You know, we're shifting more and more left, shifting it towards developers.

00:08:05.124 --> 00:08:07.819
<v Jim Hirschauer>And developers need a great developer experience.

00:08:07.819 --> 00:08:13.627
<v Jim Hirschauer>They need the right tooling to help them to not toil, to not waste time to not burn out.

00:08:13.627 --> 00:08:13.838
<v Jim Hirschauer>Right.

00:08:14.468 --> 00:08:18.627
<v Jim Hirschauer>And I'd love to hear about whatever you're working on in that area of developer experience.

00:08:19.528 --> 00:08:20.038
<v Kyle Shelton>Yeah.

00:08:20.088 --> 00:08:24.617
<v Kyle Shelton>Developers are, I would say, every business's golden goose, right?

00:08:24.617 --> 00:08:29.442
<v Kyle Shelton>Like they are producing your products solutions that you sell to your customers, right?

00:08:29.442 --> 00:08:37.336
<v Kyle Shelton>And so the, the more golden eggs that they can produce the better your company and your products evolution are gonna be.

00:08:37.336 --> 00:08:47.980
<v Kyle Shelton>So it's natural in my mind to make sure that your golden gooses are always producing golden eggs as fast and as reliable and as secure as possible, right?

00:08:47.980 --> 00:08:49.571
<v Kyle Shelton>You gotta keep'em away from the foxes.

00:08:49.571 --> 00:08:59.375
<v Kyle Shelton>And I'm from Texas and so I like to make farm references, but you gotta think of, you know, If you got chickens that are, that are, that are producing eggs, you want to keep'em in a coop.

00:08:59.375 --> 00:09:02.985
<v Kyle Shelton>You want to keep'em away from the foxes and you want to keep'em fed and watered and happy.

00:09:02.985 --> 00:09:03.255
<v Kyle Shelton>Right?

00:09:03.255 --> 00:09:04.635
<v Kyle Shelton>And they keep on producing eggs.

00:09:04.640 --> 00:09:20.807
<v Kyle Shelton>And so same concepts kind of go with the developer experience and, and one of the things that I'm really passionate about is build internal platforms and throughout my career in some form of another I've helped work on or build internal external platforms.

00:09:20.807 --> 00:09:38.437
<v Kyle Shelton>You help build a platform to either build a product on or build a tool or, or whatever, but it's, it's kind of like the platforms that help the builders build are some of your most important systems, and oftentimes they get neglected the most, right?

00:09:38.437 --> 00:09:43.022
<v Kyle Shelton>And they're the most, you know, Rube Goldberg like strung together machines.

00:09:43.022 --> 00:09:58.544
<v Kyle Shelton>Whereas if you spend a little bit more time optimizing those platforms and building those platforms so that the developers aren't spending a lot of time messing with the platform itself versus building whatever they're supposed to be building the results are astounding.

00:09:58.544 --> 00:10:02.224
<v Kyle Shelton>I mean, and you look at these cloud native companies that were able to grow.

00:10:02.229 --> 00:10:04.153
<v Kyle Shelton>And like if you look at a company like Spotify, right?

00:10:04.153 --> 00:10:07.068
<v Kyle Shelton>And how that, that product just took off.

00:10:07.068 --> 00:10:10.849
<v Kyle Shelton>And boom, you know, overnight they revolutionized music, right?

00:10:10.849 --> 00:10:17.871
<v Kyle Shelton>And they're having to hire thousands of developers because they have to keep up with the demand of the product and be able to sustain the business.

00:10:17.895 --> 00:10:22.211
<v Kyle Shelton>And one of the things they did was they created backstage.io and that was their developer portal.

00:10:22.211 --> 00:10:26.821
<v Kyle Shelton>And what they did is they streamlined how developers could build.

00:10:27.201 --> 00:10:36.585
<v Kyle Shelton>And my role in that is as a DevOps engineer, I build that platform for them to build their features for the business.

00:10:36.600 --> 00:10:37.796
<v Kyle Shelton>And I help them.

00:10:37.796 --> 00:10:49.855
<v Kyle Shelton>And, and I think the way to do this at scale is to have a portal or a single source for the developers to go to, to either learn about the landscape.

00:10:49.860 --> 00:10:53.125
<v Kyle Shelton>I think a lot of companies get onboarding wrong.

00:10:53.186 --> 00:11:05.519
<v Kyle Shelton>They might have a high churn of developers because of the market and just the nature of the tech industry and they spend a lot more time than they should on getting developers to where they're actually doing the work.

00:11:05.519 --> 00:11:15.250
<v Kyle Shelton>Because, you know, whenever you start a job, you've got that kind of grace period where you don't know anything about the environment, you don't know anything about the software patterns that the company's using.

00:11:15.250 --> 00:11:23.155
<v Kyle Shelton>You don't know anything about the monitoring and observability, and then, you know, there's a lot that you have to learn in order to be successful as a developer.

00:11:23.178 --> 00:11:38.928
<v Kyle Shelton>And so if you can build a portal or a platform that takes care of, that takes care of your software patterns that you can implement, because I know that's another problem that you might have, is you might have 17 different patterns and you have 17 different skillsets and nobody's on the same page.

00:11:38.990 --> 00:11:40.460
<v Kyle Shelton>That's how those things can come.

00:11:40.460 --> 00:12:00.365
<v Kyle Shelton>So you know, making the builder experience a lot more quick to be able to develop products or, or standardize your development patterns or I've seen some stories with companies that are leveraging backstage to make cost optimization kind of gamified by creating like, Hey, here's your spend score.

00:12:00.365 --> 00:12:10.048
<v Kyle Shelton>You're optimized, you know, up to 98%, and I know Harness has a really cool cost management tool that can help you with looking at right-sizing information.

00:12:10.048 --> 00:12:20.938
<v Kyle Shelton>As a TAM, when I worked at AWS, That was the first thing I did because to me that's a low hanging fruit and you know, you can go into Trusted Advisor in AWS and they'll give you all your right sizing recommendations.

00:12:20.989 --> 00:12:32.578
<v Kyle Shelton>I think each cloud is different and that's why I like Harness because we have multiple environments and we might be using multiple providers and being able to aggregate that I really like how you can do that with Harness.

00:12:32.578 --> 00:12:38.804
<v Kyle Shelton>But yeah, I think, you know, gamifying that and showing your developers that say, Hey, if you build this right, you get a score.

00:12:39.164 --> 00:12:42.975
<v Kyle Shelton>Maybe you can do pizza parties or something for the highest score each quarter.

00:12:42.975 --> 00:12:51.315
<v Kyle Shelton>You know I found that gamifying and putting things in competition produces results right, because it kind of gives people motivation to do better.

00:12:51.554 --> 00:12:52.004
<v Kyle Shelton>Absolutely.

00:12:52.062 --> 00:12:56.111
<v Kyle Shelton>And, and so yeah, so on that I'm obsessed with the building.

00:12:56.292 --> 00:12:56.711
<v Kyle Shelton>Yeah,

00:12:56.772 --> 00:12:56.981
<v Jim Hirschauer>yeah.

00:12:56.981 --> 00:12:57.581
<v Jim Hirschauer>On, sorry, go ahead.

00:12:57.586 --> 00:13:00.761
<v Jim Hirschauer>Along those lines, you know, along the lines of results, right?

00:13:00.761 --> 00:13:01.631
<v Jim Hirschauer>So do you have any mm-hmm.

00:13:01.876 --> 00:13:06.042
<v Jim Hirschauer>Examples or stories for us of results that you've seen?

00:13:06.101 --> 00:13:10.211
<v Jim Hirschauer>Positive results for the business that you've seen, or for the developers themselves?

00:13:10.211 --> 00:13:15.192
<v Jim Hirschauer>It could be mental health, it could be in velocity, whatever the tangible benefits might be.

00:13:17.027 --> 00:13:21.287
<v Jim Hirschauer>What are the real benefits that come from really investing heavily in developer experience?

00:13:22.221 --> 00:13:42.871
<v Kyle Shelton>There's a white paper that AWS did with Toyota Motors, North America and their chauffeur platform and, and just the velocity of onboarding new developers and they were kind of forced this, and a lot of people are forced to this because traditionally, pre covid, everybody's in the office, you had everybody in one place and you could kind of enforce things better in an office versus.

00:13:43.231 --> 00:13:45.211
<v Kyle Shelton>The world shuts down everybody's remote.

00:13:45.241 --> 00:13:51.366
<v Kyle Shelton>Okay, well we have to create a whole new remote working solution.

00:13:51.376 --> 00:14:00.724
<v Kyle Shelton>And so if you look at what they did the cost savings, the ability to move fast and pivot fast from a POC to like an actual working solution.

00:14:01.293 --> 00:14:03.038
<v Kyle Shelton>And, also security-wise too.

00:14:03.038 --> 00:14:10.042
<v Kyle Shelton>They're able to quickly implement their security orchestration and patterns because all the development goes through one source of truth.

00:14:10.042 --> 00:14:11.297
<v Kyle Shelton>So you can get in front of that.

00:14:11.322 --> 00:14:17.832
<v Kyle Shelton>One of the cool things that we did at Amazon, I always try to replicate, what is done there because it's a proven system that works.

00:14:17.837 --> 00:14:26.352
<v Kyle Shelton>And I think their onboarding system, like there was no question of where I was, what I'm doing, and what I need to do to get to where I could do my job.

00:14:26.352 --> 00:14:26.623
<v Kyle Shelton>Right.

00:14:26.628 --> 00:14:26.832
<v Kyle Shelton>And I.

00:14:26.938 --> 00:14:30.083
<v Kyle Shelton>You can implement a system like that with a developer portal.

00:14:30.083 --> 00:14:33.532
<v Kyle Shelton>You know, because you can just send'em to the developer portal, you know, stage one.

00:14:33.532 --> 00:14:38.601
<v Kyle Shelton>They'll go through this, and even Amazon's doing the gamified training, right?

00:14:38.606 --> 00:14:42.201
<v Kyle Shelton>So you can get your certifications through the game and making it fun, right?

00:14:42.206 --> 00:14:42.261
<v Kyle Shelton>Yeah.

00:14:42.261 --> 00:14:44.331
<v Kyle Shelton>Make training fun, make education fun.

00:14:44.331 --> 00:14:48.351
<v Kyle Shelton>Your developers are also, these are some of the smartest people in your organization, right?

00:14:48.351 --> 00:14:52.912
<v Kyle Shelton>These are highly intelligent software engineers, developers, programmers.

00:14:52.932 --> 00:14:58.368
<v Kyle Shelton>Give them the ability to continue to grow the curiosity Knowledge-based solutions are great.

00:14:58.368 --> 00:14:59.778
<v Kyle Shelton>And you can do that in a developer portal.

00:14:59.918 --> 00:15:00.187
<v Jim Hirschauer>Yeah.

00:15:00.227 --> 00:15:02.118
<v Jim Hirschauer>IDPs are really hot topic right now.

00:15:02.118 --> 00:15:03.437
<v Jim Hirschauer>Everybody's looking into, yeah.

00:15:03.442 --> 00:15:05.087
<v Jim Hirschauer>How do they utilize them?

00:15:05.087 --> 00:15:06.408
<v Jim Hirschauer>How do they implement them?

00:15:06.408 --> 00:15:07.427
<v Jim Hirschauer>What are the benefits?

00:15:07.758 --> 00:15:13.697
<v Jim Hirschauer>So, as someone who has experience here, what are your top pieces of advice for anyone that's heading down that path?

00:15:13.697 --> 00:15:18.107
<v Jim Hirschauer>How should they get started and what's the best approach for them to take when they look at this holisticly.

00:15:19.398 --> 00:15:21.077
<v Kyle Shelton>Yeah, I think there's two things.

00:15:21.077 --> 00:15:22.712
<v Kyle Shelton>I think buy-in from the top down.

00:15:22.712 --> 00:15:41.115
<v Kyle Shelton>So having buy-in from your leadership that, hey, in order to make this work, you have to kind of draw the line in the sand and say, this is the new way we're gonna do things and everything whether it's team silo isolation or, or just a bunch of patterns or, or lack of observability or whatever.

00:15:41.222 --> 00:15:52.727
<v Kyle Shelton>This is not something that happens overnight, but you have to make that commitment to say, Okay, from here on out, each business unit is, and it's more about the DevOps culture in general, right?

00:15:52.727 --> 00:15:54.738
<v Kyle Shelton>As like you build it, you own it.

00:15:55.048 --> 00:16:06.686
<v Kyle Shelton>But having that top down buy-in, I think is the most crucial thing because it's hard to implement such a dramatic shift of the way you do things from the bottom up.

00:16:06.956 --> 00:16:07.225
<v Kyle Shelton>Yeah.

00:16:07.285 --> 00:16:10.166
<v Kyle Shelton>In my opinion and in my experience, that's the number one thing.

00:16:10.166 --> 00:16:15.535
<v Kyle Shelton>And then the number two thing is don't take a set solution that somebody else has built.

00:16:15.596 --> 00:16:18.176
<v Kyle Shelton>Look at what you have and and leverage what you have.

00:16:18.225 --> 00:16:21.885
<v Kyle Shelton>Help those that might be a little bit stubborn in this transition.

00:16:22.096 --> 00:16:25.245
<v Kyle Shelton>Help them grow, help them educate, help them learn.

00:16:25.245 --> 00:16:27.885
<v Kyle Shelton>The technology that's out now is, is outstanding.

00:16:27.885 --> 00:16:35.130
<v Kyle Shelton>And if you can take your programmatic mind, And then take these concepts and then just adjust them a little bit to the, you know grow, grow, grow internal.

00:16:35.130 --> 00:16:36.865
<v Kyle Shelton>You gotta foster what you have.

00:16:36.865 --> 00:16:40.775
<v Kyle Shelton>And there's no one playbook, there's no one system that's gonna work.

00:16:40.816 --> 00:16:41.985
<v Kyle Shelton>So hopefully that makes sense.

00:16:42.046 --> 00:16:42.946
<v Jim Hirschauer>Yeah, yeah, absolutely.

00:16:42.946 --> 00:16:43.831
<v Jim Hirschauer>And, I agree with you.

00:16:43.831 --> 00:16:45.691
<v Jim Hirschauer>Every company is unique.

00:16:45.701 --> 00:16:48.221
<v Jim Hirschauer>Every company has their own unique structure.

00:16:48.221 --> 00:16:49.691
<v Jim Hirschauer>They're unique challenges.

00:16:49.990 --> 00:16:52.250
<v Jim Hirschauer>They're working on usually unique products.

00:16:52.711 --> 00:16:58.350
<v Jim Hirschauer>And it makes sense that yes, you have to treat Overall solution as something that's gonna be unique to your company.

00:16:58.350 --> 00:17:03.721
<v Jim Hirschauer>And it can have shared components I think there are off the shelf solutions that people are certainly gonna purchase.

00:17:03.725 --> 00:17:06.480
<v Jim Hirschauer>Like observability for example, is, is one of those, right?

00:17:06.480 --> 00:17:08.641
<v Jim Hirschauer>There's many different observability solutions.

00:17:08.641 --> 00:17:13.230
<v Jim Hirschauer>So people are gonna take that and they're gonna customize it and make it fit for their own use cases.

00:17:13.590 --> 00:17:18.800
<v Jim Hirschauer>So, certainly agree with that there's no one single off the shelf solution that makes sense for everybody.

00:17:19.641 --> 00:17:22.066
<v Kyle Shelton>Yeah, and another thing too is things change fast.

00:17:22.066 --> 00:17:29.445
<v Kyle Shelton>If you look five years ago, there wasn't a cdk, there wasn't, you know, Terraform blueprints and there, you know, all these new things.

00:17:29.445 --> 00:17:40.076
<v Kyle Shelton>And as the technology evolves especially with open source software, like it's fascinating how quickly an idea becomes a product, becomes a community.

00:17:40.076 --> 00:17:40.465
<v Kyle Shelton>Right?

00:17:40.476 --> 00:17:40.836
<v Kyle Shelton>Right.

00:17:40.840 --> 00:17:41.965
<v Kyle Shelton>It's, it's insane.

00:17:41.971 --> 00:17:43.484
<v Kyle Shelton>And so you know, be ready.

00:17:43.585 --> 00:17:49.414
<v Kyle Shelton>Just when you think you understand things, you're probably gonna have to pivot and reinvent yourself as a business.

00:17:49.414 --> 00:17:52.805
<v Kyle Shelton>But if you look at the ones that are still around, that have been around for a long time, they've.

00:17:53.644 --> 00:17:53.795
<v Kyle Shelton>Yeah.

00:17:53.795 --> 00:17:55.402
<v Kyle Shelton>And you know, it's part of it.

00:17:55.402 --> 00:17:57.652
<v Jim Hirschauer>I think that's part of the beauty of what we get to do.

00:17:57.682 --> 00:17:58.852
<v Jim Hirschauer>We work in technology, right?

00:17:58.852 --> 00:18:02.511
<v Jim Hirschauer>And technology changes very rapidly, and we have to keep adapting with it.

00:18:02.511 --> 00:18:10.432
<v Jim Hirschauer>We have to keep learning and we have to apply that to our business, whatever company we're working for, and that's where the real magic happens.

00:18:10.432 --> 00:18:16.487
<v Jim Hirschauer>Being able to apply these fast moving technologies to help companies achieve their business use case and their business goals.

00:18:17.676 --> 00:18:18.777
<v Kyle Shelton>Yeah, it's amazing.

00:18:18.777 --> 00:18:21.980
<v Kyle Shelton>And it's really fun when you can see direct results.

00:18:22.051 --> 00:18:22.652
<v Kyle Shelton>Yeah, it's cool.

00:18:22.801 --> 00:18:23.132
<v Jim Hirschauer>Yeah.

00:18:23.582 --> 00:18:23.971
<v Jim Hirschauer>Awesome.

00:18:24.362 --> 00:18:30.541
<v Jim Hirschauer>All right, well we've covered off the, the tech material, and I want to jump into our second, just for fun segment.

00:18:31.031 --> 00:18:32.471
<v Jim Hirschauer>I've worked in it for a long time.

00:18:33.102 --> 00:18:35.682
<v Jim Hirschauer>As our listeners know, I've had my share of mess-ups.

00:18:35.682 --> 00:18:39.672
<v Jim Hirschauer>I've shared a couple in the past before, and I love asking this question.

00:18:40.241 --> 00:18:42.132
<v Jim Hirschauer>Kyle, what's your worst it mess up?

00:18:43.136 --> 00:18:45.335
<v Kyle Shelton>Okay, so this is a painful one.

00:18:45.335 --> 00:18:51.065
<v Kyle Shelton>I really thought I was getting fired after this, so this was about 12 years ago.

00:18:51.111 --> 00:18:54.651
<v Kyle Shelton>This is when I had just first moved to Colorado Springs.

00:18:54.692 --> 00:19:02.731
<v Kyle Shelton>I worked for about eight months as a contractor with Verizon Wireless and had the choice for a full-time job in Midland or Colorado Springs.

00:19:02.731 --> 00:19:04.261
<v Kyle Shelton>I chose Colorado Springs.

00:19:05.063 --> 00:19:09.202
<v Kyle Shelton>I was managing all the cabling contractors for the data center.

00:19:09.202 --> 00:19:11.692
<v Kyle Shelton>So I was in charge of all the transport equipment.

00:19:11.692 --> 00:19:21.098
<v Kyle Shelton>I worked with all the Cisco juniper routers, switches, firewalls, load balancers, anything layer two, layer three, transport in the core data center, I was a part of.

00:19:21.098 --> 00:19:30.320
<v Kyle Shelton>And so one day I was executing a night change, so I got in, I, I used to work 10:00 PM to 6:00 AM and execute all my maintenance windows through that.

00:19:30.320 --> 00:19:32.421
<v Kyle Shelton>When, you know, call volume is at the lows.

00:19:32.425 --> 00:19:38.046
<v Kyle Shelton>This was back in the day of like if, if we don't provide the cell service, there could be credits back on an account.

00:19:38.046 --> 00:19:38.945
<v Kyle Shelton>They're not making money.

00:19:39.036 --> 00:19:42.246
<v Kyle Shelton>You know, it's a big thing at a carrier level when you break things.

00:19:42.425 --> 00:19:42.556
<v Kyle Shelton>Sure.

00:19:42.586 --> 00:19:47.616
<v Kyle Shelton>So I was executing a mop and on all of which is a method of procedure.

00:19:47.616 --> 00:19:56.212
<v Kyle Shelton>And on one, all of these you had your basic, I don't know if you remember a Cisco like copy TFT flash, and you had to copy your Cisco configuration.

00:19:56.230 --> 00:20:00.071
<v Kyle Shelton>To the like local little flash drive and to the hard disc, right?

00:20:00.071 --> 00:20:05.166
<v Kyle Shelton>And so on, on like my third maintenance window, I did not do that.

00:20:05.166 --> 00:20:07.777
<v Kyle Shelton>I, I just skipped through that and I was saying, it's no big deal.

00:20:07.777 --> 00:20:08.676
<v Kyle Shelton>Everything will be fine.

00:20:08.676 --> 00:20:16.808
<v Kyle Shelton>And so I went through and I did a, a, this was on a Cisco FEX 8500 I believe, big giant like card, you know, layer three switch.

00:20:16.808 --> 00:20:19.818
<v Kyle Shelton>And I was doing an a OS upgrade on it.

00:20:19.884 --> 00:20:29.703
<v Kyle Shelton>Well, when, when the switch came back it came back with an empty configuration file and you're talking hundreds of ports in a data center that were just unconfigured.

00:20:29.703 --> 00:20:37.996
<v Kyle Shelton>So basically I took out about six cabinets of AAA', which is the authentication system at the time for your cell phone.

00:20:38.195 --> 00:20:51.518
<v Kyle Shelton>So pretty much long story short, it took out almost the whole west region could not authenticate their phone, so anybody that turned on their phone couldn't authenticate with the network, had no service, couldn't make an emergency call, it was a big deal, like that was bad.

00:20:51.518 --> 00:20:53.713
<v Kyle Shelton>And so I figured that out really quick.

00:20:53.719 --> 00:21:10.203
<v Kyle Shelton>Well, remember I said I didn't copy the, the router config over in, at the beginning, and so I had to go in there with my laptop and an old console serial cable and line by line, copy every single configuration for that layer three switch.

00:21:10.270 --> 00:21:13.631
<v Kyle Shelton>It was, I think three to 4,000 lines, one by one.

00:21:13.635 --> 00:21:14.840
<v Kyle Shelton>It took me(brutal).

00:21:14.861 --> 00:21:15.701
<v Kyle Shelton>A long time.

00:21:15.701 --> 00:21:23.201
<v Kyle Shelton>It was one of the, I mean, and this, so, and it really didn't come up until I got that and committed it and, and the switch came back up.

00:21:23.651 --> 00:21:37.921
<v Kyle Shelton>And the service was restored and, and, but like those two and a half, three hours of me being in the data center copying line by line, cuz you couldn't do it more because the way the, if you've worked on the Cisco consoles like it, it'll paste.

00:21:38.996 --> 00:21:43.260
<v Kyle Shelton>And if you try to paste more than one line, it gets all jumbled up and it just doesn't do anything.

00:21:43.260 --> 00:21:44.820
<v Kyle Shelton>And then you have even a worse config.

00:21:44.820 --> 00:21:44.911
<v Kyle Shelton>Yes.

00:21:45.000 --> 00:21:47.080
<v Kyle Shelton>Which is it, it, it's a pain in the butt.

00:21:47.109 --> 00:21:48.160
<v Kyle Shelton>And so, yeah.

00:21:48.271 --> 00:21:54.942
<v Kyle Shelton>That I will never not copy a router config or take a backup of a configuration ever again.

00:21:54.991 --> 00:21:56.132
<v Kyle Shelton>My boss just stressed.

00:21:56.132 --> 00:22:00.932
<v Kyle Shelton>He's like, I have to write you up for this, obviously, and, and you can't do this again or you will get fired.

00:22:00.932 --> 00:22:02.971
<v Kyle Shelton>But he's like, you have to follow exactly.

00:22:03.001 --> 00:22:07.382
<v Kyle Shelton>Even if you know it's gonna break, then back out and say this broke.

00:22:07.531 --> 00:22:13.211
<v Kyle Shelton>But if you don't follow the procedure provided by engineering it is 100% your fault.

00:22:13.261 --> 00:22:22.501
<v Kyle Shelton>If they give you a bad procedure and it's not tested thoroughly, then the, the onus is on them because they, they did not set you up to succeed.

00:22:22.501 --> 00:22:27.284
<v Kyle Shelton>And there's all those checks and balances and, and so yeah, it was a scary time.

00:22:28.528 --> 00:22:29.194
<v Kyle Shelton>But yeah.

00:22:29.404 --> 00:22:30.035
<v Kyle Shelton>That, that was it.

00:22:30.515 --> 00:22:30.845
<v Jim Hirschauer>Yeah.

00:22:30.845 --> 00:22:37.365
<v Jim Hirschauer>You know, I, I heard a saying a really long time ago when I first started in, in my IT career and mm-hmm.

00:22:37.398 --> 00:22:39.798
<v Jim Hirschauer>The saying was, you're only as good as your last backup.

00:22:40.338 --> 00:22:40.759
<v Jim Hirschauer>Yep.

00:22:40.763 --> 00:22:42.108
<v Jim Hirschauer>And I really took that to heart.

00:22:42.108 --> 00:22:42.528
<v Jim Hirschauer>I'm like, oh.

00:22:43.084 --> 00:22:44.104
<v Jim Hirschauer>Yeah, that makes sense.

00:22:44.104 --> 00:22:44.193
<v Jim Hirschauer>Mm-hmm.

00:22:44.443 --> 00:22:47.703
<v Jim Hirschauer>If, if everything blows up, all I have left is my last backup.

00:22:47.703 --> 00:22:47.794
<v Jim Hirschauer>Right.

00:22:48.153 --> 00:22:49.564
<v Jim Hirschauer>And who knows how long ago that was.

00:22:49.564 --> 00:22:49.834
<v Jim Hirschauer>Right.

00:22:49.894 --> 00:22:52.804
<v Jim Hirschauer>I'm hoping it's, you know, less than 12 hours old.

00:22:53.074 --> 00:22:56.044
<v Jim Hirschauer>In those days, it was probably, could have been up to 24 hours old.

00:22:56.314 --> 00:22:57.453
<v Jim Hirschauer>And so if everything mm-hmm.

00:22:57.784 --> 00:23:02.463
<v Jim Hirschauer>Just, you know, worst case scenario, you're losing up to 24 hours of data potentially.

00:23:02.574 --> 00:23:04.634
<v Jim Hirschauer>But at least you have something, you have something to fall back on.

00:23:04.634 --> 00:23:09.854
<v Jim Hirschauer>I can't imagine sitting and having to copy three to 4,000 lines of configuration one at a time.

00:23:09.858 --> 00:23:09.943
<v Jim Hirschauer>Right.

00:23:10.304 --> 00:23:11.324
<v Jim Hirschauer>Any idea how long that took?

00:23:12.838 --> 00:23:23.209
<v Kyle Shelton>Yeah, I went home at about 1:30 2:00 PM I remember the outage recovered, so the upgrades, like the upgrades started at like 1:30 in the morning.

00:23:23.269 --> 00:23:23.338
<v Kyle Shelton>Mm-hmm.

00:23:23.355 --> 00:23:28.786
<v Kyle Shelton>It bled through the maintenance, so there's acceptable time, like with outages.

00:23:28.786 --> 00:23:32.895
<v Kyle Shelton>And so I had up until like 5:00 AM and then I ended.

00:23:34.256 --> 00:23:36.736
<v Kyle Shelton>I think recovering at around eight or 9:00 AM Okay.

00:23:37.155 --> 00:23:39.256
<v Kyle Shelton>So I wanna say probably three to four hours.

00:23:39.586 --> 00:23:39.645
<v Kyle Shelton>Yeah.

00:23:39.645 --> 00:23:40.066
<v Kyle Shelton>You know?

00:23:40.125 --> 00:23:47.994
<v Kyle Shelton>But still, my manager told me that those two or three hours was over a million dollars worth of money lost mm-hmm.

00:23:48.234 --> 00:23:48.894
<v Kyle Shelton>For the company.

00:23:48.894 --> 00:23:48.954
<v Kyle Shelton>Wow.

00:23:49.045 --> 00:23:56.835
<v Kyle Shelton>And is that, and, and then, so like, Like, you can only imagine we're talking, this is only six out of like, you know, 300 cabinets too.

00:23:57.105 --> 00:23:59.055
<v Kyle Shelton>Very small portion.

00:23:59.444 --> 00:24:02.845
<v Kyle Shelton>But you take out all the network, all the layer two.

00:24:02.884 --> 00:24:03.285
<v Kyle Shelton>Yeah.

00:24:03.434 --> 00:24:04.305
<v Kyle Shelton>Things aren't gonna work.

00:24:04.875 --> 00:24:05.035
<v Jim Hirschauer>Yeah.

00:24:05.805 --> 00:24:08.537
<v Jim Hirschauer>Well that's it's a tough lesson to learn for sure.

00:24:08.537 --> 00:24:10.126
<v Jim Hirschauer>But we're all human.

00:24:10.126 --> 00:24:11.596
<v Jim Hirschauer>It happens to every single one of us.

00:24:11.596 --> 00:24:14.997
<v Jim Hirschauer>It's happened to every single person who works in IT if they've worked for long enough.

00:24:16.017 --> 00:24:16.467
<v Kyle Shelton>Yeah.

00:24:16.522 --> 00:24:20.153
<v Kyle Shelton>And you learn the backup lesson, either the easy way or the hard way.

00:24:20.167 --> 00:24:22.759
<v Kyle Shelton>You know, most of us learn it the hard way, right?

00:24:22.759 --> 00:24:23.000
<v Kyle Shelton>Yeah.

00:24:23.000 --> 00:24:23.180
<v Kyle Shelton>Yeah.

00:24:23.180 --> 00:24:24.480
<v Kyle Shelton>But yeah, crazy.

00:24:25.050 --> 00:24:25.530
<v Jim Hirschauer>Alright.

00:24:25.566 --> 00:24:28.171
<v Jim Hirschauer>Well listen, Kyle, it's been a pleasure speaking with you today.

00:24:28.175 --> 00:24:38.256
<v Jim Hirschauer>Thanks so much for joining us and sharing your knowledge and your wisdom, your ice fishing story, as well as being vulnerable and sharing your multimillion dollar iT mess up.

00:24:38.405 --> 00:24:39.705
<v Jim Hirschauer>I really appreciate that.

00:24:39.720 --> 00:24:53.044
<v Jim Hirschauer>To all of our listeners if you're an SRE or if you're in a related DevOps type role and you wanna be a guest speaker on ShipTalk, please send an email to podcast@shiptalk.io and we'll get back to you.

00:24:53.074 --> 00:24:54.483
<v Jim Hirschauer>Thanks again, Kyle.

00:24:54.953 --> 00:24:55.584
<v Jim Hirschauer>Appreciate it.

00:24:55.913 --> 00:24:56.663
<v Jim Hirschauer>That's all for now.

00:24:56.663 --> 00:24:57.233
<v Jim Hirschauer>Until next time.