WEBVTT

00:00:07.469 --> 00:00:08.220
<v Jim Hirschauer>Alright.

00:00:10.259 --> 00:00:12.269
<v Jim Hirschauer>Welcome to ShipTalk, the SRE edition.

00:00:12.509 --> 00:00:14.310
<v Jim Hirschauer>I'm Jim Hirschauer, your host for today.

00:00:14.869 --> 00:00:22.920
<v Jim Hirschauer>ShipTalk is a DevOps podcast, brought to you by Harness, the software delivery platform, and the SRE edition focuses on reliability topics.

00:00:23.129 --> 00:00:25.710
<v Jim Hirschauer>My guest today is Hilliary Lipsig from Red Hat.

00:00:26.039 --> 00:00:27.089
<v Jim Hirschauer>Hilliary, welcome to the show.

00:00:27.780 --> 00:00:28.289
<v Hilliary Lipsig>Thanks, Jim.

00:00:28.289 --> 00:00:29.010
<v Hilliary Lipsig>Happy to be here.

00:00:29.699 --> 00:00:30.809
<v Hilliary Lipsig>My name's Hillary Lipsig.

00:00:30.839 --> 00:00:33.414
<v Hilliary Lipsig>I am Chief Mermaid at Red Hat.

00:00:33.534 --> 00:00:35.875
<v Hilliary Lipsig>And if you look me up on LinkedIn, that's really there.

00:00:35.905 --> 00:00:37.585
<v Hilliary Lipsig>If you've seen my talks, that's my title.

00:00:38.155 --> 00:00:48.799
<v Hilliary Lipsig>And the reason for that title is because I am the technical lead for a small subdivision in Red Hat called Strategy Enablement and Architecture out of our larger service delivery organization.

00:00:49.549 --> 00:00:58.789
<v Hilliary Lipsig>And when they said, Hey You really can't have principal site reliability engineer on your job title anymore because you're not really, you know, an SRE that way.

00:00:59.060 --> 00:01:01.969
<v Hilliary Lipsig>What do you think a good business card title would be for you?

00:01:02.299 --> 00:01:07.995
<v Hilliary Lipsig>I said chief Mermaid, officer because I was finally my opportunity to grow up and be a mermaid like I'd always wanted.

00:01:07.995 --> 00:01:09.724
<v Hilliary Lipsig>So they laughed but we did it.

00:01:09.935 --> 00:01:14.974
<v Hilliary Lipsig>So that is that is my official unofficial business card title which is delightful.

00:01:15.605 --> 00:01:18.935
<v Jim Hirschauer>That's probably the greatest title I've heard so far, so congrats on that.

00:01:19.745 --> 00:01:20.194
<v Hilliary Lipsig>Thank you.

00:01:20.194 --> 00:01:21.305
<v Hilliary Lipsig>I'm very proud of it.

00:01:21.305 --> 00:01:24.605
<v Hilliary Lipsig>Someday they will probably make me change it, but today is not that day.

00:01:25.144 --> 00:01:25.564
<v Jim Hirschauer>Awesome.

00:01:25.564 --> 00:01:27.094
<v Jim Hirschauer>Keep it as long as you can, for sure.

00:01:28.534 --> 00:01:36.459
<v Jim Hirschauer>Well Hilliary, you know, we've talked about the format of the show We have a couple of sections that are called just for fun, and then we get into the main topic.

00:01:36.489 --> 00:01:38.680
<v Jim Hirschauer>So we're gonna start off with a just for fun section.

00:01:39.280 --> 00:01:52.530
<v Jim Hirschauer>And we, when we were talking earlier, you, you used a word that I had never heard before, so I'm not sure if you made up this word, or at least you're like a really early adopter of this new word, but I feel like we should all know what this word is.

00:01:52.530 --> 00:01:53.609
<v Jim Hirschauer>So why don't you go and explain that.

00:01:54.525 --> 00:01:55.125
<v Hilliary Lipsig>Sure.

00:01:55.275 --> 00:01:58.245
<v Hilliary Lipsig>So people tell me, I made up this word.

00:01:58.245 --> 00:02:02.924
<v Hilliary Lipsig>I do not believe it, but the people I thought I learned it from insisted they learned it from me.

00:02:02.924 --> 00:02:04.334
<v Hilliary Lipsig>So who knows where this came from?

00:02:04.435 --> 00:02:14.004
<v Hilliary Lipsig>So when you're looking at a pod in Kubernetes and it has an out of memory or an OOM somewhere along the way I said, Ugh, that's ooming.

00:02:14.805 --> 00:02:16.305
<v Hilliary Lipsig>And people are like, what?

00:02:16.310 --> 00:02:17.245
<v Hilliary Lipsig>And I said, yeah, ooming.

00:02:17.264 --> 00:02:19.275
<v Hilliary Lipsig>It's like, you know, you say oom for out of memory.

00:02:19.275 --> 00:02:20.564
<v Hilliary Lipsig>It's, it's ooming, it's a verb.

00:02:20.564 --> 00:02:21.344
<v Hilliary Lipsig>It's delightful.

00:02:21.974 --> 00:02:25.604
<v Hilliary Lipsig>And this has caught on to some degree here in Red Hat.

00:02:25.604 --> 00:02:26.715
<v Hilliary Lipsig>Others have picked it up.

00:02:27.134 --> 00:02:30.044
<v Hilliary Lipsig>Again, I'm credited with this.

00:02:30.074 --> 00:02:32.055
<v Hilliary Lipsig>I'm certain I did not come up with this.

00:02:32.055 --> 00:02:33.914
<v Hilliary Lipsig>I must have heard this somewhere.

00:02:34.215 --> 00:02:38.865
<v Hilliary Lipsig>So if anybody knows who I learned it from, they deserve all the accolades.

00:02:38.944 --> 00:02:42.215
<v Hilliary Lipsig>But I will tell you that Like I said, it's delightful.

00:02:42.634 --> 00:02:48.935
<v Hilliary Lipsig>And I have been on a call with a customer that was very serious, very tense.

00:02:48.935 --> 00:02:51.034
<v Hilliary Lipsig>They were rightfully so, really upset.

00:02:51.544 --> 00:03:03.349
<v Hilliary Lipsig>And I used the term and then I paused to explain the term and you know, it's not like magically everything was better, but like there, suddenly there were smiles on the call.

00:03:03.349 --> 00:03:11.240
<v Hilliary Lipsig>People were laughing and it, it seriously reduced the whole tension of the situation because I wasn't taking myself or anything very seriously.

00:03:11.300 --> 00:03:16.759
<v Hilliary Lipsig>Or the right level of levity is really what I wanna say is what that brings into the conversation.

00:03:17.219 --> 00:03:17.324
<v Hilliary Lipsig>Yeah.

00:03:17.875 --> 00:03:19.495
<v Hilliary Lipsig>And so then we got back to our very serious,

00:03:20.110 --> 00:03:21.009
<v Jim Hirschauer>discussions.

00:03:21.939 --> 00:03:22.689
<v Jim Hirschauer>Well, that's awesome.

00:03:22.729 --> 00:03:33.110
<v Jim Hirschauer>I I love the word, and I, I'm curious though, can you use the word a little more generically or does it have to be like, really related to an out of memory error?

00:03:33.169 --> 00:03:34.610
<v Jim Hirschauer>Can, can I use that word?

00:03:34.610 --> 00:03:39.830
<v Jim Hirschauer>If, let's say the system is just like chugging along and performing poorly, can I say, oh, it's, it's ooming.

00:03:39.830 --> 00:03:41.539
<v Jim Hirschauer>Is it almost like googling something?

00:03:43.240 --> 00:03:55.710
<v Hilliary Lipsig>No, I, I do feel like it is kind of pretty specific because it maps back to a very, like, you see OOM visually on the like OC get pod, or not oc I'm sorry, that's a very OpenShift thing.

00:03:56.110 --> 00:03:58.669
<v Hilliary Lipsig>kubctl, like get pods or whatever.

00:03:58.675 --> 00:03:59.150
<v Hilliary Lipsig>Mm-hmm.

00:03:59.250 --> 00:04:00.449
<v Hilliary Lipsig>I work in OpenShift.

00:04:00.449 --> 00:04:02.259
<v Hilliary Lipsig>We have our own CLI.

00:04:02.284 --> 00:04:10.400
<v Hilliary Lipsig>It's opinionated and I have that completely memorized, much to my occasional detriment when I wonder why it doesn't work on vanilla Kubernetes.

00:04:10.620 --> 00:04:10.979
<v Jim Hirschauer>Right.

00:04:11.639 --> 00:04:12.210
<v Jim Hirschauer>All right, good.

00:04:12.210 --> 00:04:22.389
<v Jim Hirschauer>So I've learned today that I can use this word ooming if I see an out of memory exception, so next time I see one, that's definitely, it's ingrained in my memory now for sure.

00:04:22.509 --> 00:04:22.990
<v Jim Hirschauer>I love it.

00:04:23.720 --> 00:04:30.639
<v Jim Hirschauer>I feel like I'm just gonna give you credit cause you're the first person I've heard it from, so if anyone asks me, I'm gonna say I heard it from you.

00:04:30.699 --> 00:04:35.358
<v Jim Hirschauer>And origin being unknown, I think you really deserve the credit at this point.

00:04:35.408 --> 00:04:36.744
<v Jim Hirschauer>If it really starts taking off.

00:04:37.923 --> 00:04:40.223
<v Hilliary Lipsig>I will humbly accept that credit, but thank you.

00:04:40.843 --> 00:04:41.283
<v Jim Hirschauer>Okay.

00:04:42.043 --> 00:04:42.603
<v Jim Hirschauer>Alright.

00:04:42.624 --> 00:04:46.814
<v Jim Hirschauer>So let's get a little more serious now and, and jump into our main topic.

00:04:46.843 --> 00:05:01.514
<v Jim Hirschauer>So You know, we had a chance yesterday to chat a little bit before the show, and you were talking about observability and, you know, I've been in and around the observability space personally for over a decade.

00:05:01.824 --> 00:05:11.874
<v Jim Hirschauer>My background as a practitioner goes back into the monitoring realm, which is kind of the, I'd say the, the granddaddy of, of modern observability and.

00:05:12.759 --> 00:05:23.798
<v Jim Hirschauer>I think most of us that are either on the show right now or listening to the show, we all know how and why we should be using observability to identify and resolve production incidents.

00:05:24.399 --> 00:05:29.709
<v Jim Hirschauer>But yesterday you said to me that we can, and we should be doing more with all of that data.

00:05:30.038 --> 00:05:31.928
<v Jim Hirschauer>So I'd love for you to explain your position on that.

00:05:33.129 --> 00:05:33.699
<v Hilliary Lipsig>Sure.

00:05:33.798 --> 00:05:37.499
<v Hilliary Lipsig>And I'd like to hope this isn't a controversial position one of the.

00:05:37.918 --> 00:05:53.538
<v Hilliary Lipsig>One of the projects that I'm, I'm working on with Red Hat is it's a data project and we were working through the personas of how, how do we be a more data-driven organization and, and what kind of data do we really need to be that?

00:05:54.108 --> 00:05:56.329
<v Hilliary Lipsig>And so, and I swear this is not an original thought.

00:05:56.329 --> 00:05:59.629
<v Hilliary Lipsig>I'm sure I learned this as from somebody who was much smarter than I am.

00:06:00.228 --> 00:06:00.738
<v Hilliary Lipsig>But.

00:06:01.303 --> 00:06:10.613
<v Hilliary Lipsig>I said that, you know, one of the things that we could be doing is actually demonstrating our cloud like usage metrics.

00:06:10.613 --> 00:06:13.884
<v Hilliary Lipsig>So what are we doing with CPU and what are we doing with memory?

00:06:15.084 --> 00:06:20.694
<v Hilliary Lipsig>And then using that to compare to, you know, when we're pla when we're doing these plannings for these systems, we typically have some sort of profile.

00:06:22.478 --> 00:06:32.199
<v Hilliary Lipsig>So the first is, of course, that profiling, like where are we in our, our usage compared to how we you know, how we forecasted to make sure we're, you know, kind of keeping track of that.

00:06:32.204 --> 00:06:38.004
<v Hilliary Lipsig>That'll all maps back to spend, which everybody is really caring about right now which you should always care about actually.

00:06:38.543 --> 00:06:51.363
<v Hilliary Lipsig>And the other thing that I said was, you know, It's important to watch trends as well in these usage and not just like looking over the course of the month or course of a day, but looking a little bit more granularly.

00:06:51.744 --> 00:07:03.824
<v Hilliary Lipsig>And so there's actually an example from the SRE team that I, I joined at Red Hat actually as to that was a team I came onto that I'm, I'm no longer a part of although I still love them and they love me, I think.

00:07:04.033 --> 00:07:15.269
<v Hilliary Lipsig>But anyway, the There was some thresholds that were set up in Prometheus that would create warning alerts and it would just be for things like CPU and memory and so forth.

00:07:15.778 --> 00:07:21.939
<v Hilliary Lipsig>And because of these thresholds we noticed things like memory leaks in the code.

00:07:21.988 --> 00:07:26.369
<v Hilliary Lipsig>And there's a couple things that like happens when a memory leak is happening, a couple symptoms.

00:07:26.699 --> 00:07:29.369
<v Hilliary Lipsig>So it's not just that your memory usage goes up and up and up.

00:07:29.934 --> 00:07:35.593
<v Hilliary Lipsig>But actually your CPU will start churning and churning and churning even in virtualized hardware.

00:07:36.014 --> 00:07:42.584
<v Hilliary Lipsig>It will work harder and harder to try and reclaim as much memory as possible to try to prevent an out of memory.

00:07:43.043 --> 00:07:53.884
<v Hilliary Lipsig>And so taking a look at that kind of data, and looking at that data a little bit more granularly, we can start actually looking for issues in our software.

00:07:53.889 --> 00:07:58.024
<v Hilliary Lipsig>And of course, so we had set alerts on that to let us know that there might be a memory leak.

00:07:58.363 --> 00:08:02.473
<v Hilliary Lipsig>But you can actually with more data, cuz again, that was one simple Prometheus rule.

00:08:02.874 --> 00:08:05.394
<v Hilliary Lipsig>You can actually get a little bit more intelligent about it as well.

00:08:05.903 --> 00:08:17.228
<v Hilliary Lipsig>And so, That's one of the things that I am trying to drive this kind of a, a cultural push within Red Hat is and I know I'm not alone in this, I really don't wanna say that there's nobody else doing this.

00:08:17.233 --> 00:08:20.233
<v Hilliary Lipsig>It's just I've, I've, I've joined onto this to this type of an effort.

00:08:20.744 --> 00:08:26.533
<v Hilliary Lipsig>It's this idea that we should be using our metrics to look and see if we can be improving our software.

00:08:26.538 --> 00:08:34.153
<v Hilliary Lipsig>Because if we profile our software and then we run our software with like, The profiles we've defined.

00:08:34.153 --> 00:08:39.254
<v Hilliary Lipsig>So we've got memory limits and CPU requests and yes, CPU limits.

00:08:39.724 --> 00:08:42.724
<v Hilliary Lipsig>I know that CPU limits are like, people are like, you don't need them.

00:08:42.724 --> 00:08:45.599
<v Hilliary Lipsig>I will argue that it's a whole other 25 minute conversation.

00:08:46.028 --> 00:08:47.769
<v Hilliary Lipsig>And, but yes, CPU limits as well.

00:08:48.308 --> 00:09:07.313
<v Hilliary Lipsig>And so, If we're constantly running close to your limits or even occasionally, like hitting the limits and then things restart and run fine for a while those might not necessarily all result in incidents or downtime or alerts, but all these things tell you that your software's probably not operating as well as it could be.

00:09:08.063 --> 00:09:18.673
<v Hilliary Lipsig>And so, We wanna look into why is our software not operating as well as it could be Because the nature of, of working in a cloud native environment or Kubernetes is often that I have seen.

00:09:18.974 --> 00:09:26.984
<v Hilliary Lipsig>Sometimes certain things can kind of be hidden for a while because it's highly available because it restarts and brings itself back up to a desired state.

00:09:27.224 --> 00:09:33.683
<v Hilliary Lipsig>There might not be any seriously noticeable blip but that doesn't mean that the blips that exist are acceptable either.

00:09:34.254 --> 00:09:34.744
<v Jim Hirschauer>Yeah.

00:09:35.464 --> 00:09:37.293
<v Jim Hirschauer>So, you know, you make an interesting point.

00:09:37.344 --> 00:09:40.313
<v Jim Hirschauer>Being proactive with this type of activity.

00:09:40.764 --> 00:09:43.323
<v Jim Hirschauer>First of all, most companies have this type of data.

00:09:43.514 --> 00:09:47.528
<v Jim Hirschauer>They have observability solutions in-house or in place at least.

00:09:47.859 --> 00:09:52.089
<v Jim Hirschauer>And there's a, a wealth of data at most companies.

00:09:53.438 --> 00:09:59.408
<v Jim Hirschauer>What I think is interesting about this is, is you made the point really early on that we should always be concerned with costs.

00:09:59.408 --> 00:10:22.818
<v Jim Hirschauer>And what I've seen happen is to, over provision your resources with your cloud computing provider, or we used to do this, you know, back in the days of the plain old data center, we just used to over provision the heck out of our hardware so that we could avoid these issues for as long as possible, avoid the out of memory exception from kicking in and, and causing an error, causing the impact to our users.

00:10:23.359 --> 00:10:27.254
<v Jim Hirschauer>But that's in my mind, especially in today's world, is, completely unacceptable.

00:10:27.254 --> 00:10:41.879
<v Jim Hirschauer>That goes out the window and we have to do a better job, as an IT community of creating services that respond and behave properly, that we don't have to over provision and, and really waste a ton of money on.

00:10:42.479 --> 00:10:44.249
<v Jim Hirschauer>So I think you're really onto something here.

00:10:44.609 --> 00:11:00.619
<v Jim Hirschauer>You know, just besides the fact that you can make an overall better customer experience by having better performing software, you can end up saving your company significant cost across all of the different services that most companies have due to that over provisioning.

00:11:02.298 --> 00:11:02.688
<v Hilliary Lipsig>Yeah.

00:11:02.788 --> 00:11:10.048
<v Hilliary Lipsig>And this is actually one of the, this goes into the, and I, I, I swear I did not intend to derail our entire conversation into CPU limits.

00:11:10.948 --> 00:11:15.568
<v Hilliary Lipsig>But it actually, it actually came up very recently and I explained to somebody all the reasons why I'm in favor of them.

00:11:15.573 --> 00:11:22.769
<v Hilliary Lipsig>And I'll give a, a high level overview here, which is one, I'm not only dealing with virtualized hardware, I'm dealing with real hardware.

00:11:23.129 --> 00:11:27.163
<v Hilliary Lipsig>So, I can't just let processes do whatever they want.

00:11:27.323 --> 00:11:29.244
<v Hilliary Lipsig>It actually is, is finite.

00:11:29.543 --> 00:11:33.953
<v Hilliary Lipsig>And if you're overcommitting your CPU on real hardware, you're shortening your hardware's lifespan.

00:11:34.403 --> 00:11:39.504
<v Hilliary Lipsig>So if you think about how expensive a server is, first of all, that's extremely wasteful to short its lifespan.

00:11:40.104 --> 00:11:45.953
<v Hilliary Lipsig>Second of all, it's drawing more energy, which is extremely wasteful cause it causes more pull there and more actually.

00:11:46.563 --> 00:11:51.484
<v Hilliary Lipsig>Impacts the overall ambient temperature, which means the AC needs to run wherever that server is located more often.

00:11:52.083 --> 00:12:00.744
<v Hilliary Lipsig>So as you said, it's not just about the immediate company's fiscal situation, but then we start getting into the environmental impacts of not having performant software.

00:12:01.193 --> 00:12:08.464
<v Hilliary Lipsig>Which I think about all the time, because when I'm thinking about the size of, of the Red Hat customer base and, and so forth.

00:12:08.464 --> 00:12:09.903
<v Hilliary Lipsig>These, this enterprise scales.

00:12:10.563 --> 00:12:12.004
<v Hilliary Lipsig>These are actually things that start coming up.

00:12:12.004 --> 00:12:18.033
<v Hilliary Lipsig>We have projects around sustainability and, and environmental impact as I think a lot of major companies do these days.

00:12:18.464 --> 00:12:20.948
<v Hilliary Lipsig>And so it's, these are the types of things that, you know, we're caring about.

00:12:20.953 --> 00:12:22.749
<v Hilliary Lipsig>These are the types of ways we're wanting to use data.

00:12:23.078 --> 00:12:24.969
<v Hilliary Lipsig>And since we already have the data, it.

00:12:25.479 --> 00:12:33.249
<v Hilliary Lipsig>Honestly would be almost irresponsible not to use it to figure out how to make everything better holistically and looking at the holistic picture of our software.

00:12:33.568 --> 00:12:37.139
<v Hilliary Lipsig>And since I'm somebody with an operational background, I care about how it's performing.

00:12:38.009 --> 00:12:39.149
<v Jim Hirschauer>Yeah, makes sense.

00:12:39.719 --> 00:12:41.188
<v Jim Hirschauer>So let me ask you this.

00:12:41.288 --> 00:12:44.769
<v Jim Hirschauer>Sometimes it's really easy to say we should do this thing.

00:12:45.219 --> 00:13:00.339
<v Jim Hirschauer>The reality that most companies are faced with and, and most SREs or folks who are responsible for reliability and performance of overall systems is that they are typically pretty far behind in their main job responsibilities.

00:13:00.344 --> 00:13:25.043
<v Jim Hirschauer>So, what advice can you give folks to start to transition from using their observability data in a reactive methodology where they're just putting out fires all the time to being able to truly switch over into this proactive mode where they're, looking at things ahead of time and, and trying to make sure that the systems are actually performant in the right way so that they can right size things.

00:13:25.043 --> 00:13:28.943
<v Jim Hirschauer>I know it's, it's just a very difficult thing to make that transition in some companies.

00:13:28.943 --> 00:13:30.594
<v Jim Hirschauer>So how would you suggest folks do that?

00:13:31.849 --> 00:13:35.339
<v Hilliary Lipsig>And so I wanna actually point out here Red Hat is it's no different.

00:13:35.729 --> 00:13:38.278
<v Hilliary Lipsig>We're struggling through some of these same problems right now.

00:13:38.399 --> 00:13:45.913
<v Hilliary Lipsig>And one of the initiatives I am responsible for at Red Hat is something called hybrid, SRE, which is an interesting name.

00:13:46.724 --> 00:13:51.134
<v Hilliary Lipsig>And this is the idea that we stop throwing code over the wall to SREs.

00:13:51.403 --> 00:14:00.553
<v Hilliary Lipsig>I spent 11 years in quality engineering before I moved to SRE, and I experienced in quality engineering how bad throwing code over the wall can be.

00:14:00.884 --> 00:14:22.083
<v Hilliary Lipsig>And so what, what I did when I was head of quality engineering at my, in my last role was I actually had the quality engineers working on the engineering teams working on unit testing, making unit tests a little bit more similar to kind of more like the, the standard way of quality engineering, thinking about testing with, of like integration and regression and putting that, shifting it left into the unit testing framework.

00:14:22.573 --> 00:14:24.793
<v Hilliary Lipsig>Especially with microservices, that's really easy to do.

00:14:24.948 --> 00:14:25.489
<v Hilliary Lipsig>It's much harder.

00:14:25.489 --> 00:14:27.729
<v Hilliary Lipsig>It was much harder in the monolithic architectures.

00:14:28.239 --> 00:14:34.538
<v Hilliary Lipsig>And we really just need to be doing the same thing, that same type of partnership with SRE and service engineers, like was what we call them.

00:14:35.229 --> 00:14:48.219
<v Hilliary Lipsig>And so we've started doing, we've, we've pushed off this initiative and it's about getting engineers closer to their operations and the operations folks closer to the engineering so that you've got better communication, better partnership.

00:14:48.578 --> 00:14:50.948
<v Hilliary Lipsig>And a lot of that is actually process engineering.

00:14:51.869 --> 00:14:56.788
<v Hilliary Lipsig>Where we're putting together processes of like, okay, here's how we're going to do these things.

00:14:57.208 --> 00:15:13.198
<v Hilliary Lipsig>And so what we're doing is by putting in these processes that generally make the software more reliable and give it a, a baseline higher level of service maturity out the gate that actually frees up the SREs to do a little bit more of that proactive work.

00:15:13.769 --> 00:15:21.813
<v Hilliary Lipsig>So it really requires a strong partnership between SRE and the software developers who are on the service kind of dividing and conquering.

00:15:22.083 --> 00:15:30.484
<v Hilliary Lipsig>So SRE should be putting out things and guidance and best practices to their engineering teams to say, Hey, in our environments, this is what we know.

00:15:30.604 --> 00:15:33.303
<v Hilliary Lipsig>These are the good patterns that make for performant software.

00:15:34.024 --> 00:15:39.288
<v Hilliary Lipsig>And then, It's incumbent on engineering to deliver software that matches those patterns.

00:15:39.589 --> 00:15:40.369
<v Hilliary Lipsig>That helps.

00:15:40.668 --> 00:15:44.958
<v Hilliary Lipsig>And then it must be an ongoing partnership of like, okay, how are we performing?

00:15:44.958 --> 00:15:46.369
<v Hilliary Lipsig>How many incidents are we getting?

00:15:46.639 --> 00:15:48.558
<v Hilliary Lipsig>What kind of trends are we seeing in the incidents?

00:15:48.558 --> 00:15:56.749
<v Hilliary Lipsig>There's A great practice that our platform SRE team does, which is where they actually take like some major incidents, especially anything that's recurrent.

00:15:57.208 --> 00:16:03.028
<v Hilliary Lipsig>And they put together what we call a tiger team and they'll go and kind of like do a deep dive and it's cross-functional.

00:16:03.489 --> 00:16:09.359
<v Hilliary Lipsig>And so that's really, The thing that I, I have to say is the way to get there, it's what we've been doing.

00:16:09.359 --> 00:16:10.558
<v Hilliary Lipsig>It's what we see working.

00:16:10.918 --> 00:16:18.298
<v Hilliary Lipsig>We've seen some really great outcomes of like, you know, certain failures going away completely or, you know, being reduced.

00:16:18.298 --> 00:16:21.568
<v Hilliary Lipsig>And so, and we also have requirements of toil.

00:16:21.928 --> 00:16:24.448
<v Hilliary Lipsig>If you're having SRE, run your service.

00:16:25.119 --> 00:16:28.178
<v Hilliary Lipsig>Then our toil levels must be below 50%.

00:16:28.658 --> 00:16:47.974
<v Hilliary Lipsig>If you breach that, if your service breaches it, then we can't keep our commitments to our customers, which means that engineering teams must come on and start actively doing that toil until they can basically help automate it away or resolve the issues within the service that is leading to that toil, that manual labor.

00:16:48.364 --> 00:16:54.844
<v Hilliary Lipsig>So there's a kind of a lot of little pieces that go into this and it's about relationships and process primarily.

00:16:55.354 --> 00:16:59.354
<v Hilliary Lipsig>And I would just say you need to implement these things iteratively.

00:16:59.683 --> 00:17:09.193
<v Hilliary Lipsig>If you are looking holistically at the problems in your organization or the problems with your communication, there's gonna be some really bigger issues and then there's gonna be some probably low hanging fruit.

00:17:09.703 --> 00:17:11.503
<v Hilliary Lipsig>And definitely just do that.

00:17:11.509 --> 00:17:11.983
<v Hilliary Lipsig>Do that.

00:17:12.003 --> 00:17:14.193
<v Hilliary Lipsig>Here's a low hanging fruit thing that we could be doing better.

00:17:14.193 --> 00:17:15.243
<v Hilliary Lipsig>Let's go solve that.

00:17:16.144 --> 00:17:35.358
<v Hilliary Lipsig>And I love the cross-functional the cross-functional tiger teams for that, which is also similarly, we have a cross-functional chaos engineering game where we play with services and we break them and do incident response and like engineers and CEE and SRE all take turns playing each other's role in an incident response to learn about things.

00:17:35.358 --> 00:17:38.769
<v Hilliary Lipsig>And we usually get good insights into our software out of those games as well.

00:17:39.048 --> 00:17:41.989
<v Hilliary Lipsig>Which results in better, more re resilient and reliant software.

00:17:43.249 --> 00:17:43.429
<v Jim Hirschauer>Wow.

00:17:43.433 --> 00:17:51.503
<v Jim Hirschauer>I feel like maybe I need to have you back on the show for a completely separate show about that topic chaos engineering and, and working on resiliency.

00:17:51.844 --> 00:17:53.344
<v Jim Hirschauer>That's a huge topic in and of itself.

00:17:54.838 --> 00:17:55.679
<v Hilliary Lipsig>Yeah, it is.

00:17:55.769 --> 00:18:02.429
<v Hilliary Lipsig>And these are all just pieces of how we're putting together this bigger holistic data story at Red Hat, right?

00:18:02.798 --> 00:18:17.038
<v Hilliary Lipsig>That's one of the things that feeds into our overall service health index that we're working towards, and our overall data story of here's here is how we are doing, here's how things are, and then ultimately working towards even more proactive work.

00:18:18.009 --> 00:18:18.223
<v Jim Hirschauer>Yeah.

00:18:18.523 --> 00:18:22.294
<v Jim Hirschauer>You know so I do work for Harness and what you said really resonated with me.

00:18:22.354 --> 00:18:28.564
<v Jim Hirschauer>At Harness we are building our software to align with exactly with what you were just talking about.

00:18:28.564 --> 00:18:39.453
<v Jim Hirschauer>So, you know, trying to make sure those processes are in place and automating the process across the software delivery lifecycle to ensure quality code, reliable code.

00:18:39.769 --> 00:18:45.558
<v Jim Hirschauer>Making its way through the, the life cycle so that when it hits production, you're in much better shape overall.

00:18:45.858 --> 00:18:50.028
<v Jim Hirschauer>So like from a a philosophical perspective, I think we're completely aligned there.

00:18:50.088 --> 00:18:55.493
<v Jim Hirschauer>It's really hard to do in practice, without good tooling to help you.

00:18:55.499 --> 00:19:02.919
<v Jim Hirschauer>Process can be very difficult in and of itself and it's hard to control that unless you actually like automate it in some ways what we've been finding.

00:19:03.729 --> 00:19:04.719
<v Hilliary Lipsig>Yeah, absolutely.

00:19:04.719 --> 00:19:05.499
<v Hilliary Lipsig>A hundred percent.

00:19:05.979 --> 00:19:09.973
<v Hilliary Lipsig>And there are a lot of pieces of the things that I talked about that are automated.

00:19:10.483 --> 00:19:12.943
<v Hilliary Lipsig>And even as much as the chaos engineering stuff, right?

00:19:12.943 --> 00:19:15.163
<v Hilliary Lipsig>Lots of chaos, engineering things exist.

00:19:15.513 --> 00:19:17.943
<v Hilliary Lipsig>We run it as a game because it's a team building as well.

00:19:18.273 --> 00:19:18.394
<v Hilliary Lipsig>Yeah.

00:19:18.473 --> 00:19:22.884
<v Hilliary Lipsig>And so that also is fun cuz people get to be red team and they have to like reverse engineer their software.

00:19:22.884 --> 00:19:23.634
<v Hilliary Lipsig>How am I gonna break it?

00:19:23.753 --> 00:19:32.713
<v Hilliary Lipsig>We, we run it as a live game, but there's all kinds of automation opportunities places where GitOps can really we use GitOps to actually solve alerts in the SRE teams at Red Hat.

00:19:33.124 --> 00:19:37.653
<v Hilliary Lipsig>So places where things like that can really come in and, and bolster the actions.

00:19:37.659 --> 00:19:42.153
<v Hilliary Lipsig>Automation should really be about making your humans more efficient, right?

00:19:42.483 --> 00:19:46.784
<v Hilliary Lipsig>And so any, any kind of automated tool that does that is probably a great idea.

00:19:47.743 --> 00:19:48.374
<v Jim Hirschauer>Absolutely.

00:19:49.044 --> 00:19:55.544
<v Jim Hirschauer>Alright, well, we are out of time on our main topic, and it's been really interesting and insightful and I, I really mean it.

00:19:55.544 --> 00:20:02.773
<v Jim Hirschauer>I think I would love to have you back on the show at a later time where we could discuss more about these, you know, chaos engineering game days that you all put on.

00:20:02.773 --> 00:20:04.334
<v Jim Hirschauer>I'd love to hear some detail about that.

00:20:04.334 --> 00:20:06.913
<v Jim Hirschauer>So if you're willing to love to have you back on the show.

00:20:07.693 --> 00:20:08.624
<v Hilliary Lipsig>Oh yeah, absolutely.

00:20:08.624 --> 00:20:11.144
<v Hilliary Lipsig>That's it was an initiative I'm extremely proud of.

00:20:11.203 --> 00:20:12.523
<v Hilliary Lipsig>It was actually one of mine.

00:20:12.523 --> 00:20:13.874
<v Hilliary Lipsig>It's not like I came up with the game.

00:20:13.874 --> 00:20:16.663
<v Hilliary Lipsig>It follows the capture, the flag style pattern.

00:20:16.763 --> 00:20:17.374
<v Hilliary Lipsig>Mm-hmm.

00:20:17.624 --> 00:20:20.443
<v Hilliary Lipsig>But the initiative was mine and it's caught on and it's gone very well.

00:20:20.443 --> 00:20:21.403
<v Hilliary Lipsig>I'm very proud of that one.

00:20:21.884 --> 00:20:22.304
<v Jim Hirschauer>Okay.

00:20:22.663 --> 00:20:26.273
<v Jim Hirschauer>Yeah, we will definitely save that topic and talk about that on a future show.

00:20:26.513 --> 00:20:30.094
<v Jim Hirschauer>Right now we're gonna transition to just for fun, number two.

00:20:30.693 --> 00:20:33.604
<v Jim Hirschauer>So, Hilliary, outside of work, what's your favorite hobby?

00:20:34.874 --> 00:20:37.094
<v Hilliary Lipsig>My favorite hobby is called HEMA.

00:20:37.153 --> 00:20:40.034
<v Hilliary Lipsig>It is Historical European martial Arts.

00:20:40.784 --> 00:20:43.034
<v Hilliary Lipsig>And it is a style of sword fighting.

00:20:43.064 --> 00:20:52.854
<v Hilliary Lipsig>Specifically I'm learning what is called the Meyer System, and it is a late German style of fencing, like what Knights, you know, would've done and would've trained in.

00:20:53.203 --> 00:20:53.693
<v Hilliary Lipsig>Okay.

00:20:53.749 --> 00:20:58.334
<v Hilliary Lipsig>And it also, so it includes in addition to the sword techniques, also grappling and wrestling techniques as well.

00:20:58.824 --> 00:20:59.413
<v Hilliary Lipsig>Oh, wow.

00:20:59.463 --> 00:21:00.574
<v Hilliary Lipsig>It's very fun.

00:21:00.733 --> 00:21:07.179
<v Hilliary Lipsig>For people who follow me on Twitter, you will see there's a clip from my first tournament, first match of me just getting whacked on the head.

00:21:07.689 --> 00:21:08.739
<v Jim Hirschauer>And what's your, what's your Twitter?

00:21:09.098 --> 00:21:10.479
<v Jim Hirschauer>What's your Twitter handle real quick.

00:21:11.108 --> 00:21:14.278
<v Hilliary Lipsig>Caffeinated integrations into caffeinate.

00:21:14.308 --> 00:21:20.108
<v Hilliary Lipsig>It's at int the number two and then Caffeinate(@Int2Caffeinate), which I cannot spell because I have dyslexia.

00:21:20.804 --> 00:21:21.239
<v Jim Hirschauer>Okay.

00:21:21.409 --> 00:21:22.358
<v Jim Hirschauer>No, no problem.

00:21:22.598 --> 00:21:23.888
<v Jim Hirschauer>I think people will be able to find you.

00:21:25.028 --> 00:21:25.568
<v Hilliary Lipsig>Probably.

00:21:25.568 --> 00:21:26.169
<v Hilliary Lipsig>Probably, yeah.

00:21:26.169 --> 00:21:28.854
<v Hilliary Lipsig>So It's a very fun somewhat expensive sport.

00:21:28.884 --> 00:21:39.013
<v Hilliary Lipsig>We fight with the swords are 48 inches long, so that's four feet in the Imperial system and about a meter and a third in the metric system.

00:21:39.644 --> 00:21:44.378
<v Hilliary Lipsig>And yeah, so they're kind of heavy.

00:21:44.509 --> 00:21:44.989
<v Hilliary Lipsig>They're big.

00:21:44.989 --> 00:21:45.048
<v Hilliary Lipsig>Yeah.

00:21:45.588 --> 00:21:54.858
<v Hilliary Lipsig>And it's great stress relief though because when you have all the armor on and everybody you know, is fighting with control, that's a really big thing with the, the sport is you must use control.

00:21:54.858 --> 00:21:57.828
<v Hilliary Lipsig>Cause the swords are heavy and dangerous even though they're training swords.

00:21:58.489 --> 00:22:01.999
<v Hilliary Lipsig>And then after that it's great stress relief cuz you're just hitting your friends with a really big stick.

00:22:04.519 --> 00:22:05.058
<v Jim Hirschauer>I love it.

00:22:05.058 --> 00:22:08.209
<v Jim Hirschauer>It's, I've, I had never heard of this before you mentioned it.

00:22:08.388 --> 00:22:12.519
<v Jim Hirschauer>It's, it's like this whole new world of activity that I had no idea existed.

00:22:12.578 --> 00:22:13.088
<v Jim Hirschauer>It's amazing.

00:22:14.439 --> 00:22:16.689
<v Hilliary Lipsig>It is an extremely fun sport.

00:22:16.689 --> 00:22:18.429
<v Hilliary Lipsig>It is a very friendly sport.

00:22:18.429 --> 00:22:26.638
<v Hilliary Lipsig>The people in HEMA are, it's just the type of culture where after a tournament match, right, you've just been just wailing on each other, right.

00:22:27.413 --> 00:22:30.023
<v Hilliary Lipsig>You hug, you give a just big hug, just hug it out, right?

00:22:30.023 --> 00:22:30.773
<v Hilliary Lipsig>Hold the whole thing.

00:22:30.773 --> 00:22:34.163
<v Hilliary Lipsig>People are smiling, they're hugging, you're sweaty and disgusting.

00:22:34.163 --> 00:22:41.733
<v Hilliary Lipsig>This is a stranger, and you're still full force as hard as you can hugging this person because that's like the type of the type of attitude that it has.

00:22:41.733 --> 00:22:47.403
<v Hilliary Lipsig>And if you've done other types of martial arts, I will tell you that the tournament vibes are not the same.

00:22:47.403 --> 00:22:55.384
<v Hilliary Lipsig>This is a very joyful sport and some of the absolute coolest people I have ever met including my best friend and her husband and my own husband participate.

00:22:56.784 --> 00:22:59.183
<v Jim Hirschauer>It sounds amazing from the way you describe it.

00:22:59.243 --> 00:23:00.743
<v Jim Hirschauer>I, I, I definitely wanna check it out.

00:23:00.743 --> 00:23:05.124
<v Jim Hirschauer>I live in Austin, so it's highly likely there's a, a place near me that can teach me this.

00:23:05.124 --> 00:23:06.354
<v Jim Hirschauer>And it sounds like it's good exercise.

00:23:06.354 --> 00:23:09.713
<v Jim Hirschauer>It's, it's, I'd imagine it's really strenuous.

00:23:10.388 --> 00:23:11.989
<v Hilliary Lipsig>It is a full body exercise.

00:23:11.993 --> 00:23:16.388
<v Hilliary Lipsig>Like you, in order to be doing this correctly, you must be engaging like every single muscle.

00:23:16.388 --> 00:23:21.878
<v Hilliary Lipsig>I was at a class last night and everything hurts, but it's like the good type of hurt.

00:23:22.219 --> 00:23:22.709
<v Jim Hirschauer>Yeah.

00:23:23.749 --> 00:23:25.038
<v Jim Hirschauer>Alright.

00:23:25.419 --> 00:23:25.989
<v Jim Hirschauer>Okay.

00:23:26.088 --> 00:23:31.169
<v Jim Hirschauer>Listen, Hillary, thank you so much for being on the show, sharing your new word with us.

00:23:31.169 --> 00:23:33.298
<v Jim Hirschauer>I love that"ooming" as a verb.

00:23:33.719 --> 00:23:38.009
<v Jim Hirschauer>So remember that our listeners need to start using that whenever they see Out Of Memory exceptions.

00:23:38.469 --> 00:23:45.098
<v Jim Hirschauer>I love that you shared a completely new sport and hobby with me, so I'm excited to, to dig into that.

00:23:45.519 --> 00:23:48.528
<v Jim Hirschauer>And your main topic was just incredibly insightful.

00:23:48.534 --> 00:23:59.453
<v Jim Hirschauer>So, I think it's something that I never really considered all that we can do with observability data and all that's possible and all the really good reasons for us that we should do it.

00:23:59.453 --> 00:24:02.544
<v Jim Hirschauer>So you're very humble, but you are super insightful.

00:24:02.548 --> 00:24:05.273
<v Jim Hirschauer>So I just wanted to thank you for, for everything that you shared today.

00:24:06.653 --> 00:24:08.064
<v Hilliary Lipsig>Well, thank you so much for having me on.

00:24:08.064 --> 00:24:10.943
<v Hilliary Lipsig>This was a real joy for me and I'd be happy to come back anytime.

00:24:11.394 --> 00:24:12.084
<v Jim Hirschauer>Fantastic.

00:24:12.084 --> 00:24:26.513
<v Jim Hirschauer>I'm looking forward to it and to all of our listeners, if you are an SRE or if you're in a related role and you want to be a guest speaker on ShipTalk, please send an email to podcast@shiptalk.io and we'll get back to you.

00:24:27.199 --> 00:24:27.888
<v Jim Hirschauer>That's all for now.

00:24:28.009 --> 00:24:28.578
<v Jim Hirschauer>Until next time.