WEBVTT

00:00:10.740 --> 00:00:12.470
<v Jim Hirschauer>Welcome to Ship Talk, the SRE edition.

00:00:12.794 --> 00:00:14.564
<v Jim Hirschauer>I'm Jim Hirschauer, your host for today.

00:00:15.015 --> 00:00:19.605
<v Jim Hirschauer>And Ship Talk is a DevOps podcast, brought to you by Harness software delivery platform.

00:00:19.964 --> 00:00:23.460
<v Jim Hirschauer>And in the SRE edition we focus on reliability topics.

00:00:23.483 --> 00:00:25.463
<v Jim Hirschauer>My guest today is Adrianne Cockcroft.

00:00:25.492 --> 00:00:26.632
<v Jim Hirschauer>Adrian, welcome to the show.

00:00:27.713 --> 00:00:28.088
<v Adrian Cockcroft>Hi there.

00:00:28.100 --> 00:00:28.699
<v Adrian Cockcroft>Great to be here.

00:00:29.210 --> 00:00:30.199
<v Jim Hirschauer>Appreciate you coming on.

00:00:30.277 --> 00:00:38.258
<v Jim Hirschauer>Look, I'm sure most of our listeners already know who you are, but if you don't mind, could you please take a minute to share a little bit about your background and what you're up to these days?

00:00:38.314 --> 00:00:50.131
<v Adrian Cockcroft>Well I've got a long career so it's hard to fit it into a few seconds But I retired last summer from Amazon and I'm now doing advisory and consulting kind of work and sort of semi-retired state.

00:00:50.131 --> 00:00:53.505
<v Adrian Cockcroft>And I was at Amazon as a VP for about six years.

00:00:53.505 --> 00:00:59.813
<v Adrian Cockcroft>Did open source and sustainability and a whole lot around Just all the helping customers move to cloud basically.

00:01:00.353 --> 00:01:07.168
<v Adrian Cockcroft>Before I was at battery Ventures VC firm, middle of all of the sort of containerization of computing and all that kind of stuff.

00:01:07.168 --> 00:01:15.328
<v Adrian Cockcroft>And most people know me cuz I was at Netflix for seven years and I, I kind of feel like the last decade I've just been explaining Netflix to people over and over again.

00:01:15.347 --> 00:01:15.787
<v Adrian Cockcroft>Right.

00:01:15.867 --> 00:01:21.658
<v Adrian Cockcroft>Cause we seemed like we invented a whole bunch of stuff there or figured out how to do things that people seem to find fascinating.

00:01:21.957 --> 00:01:22.768
<v Adrian Cockcroft>So that's pretty much it.

00:01:22.768 --> 00:01:27.423
<v Adrian Cockcroft>And then back in the long, long history, I was at Sun and eBay and a few other things.

00:01:28.233 --> 00:01:30.423
<v Jim Hirschauer>Well, congratulations on an absolutely amazing career.

00:01:30.423 --> 00:01:32.643
<v Jim Hirschauer>There's some fantastic highlights to your career for sure.

00:01:33.543 --> 00:01:33.692
<v Jim Hirschauer>Thanks.

00:01:34.143 --> 00:01:45.632
<v Jim Hirschauer>So before we get into the, the heavy meaty topic that I'd like to talk about today, we have this section that we do called just for fun on the show, and I ask people interesting things about themselves.

00:01:45.632 --> 00:01:52.521
<v Jim Hirschauer>So Adrian, I'd love to know, I'm sure you have lots of interesting things going on in your life outside of just work and technology.

00:01:52.521 --> 00:01:53.582
<v Jim Hirschauer>What's your favorite hobby?

00:01:55.408 --> 00:02:02.361
<v Adrian Cockcroft>So I've been I like music and I've been, I liked to have a hobby that wasn't sitting at the keyboard.

00:02:02.962 --> 00:02:08.991
<v Adrian Cockcroft>So yeah, I could have done computer music, but that would've been more time sitting at the keyboard, and that was the thing I wanted to get away from.

00:02:08.997 --> 00:02:16.611
<v Adrian Cockcroft>So back when I was in high school, I played bass guitar and a heavy metal band, actually a heavy rock band called Granite was my first band.

00:02:16.641 --> 00:02:20.301
<v Adrian Cockcroft>So bad puns have been a feature of my naming schemes ever since.

00:02:20.331 --> 00:02:23.581
<v Adrian Cockcroft>then actually played a few gigs with a few of the bands.

00:02:23.677 --> 00:02:29.057
<v Adrian Cockcroft>And then I stopped playing basically when I got a job and got married and got busy with life.

00:02:29.074 --> 00:02:40.800
<v Adrian Cockcroft>More recently I've been playing a little bit more, and now that I quote retired, I, I started taking guitar lessons to actually learn the music theory that I never really knew, cuz I was just trying to keep up with people.

00:02:42.496 --> 00:02:49.688
<v Adrian Cockcroft>They say bass players hang around with musicians and drummers hang around with bass players and hit things So we're kind of you don't, it's pretty easy to play bass.

00:02:49.688 --> 00:02:52.491
<v Adrian Cockcroft>You can just learn some patterns and try to copy stuff.

00:02:52.497 --> 00:03:03.747
<v Adrian Cockcroft>But now, you know, I have, I keep acquiring more random equipment and trying to do things and I, I went through a bunch of old archives of stuff I was doing 40 years ago and, and put it on on SoundCloud.

00:03:03.747 --> 00:03:10.194
<v Adrian Cockcroft>So if anyone's desperately wants to hear what weird things I put up there, there's a whole lot of archival stuff.

00:03:10.194 --> 00:03:18.046
<v Adrian Cockcroft>And then a few more recent things that that I've been playing around with on soundcloud slash Adrian.Cockcroft or something similar so you can find it.

00:03:18.646 --> 00:03:22.735
<v Adrian Cockcroft>So that's, that's kind of what I do when I'm not into to that.

00:03:22.824 --> 00:03:26.664
<v Adrian Cockcroft>And then the other stuff is mostly out playing around with cars and the usual things that people do.

00:03:26.694 --> 00:03:32.204
<v Adrian Cockcroft>But that's been my like I'm actually taking lessons to try and learn what I'm doing rather than just stumbling around.

00:03:32.503 --> 00:03:33.824
<v Jim Hirschauer>Yeah, I can totally respect that.

00:03:33.824 --> 00:03:42.774
<v Jim Hirschauer>I've, I've also played guitar or tried to play guitar for many years and have never gotten really any formal training, so I'm a little bit jealous right now that you're getting to head down that path.

00:03:42.782 --> 00:03:44.742
<v Jim Hirschauer>Do you have any favorite guitarists by the way?

00:03:44.758 --> 00:03:46.799
<v Adrian Cockcroft>I like a pretty wide variety of things.

00:03:46.799 --> 00:03:49.288
<v Adrian Cockcroft>I just, you know, Jeff Beck passed away recently.

00:03:49.288 --> 00:03:55.709
<v Adrian Cockcroft>I just went and listened to the entire back catalog to kind of realize just how much feel he had that he could put into the guitar.

00:03:55.769 --> 00:03:55.979
<v Adrian Cockcroft>Yeah.

00:03:55.991 --> 00:03:56.472
<v Adrian Cockcroft>There's a.

00:03:57.312 --> 00:04:01.602
<v Adrian Cockcroft>Guitarist, most people don't know, called Eric Tessmer, who plays in Austin.

00:04:01.602 --> 00:04:03.942
<v Adrian Cockcroft>If you're ever in Austin, you can usually find him playing.

00:04:03.942 --> 00:04:10.331
<v Adrian Cockcroft>He's, he's a, like a Stevie Ray Vaughan, like guitarist, but never kind of got got to be hugely popular but i, I like him a lot.

00:04:10.572 --> 00:04:20.038
<v Adrian Cockcroft>And then more into the sort of prog rock space, sort of Robert Fripp and things that no one has any, no one else can play, basically Alright.

00:04:20.038 --> 00:04:21.877
<v Adrian Cockcroft>Complicated stuff like that.

00:04:21.877 --> 00:04:25.314
<v Adrian Cockcroft>That's kind of the the, the, the scale of it, all kinds of different things.

00:04:25.553 --> 00:04:26.033
<v Jim Hirschauer>Awesome.

00:04:26.033 --> 00:04:29.978
<v Jim Hirschauer>Well, I actually live in Austin, so I'm gonna go see if I can find Eric and and hear a show.

00:04:30.093 --> 00:04:32.523
<v Adrian Cockcroft>Yeah, he's been, we first saw him in 2005.

00:04:32.523 --> 00:04:33.483
<v Adrian Cockcroft>He's been there a long time.

00:04:33.543 --> 00:04:35.312
<v Adrian Cockcroft>He's been, he's, he played some great gigs.

00:04:35.523 --> 00:04:36.512
<v Jim Hirschauer>A great recommendation.

00:04:37.242 --> 00:04:37.543
<v Jim Hirschauer>Alright.

00:04:37.562 --> 00:04:40.043
<v Jim Hirschauer>So let's jump into the main topic, if you don't mind.

00:04:40.149 --> 00:04:41.528
<v Jim Hirschauer>Here, here's what I wanted to talk about.

00:04:41.528 --> 00:04:46.699
<v Jim Hirschauer>So, in the latest state of DevOps report that I read they introduced reliability as.

00:04:47.125 --> 00:04:55.915
<v Jim Hirschauer>Quote unquote, the fifth metric, their fifth important metric to go along with deployment frequency, lead time for changes, time to restore service and change failure rate.

00:04:56.425 --> 00:04:58.675
<v Jim Hirschauer>And I'm gonna quote what they said in the report.

00:04:58.675 --> 00:05:01.134
<v Jim Hirschauer>I'm just gonna read it directly because I think it's really interesting.

00:05:01.584 --> 00:05:08.175
<v Jim Hirschauer>They said, when reliability is poor, Software delivery performance does not predict organizational success.

00:05:08.564 --> 00:05:15.165
<v Jim Hirschauer>However, with better reliability, we begin to see positive influence of software delivery on business success.

00:05:15.475 --> 00:05:31.634
<v Jim Hirschauer>And I think that's a huge statement because we've seen so many companies really laser focused on these four metrics and so many engineering organizations thinking that if they can achieve these four metrics and, and improve on these four metrics, that that is going to directly impact their success as a business.

00:05:31.935 --> 00:05:34.154
<v Jim Hirschauer>But this statement, seems to contradict that a little bit.

00:05:34.154 --> 00:05:40.214
<v Jim Hirschauer>It says, you know, those are important, but there's this other metric reliability that you need to focus on to really be successful.

00:05:40.514 --> 00:05:42.375
<v Jim Hirschauer>So I'd love to hear what you thought about those findings.

00:05:42.507 --> 00:05:45.418
<v Adrian Cockcroft>One of the things is they didn't really define what reliability is.

00:05:46.367 --> 00:05:46.697
<v Adrian Cockcroft>Yeah.

00:05:46.848 --> 00:05:46.968
<v Adrian Cockcroft>Yeah.

00:05:46.997 --> 00:05:47.778
<v Adrian Cockcroft>They, they, yeah.

00:05:47.778 --> 00:05:49.038
<v Adrian Cockcroft>It's sort of a self-reported.

00:05:49.038 --> 00:06:05.247
<v Adrian Cockcroft>Do you think you have good reliability or not seem to be kind of what was being run off of rather than is there a particular metric and, you know, everyone measures it differently, but it's a little bit self-reported, so it may be that there's some correlation in there that's affecting it.

00:06:05.249 --> 00:06:19.754
<v Adrian Cockcroft>The thing I think is in interesting is that if the system's reliable, And it's resilient then, and it's, it basically means it's able to absorb small shocks and things going wrong without failure, because failure's there all the time.

00:06:19.754 --> 00:06:24.584
<v Adrian Cockcroft>The question is, is your system capable of masking that failure to the end users?

00:06:25.543 --> 00:06:29.473
<v Adrian Cockcroft>If that's true, then you get to try things out more aggressively.

00:06:30.553 --> 00:06:41.564
<v Adrian Cockcroft>If you're a developer and you have to go through a, you know, a really long, complicated QA process to get something out because everyone's terrified that you're gonna break the site, it slows down the pace of innovation.

00:06:41.653 --> 00:06:41.954
<v Adrian Cockcroft>Sure.

00:06:42.913 --> 00:06:52.004
<v Adrian Cockcroft>If you've got something where you can deploy it and if it breaks the system, sort of quarantines it out and you know, maybe one customer got a retry, nobody notices.

00:06:52.004 --> 00:06:52.274
<v Adrian Cockcroft>Right.

00:06:52.634 --> 00:06:56.923
<v Adrian Cockcroft>That's a very different world, and what I think is that if you've got really.

00:06:57.944 --> 00:07:02.473
<v Adrian Cockcroft>So reliability characteristics, then you can actually run faster.

00:07:02.473 --> 00:07:05.053
<v Adrian Cockcroft>And the sort of analogy here is running with scissors, right?

00:07:05.053 --> 00:07:05.233
<v Adrian Cockcroft>Yeah.

00:07:05.324 --> 00:07:07.303
<v Adrian Cockcroft>They say, don't run with scissors, you'll hurt yourself.

00:07:07.543 --> 00:07:07.634
<v Adrian Cockcroft>Mm-hmm.

00:07:07.874 --> 00:07:08.084
<v Adrian Cockcroft>right?

00:07:08.084 --> 00:07:08.894
<v Adrian Cockcroft>Everyone says that.

00:07:09.014 --> 00:07:10.004
<v Adrian Cockcroft>And then, okay.

00:07:10.009 --> 00:07:15.884
<v Adrian Cockcroft>So if you like wrap the scissors in bubble wrap or so, or some kind of case so that they're safe, you can run with scissors.

00:07:16.093 --> 00:07:16.244
<v Adrian Cockcroft>Yeah.

00:07:16.244 --> 00:07:16.603
<v Adrian Cockcroft>Right.

00:07:16.603 --> 00:07:19.213
<v Adrian Cockcroft>But you've, you've got some compensating thing around it.

00:07:19.213 --> 00:07:21.884
<v Adrian Cockcroft>So it's, it's more like making it safe to go.

00:07:23.293 --> 00:07:27.408
<v Adrian Cockcroft>and that's a lot of the value of resilience from a sort of a business point of view.

00:07:28.218 --> 00:07:40.831
<v Adrian Cockcroft>It just just means that you spend less time firefighting and you are, you can do things that would in other environments would be dangerous or actually safe in in environments that have good sort of resilience and containment properties.

00:07:40.920 --> 00:07:41.310
<v Jim Hirschauer>Right.

00:07:41.790 --> 00:07:44.581
<v Jim Hirschauer>You, you made a couple of statements in there I'd like to unpack a little bit.

00:07:44.940 --> 00:07:45.781
<v Jim Hirschauer>First one.

00:07:46.005 --> 00:07:52.906
<v Jim Hirschauer>You called out rightly so that this reliability metric isn't actually defined as a metric in the report.

00:07:52.906 --> 00:07:54.435
<v Jim Hirschauer>It's, it's just called reliability.

00:07:54.440 --> 00:08:01.485
<v Jim Hirschauer>So do you have any recommendations for folks on what they should be doing to define reliability within their organizations?

00:08:03.016 --> 00:08:09.560
<v Adrian Cockcroft>Yeah, there's, there's a long topic here, but the classical sort of definition is up and down If you've got a monolith right?

00:08:09.560 --> 00:08:10.790
<v Adrian Cockcroft>It's either up or it's down.

00:08:10.790 --> 00:08:12.800
<v Adrian Cockcroft>If it crashes, it's down, right?

00:08:12.800 --> 00:08:12.891
<v Adrian Cockcroft>Mm-hmm.

00:08:13.130 --> 00:08:20.031
<v Adrian Cockcroft>And so people say, oh, if you, it's sort of this state where it's very clearly you say This thing is either working or not working.

00:08:20.391 --> 00:08:22.430
<v Adrian Cockcroft>Very clean division between the two.

00:08:22.430 --> 00:08:26.480
<v Adrian Cockcroft>You can measure the amount of time of it's in each state and say, that's your uptime, right?

00:08:27.350 --> 00:08:31.490
<v Adrian Cockcroft>What actually happens in more sophisticated systems is they degrade slowly.

00:08:31.850 --> 00:08:33.846
<v Adrian Cockcroft>So a little bit of it's slightly broken.

00:08:33.905 --> 00:08:37.355
<v Adrian Cockcroft>Some number of customers aren't getting served, but most people are.

00:08:37.385 --> 00:08:39.255
<v Adrian Cockcroft>Okay, so it's not down, right?

00:08:39.260 --> 00:08:39.515
<v Adrian Cockcroft>Right.

00:08:39.785 --> 00:08:39.846
<v Adrian Cockcroft>Yeah.

00:08:39.995 --> 00:08:49.443
<v Adrian Cockcroft>But your, you know, 0.1% of customers are currently impacted because that feature they were trying to use or the, the route to their ISPs is down or something like that.

00:08:49.470 --> 00:08:55.710
<v Adrian Cockcroft>So it's very difficult to measure that with a straightforward up down availability metric, which is based on time.

00:08:56.309 --> 00:08:58.960
<v Adrian Cockcroft>So what I like to do is based on success rate.

00:08:59.490 --> 00:09:03.764
<v Adrian Cockcroft>So if you look at, and the classic thing we did at Netflix was streaming starts, right?

00:09:03.794 --> 00:09:04.075
<v Adrian Cockcroft>Mm-hmm.

00:09:04.154 --> 00:09:09.705
<v Adrian Cockcroft>the core metric they have is how many people started tried to start a movie and did they succeed in starting a movie?

00:09:09.705 --> 00:09:12.075
<v Adrian Cockcroft>You hit play, did you get your movie right?

00:09:12.225 --> 00:09:12.315
<v Adrian Cockcroft>Right.

00:09:13.065 --> 00:09:14.985
<v Adrian Cockcroft>And that was the metric.

00:09:14.985 --> 00:09:15.914
<v Adrian Cockcroft>And it goes up and down.

00:09:15.914 --> 00:09:18.735
<v Adrian Cockcroft>The rate is quite low at, you know, in the US at 3:00 AM.

00:09:19.455 --> 00:09:19.695
<v Adrian Cockcroft>Right.

00:09:19.695 --> 00:09:22.605
<v Adrian Cockcroft>It's like security guards watching it on their gameboys, whatever.

00:09:23.195 --> 00:09:26.985
<v Adrian Cockcroft>and which is actually, I think pretty much what it turned out to be at the time.

00:09:27.115 --> 00:09:27.475
<v Adrian Cockcroft>Yeah.

00:09:27.480 --> 00:09:35.054
<v Adrian Cockcroft>And then, you know, and peak time is whatever, you know, Sunday evening was typically the peak time, which is when the site used to fall over.

00:09:35.054 --> 00:09:39.615
<v Adrian Cockcroft>And that's annoying because everyone wanted to be at home watching TV with their kids on Sunday evening.

00:09:40.004 --> 00:09:40.095
<v Adrian Cockcroft>Right.

00:09:40.095 --> 00:09:43.514
<v Adrian Cockcroft>And that was usually when Netflix collapsed cuz it was a new all time record peak.

00:09:43.894 --> 00:09:53.144
<v Adrian Cockcroft>You know, you get these kinds of cyclic effects, but what we cared about was what we generated was a metric, which was the number of times that we thought that we were missing.

00:09:54.225 --> 00:09:55.664
<v Adrian Cockcroft>And then we did that percentage.

00:09:55.664 --> 00:10:01.424
<v Adrian Cockcroft>So it was sort of, you know, how many nines or whatever it was, some percentage of uptime, three or four nines or something like that.

00:10:01.784 --> 00:10:06.674
<v Adrian Cockcroft>But it was of attempts to deliver value to the customer that succeeded or failed.

00:10:07.065 --> 00:10:14.481
<v Adrian Cockcroft>So that's where I'd like to see people focus on reliability as based on a customer facing value delivery metric.

00:10:14.812 --> 00:10:18.142
<v Adrian Cockcroft>And it's typically there's some value being delivered by a website.

00:10:18.142 --> 00:10:22.221
<v Adrian Cockcroft>And then there's customer signup, and that's the other flow that's very standard.

00:10:22.221 --> 00:10:26.091
<v Adrian Cockcroft>You want to make sure customers successfully sign up right, to whatever your service is.

00:10:26.692 --> 00:10:35.782
<v Adrian Cockcroft>So those are probably the two canonical things that I would typically, you'd have two, and then there may be more if you've got a more complicated product or set of products.

00:10:35.782 --> 00:10:43.094
<v Adrian Cockcroft>But it looks something like the ability to get into the product and the ability to deliver value with the product are the things that need to be reliable.

00:10:43.423 --> 00:10:43.813
<v Jim Hirschauer>Got it.

00:10:43.874 --> 00:11:00.115
<v Jim Hirschauer>What I've seen companies do is use SLOs, assign SLIs and SLOs to these type of metrics and ultimately the the Google SRE handbook that says you should end up with error budgets as well, and you can use error budgets to do interesting things with.

00:11:01.005 --> 00:11:03.782
<v Jim Hirschauer>So I'm curious to get your take on error budgets.

00:11:03.782 --> 00:11:12.841
<v Jim Hirschauer>I've had conversations with a lot of folks who say they were never able to implement error budgets, they never actually got error budgets to function and provide the business value that was expected out of them.

00:11:12.865 --> 00:11:20.595
<v Jim Hirschauer>So, you know, have you ever implemented error budgets or do you have any advice for anyone trying to implement error budgets and really get the true value out of those?

00:11:20.615 --> 00:11:21.039
<v Adrian Cockcroft>Mm-hmm.

00:11:21.789 --> 00:11:22.279
<v Adrian Cockcroft>Yeah.

00:11:22.379 --> 00:11:31.139
<v Adrian Cockcroft>The Google SRE book was interesting because it was about three quarters of the same as what Netflix was doing, and a quarter was like the opposite of what Netflix was doing.

00:11:31.200 --> 00:11:31.259
<v Adrian Cockcroft>Okay.

00:11:31.649 --> 00:11:35.370
<v Adrian Cockcroft>So there was, so Netflix at the time I was there didn't have a central SRE team.

00:11:35.945 --> 00:11:40.804
<v Adrian Cockcroft>There wasn't this process of handing off to a central team and defining error budgets.

00:11:40.804 --> 00:11:44.914
<v Adrian Cockcroft>Basically, the teams owned their own code through the entire life cycle.

00:11:44.914 --> 00:11:45.455
<v Adrian Cockcroft>They owned it.

00:11:45.469 --> 00:11:52.158
<v Adrian Cockcroft>They owned a collection of microservices that were fairly stable and weren't changing very much, and they just had them running in the background.

00:11:52.163 --> 00:11:58.849
<v Adrian Cockcroft>And then there were some that they were actively working on, but they owned the business function all the way out to the end, and they were on call for when it broke.

00:12:00.033 --> 00:12:07.371
<v Adrian Cockcroft>And so the central team that was sort of like an SRE team didn't actually own the, didn't make changes to code.

00:12:08.591 --> 00:12:13.611
<v Adrian Cockcroft>Their job was to find out what was broke and call the people that owned that and get them to fix it.

00:12:13.822 --> 00:12:14.032
<v Adrian Cockcroft>Okay.

00:12:14.721 --> 00:12:21.261
<v Adrian Cockcroft>And by, by and, and it's annoying to be on call so people generally got really good at writing code that didn't break.

00:12:21.267 --> 00:12:23.241
<v Adrian Cockcroft>So we created a pain in the right place.

00:12:23.246 --> 00:12:32.922
<v Adrian Cockcroft>So people built code and systems and tooling so that their code, they didn't have to be woken up at 3:00 AM If something broke, there was the system would, would manage it.

00:12:33.476 --> 00:12:35.096
<v Adrian Cockcroft>So I think that's a good principle.

00:12:35.126 --> 00:12:40.616
<v Adrian Cockcroft>It is difficult to get people to take, you know, effectively carry a pager if you're a developer, right?

00:12:40.677 --> 00:12:46.317
<v Adrian Cockcroft>There is a mentality around it, but if you can get that to happen, it causes the whole environment to be much more.

00:12:46.647 --> 00:12:56.981
<v Adrian Cockcroft>The other big difference in Google and Netflix, Google at that time had a lot of graduate hires, a lot of young developers, and we used to wait for them to spend five or 10 years at Google before we hired them at Netflix.

00:12:57.131 --> 00:12:57.282
<v Adrian Cockcroft>Yeah.

00:12:57.341 --> 00:13:00.491
<v Adrian Cockcroft>So Netflix was very much, a much more senior crew.

00:13:00.822 --> 00:13:04.201
<v Adrian Cockcroft>Really, no graduate hires, no interns.

00:13:04.621 --> 00:13:07.022
<v Adrian Cockcroft>So it was much more experienced people.

00:13:07.351 --> 00:13:11.895
<v Adrian Cockcroft>And so there's, and it was a much smaller, sort of built differently from that point of view.

00:13:12.405 --> 00:13:15.946
<v Adrian Cockcroft>So that's kind of some, some of the core differences in the way we did stuff.

00:13:16.375 --> 00:13:22.451
<v Adrian Cockcroft>Now, we did sort of try to meet a quarterly goal on like, whatever it was, four nine s or something.

00:13:22.961 --> 00:13:32.789
<v Adrian Cockcroft>If we'd had a couple of outages that quarter, we would occasionally say, well, let's bump this chaos region failover test to the next quarter in case it goes wrong.

00:13:33.299 --> 00:13:36.200
<v Adrian Cockcroft>So we sort of used error budgets in sort of informal way.

00:13:36.779 --> 00:13:46.559
<v Adrian Cockcroft>That if we'd had a series of outages, we'd maybe pull back on some of the chaos engineering, large scale testing where we were potentially gonna cause an outage.

00:13:47.039 --> 00:13:47.340
<v Adrian Cockcroft>Right.

00:13:47.669 --> 00:13:53.970
<v Adrian Cockcroft>And sort of so I saw a few times, but we're just like, let's do it next month cuz that's next quarter and if we survive that then we're good.

00:13:53.970 --> 00:13:54.509
<v Adrian Cockcroft>But Right.

00:13:54.789 --> 00:13:59.945
<v Adrian Cockcroft>We can sort of even out the customer pain in terms of nines of error rate.

00:14:00.394 --> 00:14:03.934
<v Adrian Cockcroft>So that was, that's sort of informal error budget like thing.

00:14:03.934 --> 00:14:06.634
<v Adrian Cockcroft>But we never had a formal error budget system.

00:14:06.634 --> 00:14:16.105
<v Adrian Cockcroft>And I haven't, I haven't seen, I haven't seen it used that much as in, in a formal way, but I haven't probably looked at many environments.

00:14:16.745 --> 00:14:18.809
<v Adrian Cockcroft>But I mean, there's plenty people have SREs.

00:14:18.809 --> 00:14:20.769
<v Adrian Cockcroft>I don't hear them talking about error budgets that much.

00:14:21.215 --> 00:14:29.654
<v Jim Hirschauer>Yeah, and, and you know, quite honestly your informal process that you just mentioned is really, definitely gets to the heart of what error budgets are intended for and, and how they should work.

00:14:29.722 --> 00:14:40.238
<v Jim Hirschauer>You know, I, I get to talk to a lot of different companies and, it seems to me that one of the failings currently around error budgets is the difficulty in really implementing the process surrounding it.

00:14:40.238 --> 00:14:40.957
<v Jim Hirschauer>It's not so much.

00:14:41.452 --> 00:14:42.982
<v Jim Hirschauer>You know, implementing error budgets.

00:14:42.982 --> 00:14:52.163
<v Jim Hirschauer>After all, it's just fairly simple math, but there is a process that has to be initiated, like, Hey, we're gonna push this off in your case, we're gonna push off this test until next quarter.

00:14:52.493 --> 00:15:00.682
<v Jim Hirschauer>And that seems to be at odds with, you know, developers typically needing to meet timelines, meet deadlines, to release new features and functionality.

00:15:01.192 --> 00:15:05.062
<v Jim Hirschauer>So that becomes, yeah, you know, an interesting aspect to the error budgets.

00:15:05.743 --> 00:15:09.623
<v Adrian Cockcroft>The, the other problem you get is talking to product managers about error budgets.

00:15:10.462 --> 00:15:12.982
<v Adrian Cockcroft>Have real trouble with the, okay, say, well, how, what?

00:15:13.043 --> 00:15:14.903
<v Adrian Cockcroft>What do you want to do when the site's down?

00:15:14.903 --> 00:15:18.982
<v Adrian Cockcroft>No, I don't want it to go down Well, what do you want it to do when it's down?

00:15:19.222 --> 00:15:19.942
<v Adrian Cockcroft>And they just said No.

00:15:19.942 --> 00:15:21.413
<v Adrian Cockcroft>But they don't ever want it to go down.

00:15:21.413 --> 00:15:22.432
<v Adrian Cockcroft>They want it to be perfect.

00:15:22.432 --> 00:15:24.383
<v Adrian Cockcroft>And it says, okay, what degraded states?

00:15:24.383 --> 00:15:32.153
<v Adrian Cockcroft>So you have to kind of get them into a room and not let them out and force them to think about what degraded states are there and what should you be doing?

00:15:32.153 --> 00:15:36.212
<v Adrian Cockcroft>And if this backend service goes down, the system can do this and it can't do that.

00:15:36.638 --> 00:15:38.798
<v Adrian Cockcroft>And you have to kind of plan that into the product.

00:15:38.798 --> 00:15:45.538
<v Adrian Cockcroft>So the product itself has some resilience, baked in, or some progressive feature sort of sort of turnoff.

00:15:46.019 --> 00:15:49.109
<v Adrian Cockcroft>And you know, ultimately if the entire site's down, what are you going to do?

00:15:49.109 --> 00:16:01.308
<v Adrian Cockcroft>And one of the things we did at Netflix, we created a static site that was somewhere else on a totally different set of infrastructure that just had some movies you could watch and that we had rights to.

00:16:01.599 --> 00:16:12.823
<v Adrian Cockcroft>And it had no personalization and there was a mode where if everything was completely down, we could in theory just stand this site up and it would, everyone would redirect to it and you'd be able to just watch some stuff.

00:16:13.813 --> 00:16:21.854
<v Adrian Cockcroft>I'm not sure if it ever really got used very often, but that was kind of, that sort of meant the mental model, the process of getting there cause a lot of things to be sorted out.

00:16:21.854 --> 00:16:28.903
<v Adrian Cockcroft>Like how would you do that and how would you fail over from the main thing to some sort of backup that's at least telling you, sorry, we're down.

00:16:28.903 --> 00:16:30.644
<v Adrian Cockcroft>We're gonna be okay eventually.

00:16:30.764 --> 00:16:34.261
<v Adrian Cockcroft>Yeah, and I think, so Datadog had an outage yesterday, I think.

00:16:34.261 --> 00:16:44.365
<v Adrian Cockcroft>And one of the things, I'm not sure, I actually haven't read the report of why they went down, but their status page actually, people saying their status page was actually still up and was working well, so, so.

00:16:45.399 --> 00:16:47.110
<v Adrian Cockcroft>You have, it's the basic stuff.

00:16:47.110 --> 00:16:50.950
<v Adrian Cockcroft>If you take, if your stainless page goes down, when you go down, you'll, you know, say no.

00:16:51.009 --> 00:16:51.279
<v Adrian Cockcroft>Okay.

00:16:51.279 --> 00:16:52.990
<v Adrian Cockcroft>You've done it, done something seriously wrongly.

00:16:52.990 --> 00:16:53.289
<v Adrian Cockcroft>Right.

00:16:53.289 --> 00:16:57.340
<v Adrian Cockcroft>You have to figure out how to think through what happens in these failure mode.

00:16:57.490 --> 00:16:57.700
<v Adrian Cockcroft>Right.

00:16:57.940 --> 00:16:58.330
<v Jim Hirschauer>Okay.

00:16:59.320 --> 00:17:11.069
<v Jim Hirschauer>So that, that actually transitions into the last thing I really wanted to unpack that you've been talking about is resiliency and chaos engineering is a way to work on the resiliency of your systems.

00:17:11.138 --> 00:17:24.059
<v Jim Hirschauer>A lot of companies that I've talked to or had experience with have used chaos engineering, but it seems to be a very tactical approach where they've had one or two engineers that, that work on the practice and it's very siloed and segmented.

00:17:24.759 --> 00:17:29.355
<v Jim Hirschauer>Do you have any recommendations on how to make that more of a strategic approach within a company?

00:17:30.984 --> 00:17:31.144
<v Adrian Cockcroft>Yeah.

00:17:31.164 --> 00:17:32.855
<v Adrian Cockcroft>I think it ends up being.

00:17:32.935 --> 00:17:33.986
<v Adrian Cockcroft>Like who's responsible?

00:17:33.986 --> 00:17:42.140
<v Adrian Cockcroft>Is it a developer concern or an operations concern, or have you sort of combined those together into a, a true tech DevOps combined organization?

00:17:42.205 --> 00:17:51.086
<v Adrian Cockcroft>And I just go back to now, everyone's had backup data centers for years, and if you talk to people about how often they actually fail over to them, it's vanishingly small.

00:17:51.266 --> 00:17:56.336
<v Adrian Cockcroft>There are a few people who are, banks are regulated to at least show they could do it once a year.

00:17:56.615 --> 00:18:02.265
<v Adrian Cockcroft>Other than that, it's pretty rare to find people that have spent the money on a backup data center.

00:18:02.265 --> 00:18:04.394
<v Adrian Cockcroft>They mostly don't get the value out of it, right?

00:18:04.694 --> 00:18:12.734
<v Adrian Cockcroft>So if you look at bringing that into the modern world, what we really care about is in the cloud we've got all of these backup data centers are just there.

00:18:12.734 --> 00:18:18.674
<v Adrian Cockcroft>If you can figure out how to use them, it's much easier to provision a failover than a physical data center.

00:18:19.125 --> 00:18:21.102
<v Adrian Cockcroft>So how are you using that?

00:18:21.102 --> 00:18:30.559
<v Adrian Cockcroft>The, the standard sort of canonical cloud setup is three zones in a region, and if a zone goes down, your site should stay up, but most people don't test that.

00:18:30.559 --> 00:18:34.069
<v Adrian Cockcroft>So I would always start with, can you take a zone outage?

00:18:34.075 --> 00:18:34.819
<v Adrian Cockcroft>And why can't you?

00:18:34.880 --> 00:18:35.059
<v Adrian Cockcroft>Right.

00:18:35.089 --> 00:18:38.690
<v Adrian Cockcroft>What goes wrong if a zone goes down in your environment?

00:18:38.720 --> 00:18:40.460
<v Adrian Cockcroft>Are you routing traffic correctly?

00:18:40.460 --> 00:18:45.319
<v Adrian Cockcroft>Do you know how to operate on a two zone system in some degraded mode?

00:18:45.349 --> 00:18:56.599
<v Adrian Cockcroft>Because, That was something we sort of ran the test every two weeks at Netflix where we just shut down the zone to prove the site still worked on the other two, and every now and again it would catch something and we'd like rapidly back out of the test.

00:18:56.599 --> 00:18:56.839
<v Adrian Cockcroft>Right.

00:18:57.559 --> 00:18:57.750
<v Adrian Cockcroft>So that's...

00:18:58.924 --> 00:19:05.775
<v Adrian Cockcroft>the most basic thing is if you, if you can survive on two outta three zones, you're in good shape and most people can't even do that.

00:19:06.404 --> 00:19:09.884
<v Adrian Cockcroft>And then once you get that sorted, then you can think about multi-region.

00:19:09.914 --> 00:19:16.105
<v Adrian Cockcroft>I would not even attempt multi-region unless you figured out, unless you can show that you can do a two out of three zone.

00:19:16.615 --> 00:19:17.454
<v Adrian Cockcroft>Resilience, right?

00:19:17.454 --> 00:19:18.384
<v Adrian Cockcroft>It's just mm-hmm.

00:19:18.630 --> 00:19:23.095
<v Adrian Cockcroft>you're not, you're not close to it because you have to be able to disambiguate the failure.

00:19:23.095 --> 00:19:27.654
<v Adrian Cockcroft>That means you have to shut down a region from the failure that you need to shut down a zone.

00:19:27.894 --> 00:19:34.176
<v Adrian Cockcroft>And it's very difficult to figure out which of those is actually the problem you've got in the moment.

00:19:34.987 --> 00:19:37.616
<v Adrian Cockcroft>So, That's kind of one way of doing it.

00:19:37.616 --> 00:19:41.426
<v Adrian Cockcroft>And the other way is to just sort of have a weekly meeting that says, well, what if everything goes down?

00:19:41.426 --> 00:19:49.497
<v Adrian Cockcroft>And just have a bunch of scenarios and do it as a paper environment, and then start building chaos engineering tests into the test environment or staging environment.

00:19:49.826 --> 00:19:56.307
<v Adrian Cockcroft>And eventually, once you've convinced yourself, everything's looking good, I'd create probably a separate production environment in the cloud.

00:19:56.307 --> 00:19:57.086
<v Adrian Cockcroft>You know, another account.

00:19:57.576 --> 00:20:00.277
<v Adrian Cockcroft>And I'd set up the Chaos engineering tools in that account.

00:20:00.336 --> 00:20:05.510
<v Adrian Cockcroft>And this is for the more operations sort of focused stuff, you know, killing machines, killing zones cutting off networks.

00:20:05.510 --> 00:20:07.221
<v Adrian Cockcroft>Cutting off dependencies.

00:20:07.236 --> 00:20:16.107
<v Adrian Cockcroft>So that's, that's the, but the thing that really I think is, is new and interesting, and it's something that harnesses is working at now is continuous resilience.

00:20:16.847 --> 00:20:21.587
<v Adrian Cockcroft>So you take the CI/CD pipeline and you put chaos engineering into the pipeline.

00:20:21.587 --> 00:20:26.327
<v Adrian Cockcroft>So every new build of your microservice goes through a series of chaos tests.

00:20:26.917 --> 00:20:33.191
<v Adrian Cockcroft>to say like what happens if if when it fires up one of its dependencies isn't there or is slow.

00:20:33.280 --> 00:20:33.520
<v Adrian Cockcroft>Yeah.

00:20:33.611 --> 00:20:33.811
<v Adrian Cockcroft>Right, right.

00:20:33.820 --> 00:20:35.050
<v Adrian Cockcroft>You've got a service running.

00:20:35.050 --> 00:20:38.800
<v Adrian Cockcroft>What if the connection to the database, if the database is slow, is a common thing?

00:20:38.806 --> 00:20:39.070
<v Adrian Cockcroft>Right.

00:20:39.401 --> 00:20:41.770
<v Adrian Cockcroft>Most things keel over cuz their database got slow.

00:20:41.851 --> 00:20:42.290
<v Adrian Cockcroft>Right?

00:20:42.296 --> 00:20:42.461
<v Adrian Cockcroft>Right.

00:20:42.671 --> 00:20:46.661
<v Adrian Cockcroft>Or the connection to some authentication service fails or something like that.

00:20:47.480 --> 00:20:49.191
<v Adrian Cockcroft>You should have a graceful degradation.

00:20:49.191 --> 00:21:03.096
<v Adrian Cockcroft>So this is more test driven development, if you like, but what you're doing is you're setting it up as a chaos test in the pipeline so that you can continuously prove that this little piece of the system is resilient to failures around it.

00:21:03.306 --> 00:21:15.615
<v Adrian Cockcroft>And I think that more developer oriented approach is, is a really useful way of extending the chaos engineering principles down to something that's more, more useful and more continuously useful.

00:21:15.945 --> 00:21:27.675
<v Adrian Cockcroft>And then when you get the failover of a region or a zone, whatever, you know that your individual microservice is going to behave sensibly during that process as things are failing and failing over and recovering from it quickly.

00:21:27.736 --> 00:21:28.486
<v Adrian Cockcroft>That kind of stuff.

00:21:29.086 --> 00:21:34.435
<v Jim Hirschauer>Yeah, it's an interesting approach because ultimately there's a cultural aspect here, right?

00:21:34.465 --> 00:21:46.020
<v Jim Hirschauer>Like in your example, If you're gonna pull out a, a zone or a region, there has to be a willingness to deal with the consequences of that because you could have tremendous consequences as a business.

00:21:46.020 --> 00:21:55.604
<v Jim Hirschauer>So it seems like there needs to be buy-in culturally across the business and by making it part of a pipeline that, that seems to kind of embed it within the culture.

00:21:55.604 --> 00:21:59.023
<v Jim Hirschauer>Like there's almost no choice but for it to become part of the culture at that point.

00:22:00.594 --> 00:22:10.913
<v Adrian Cockcroft>Yeah, if you can basically say this, this microservice, if you cut one of its dependencies, it will give you a sane error message, and if you restore that dependency, it will recover quickly.

00:22:11.084 --> 00:22:11.644
<v Adrian Cockcroft>Right.

00:22:12.278 --> 00:22:13.598
<v Adrian Cockcroft>And it won't fall over.

00:22:13.598 --> 00:22:15.459
<v Adrian Cockcroft>You won't have to like restart it from scratch.

00:22:15.459 --> 00:22:19.719
<v Adrian Cockcroft>And there's, you know, cause systems come catatonic quite often once when their dependencies fail.

00:22:19.838 --> 00:22:19.989
<v Adrian Cockcroft>Yeah.

00:22:20.019 --> 00:22:20.199
<v Adrian Cockcroft>Right.

00:22:20.409 --> 00:22:25.568
<v Adrian Cockcroft>So that, that means you've built some things more resilient, but you still have to do that full scale testing.

00:22:25.598 --> 00:22:30.219
<v Adrian Cockcroft>And you know, if you go to upper management and you say, you know, what do you want to happen when the site goes down?

00:22:30.528 --> 00:22:31.088
<v Adrian Cockcroft>Right?

00:22:31.269 --> 00:22:34.239
<v Adrian Cockcroft>Do you want to, do you want it to continue to be down?

00:22:34.239 --> 00:22:52.074
<v Adrian Cockcroft>Do you want it to escalate or do you want it to recover quickly and to be resilient to various types of failure and if they're doing something that matters is either, you know, financially or, or there's a high business value to it being up, then they should be asking the engineering teams culturally, so this thing should stay up.

00:22:52.134 --> 00:22:52.403
<v Adrian Cockcroft>Right?

00:22:52.943 --> 00:22:54.354
<v Adrian Cockcroft>It's important that it stays up.

00:22:54.413 --> 00:22:54.713
<v Adrian Cockcroft>Right?

00:22:54.953 --> 00:22:56.663
<v Adrian Cockcroft>Or, or maybe it doesn't matter.

00:22:57.173 --> 00:22:57.294
<v Adrian Cockcroft>Right.

00:22:57.354 --> 00:23:03.443
<v Adrian Cockcroft>And there's other things are more important because what you're doing is, you know, a movie used to say like, Netflix is just movies.

00:23:03.443 --> 00:23:03.743
<v Adrian Cockcroft>Right.

00:23:04.193 --> 00:23:06.144
<v Adrian Cockcroft>But, and the, it's not the end of the world.

00:23:06.144 --> 00:23:08.352
<v Adrian Cockcroft>If Netflix doesn't work, you could use something else.

00:23:08.362 --> 00:23:11.991
<v Adrian Cockcroft>The way attitude we had was your TV set, you should turn it on.

00:23:11.991 --> 00:23:12.862
<v Adrian Cockcroft>It just works, right?

00:23:13.221 --> 00:23:15.652
<v Adrian Cockcroft>And that was the, it should turn on and just work.

00:23:15.652 --> 00:23:18.291
<v Adrian Cockcroft>You shouldn't have to worry about whether Netflix was up or down.

00:23:18.291 --> 00:23:21.231
<v Adrian Cockcroft>It's like you're just trying to watch tv, right?

00:23:21.231 --> 00:23:21.392
<v Adrian Cockcroft>Yeah.

00:23:21.392 --> 00:23:23.571
<v Adrian Cockcroft>And, and you shouldn't have to think the, the least.

00:23:23.571 --> 00:23:29.237
<v Adrian Cockcroft>You should be unaware of the fact that there's a whole bunch of random, weird technology stuff happening in the background.

00:23:29.237 --> 00:23:30.707
<v Adrian Cockcroft>There should be no artifacts.

00:23:30.826 --> 00:23:34.457
<v Adrian Cockcroft>It should never pause while you're watching something and it should just work, right?

00:23:34.846 --> 00:23:36.196
<v Adrian Cockcroft>So that was why it mattered.

00:23:36.201 --> 00:23:41.047
<v Adrian Cockcroft>And the brand value of that was basically the value that we sort of assigned to resilience.

00:23:41.797 --> 00:23:42.307
<v Jim Hirschauer>All right.

00:23:42.358 --> 00:23:45.096
<v Jim Hirschauer>Adrian, I believe you're gonna be speaking soon at Chaos Carnival.

00:23:45.135 --> 00:23:45.766
<v Jim Hirschauer>Is that accurate?

00:23:45.776 --> 00:23:46.736
<v Jim Hirschauer>Yep, yep.

00:23:46.765 --> 00:23:48.224
<v Jim Hirschauer>What's, what's your session gonna be about?

00:23:49.204 --> 00:23:57.076
<v Adrian Cockcroft>It's gonna be about this idea of continuous resilience and how that's kind of where we've got to in the whole chaos engineering world at this point.

00:23:57.195 --> 00:24:03.648
<v Adrian Cockcroft>I've talked to various chaos events before on sort of the history of how we got here and, and a few different things.

00:24:03.653 --> 00:24:05.989
<v Adrian Cockcroft>So focusing a bit on where we go next now.

00:24:07.263 --> 00:24:09.153
<v Jim Hirschauer>Okay, and I just pulled up the website.

00:24:09.153 --> 00:24:11.309
<v Jim Hirschauer>It looks like that's March 15th through 16th.

00:24:11.309 --> 00:24:12.420
<v Jim Hirschauer>Is that Chaos Carnival?

00:24:12.420 --> 00:24:13.859
<v Jim Hirschauer>So, yep, definitely.

00:24:13.859 --> 00:24:17.724
<v Jim Hirschauer>If you're listening in, go ahead and check Adrian out at that conference.

00:24:18.714 --> 00:24:24.684
<v Jim Hirschauer>All right, so Adrian, we've gotten through the meaty subject and now I'd like to have one more little fun segment.

00:24:24.865 --> 00:24:28.684
<v Jim Hirschauer>You've been a great sport so far for this I'd love to hear.

00:24:28.757 --> 00:24:34.096
<v Jim Hirschauer>Do you have a favorite travel story or a best travel spot that you've been to somewhere around the world?

00:24:34.193 --> 00:24:44.663
<v Adrian Cockcroft>So one of the more memorable places we went, so I was, when I was with aws, I was doing keynotes at various AWS summits around the world, and one of the ones I did was Tel Aviv.

00:24:45.294 --> 00:24:46.390
<v Adrian Cockcroft>I think I did that twice.

00:24:46.423 --> 00:24:51.354
<v Adrian Cockcroft>And the second time I went in with, well, let's go a few days earlier and go and see Petra.

00:24:51.959 --> 00:24:56.278
<v Adrian Cockcroft>Which is in Jordan, but it's like, it takes like a day or two to get there and to get back.

00:24:56.278 --> 00:25:05.578
<v Adrian Cockcroft>So we flew in a few days earlier and my wife and I flew down from Tel Aviv to Isla on the Red Sea and then got a, like a mini bus.

00:25:05.578 --> 00:25:12.479
<v Adrian Cockcroft>And it's a big hassle getting into Jordan and the border and, you know, it's, it's a pretty much developing nation kind of stuff.

00:25:12.479 --> 00:25:16.919
<v Adrian Cockcroft>So it's sort of, quite interesting exercise to just to get there.

00:25:17.398 --> 00:25:29.430
<v Adrian Cockcroft>And you get there and then you go down this canyon and then all of a sudden you see the, the, the treasury, which is everyone knows if they've watched you know, the Indiana Jones in the last crusade, right?

00:25:29.509 --> 00:25:33.019
<v Adrian Cockcroft>I guess they faked the interior of it, but the exterior of it, it's this amazing thing.

00:25:33.170 --> 00:25:39.086
<v Adrian Cockcroft>It's 2000 year old building cut into the side of, of the rock, and we went all through that whole valley.

00:25:39.086 --> 00:25:57.334
<v Adrian Cockcroft>Wife had a hurt her foot at some point, so, We got a got on a little buggy, so they had these horse drawn buggies and they're just charging off this pretty dangerous sort of this if you try walking down there were these horse drawn buggies sort of charging past you and people are camels going past you.

00:25:57.334 --> 00:26:03.273
<v Adrian Cockcroft>And we, we avoided the camels, we did the horse drawn buggy and it was kind of cool to go and see all these buildings.

00:26:03.273 --> 00:26:05.354
<v Adrian Cockcroft>It's an amazing place worth a visit.

00:26:06.124 --> 00:26:07.624
<v Adrian Cockcroft>Very much a, a different world.

00:26:07.624 --> 00:26:11.673
<v Adrian Cockcroft>And the, the story of the people that built it I is pretty interesting.

00:26:11.824 --> 00:26:15.634
<v Adrian Cockcroft>Literally 2000 years ago, and then it was lost for a long time.

00:26:15.702 --> 00:26:16.782
<v Adrian Cockcroft>Discover, rediscovered.

00:26:17.188 --> 00:26:18.988
<v Adrian Cockcroft>I guess a hundred years, couple hundred years ago.

00:26:19.768 --> 00:26:20.038
<v Jim Hirschauer>Wow.

00:26:20.038 --> 00:26:21.298
<v Jim Hirschauer>What an amazing trip to be on.

00:26:21.365 --> 00:26:22.765
<v Jim Hirschauer>I have seen a few pictures online.

00:26:22.765 --> 00:26:24.355
<v Jim Hirschauer>It looks absolutely incredible.

00:26:24.355 --> 00:26:29.365
<v Jim Hirschauer>So it's definitely on my list now to go and check that out if, apparently, if I can get there.

00:26:29.365 --> 00:26:30.954
<v Jim Hirschauer>So, should be interested.

00:26:31.315 --> 00:26:35.346
<v Adrian Cockcroft>It's yeah, we did a, like an organized tour of things starting in Tel Aviv, like.

00:26:36.041 --> 00:26:38.112
<v Adrian Cockcroft>They managed us getting there and back.

00:26:38.201 --> 00:26:38.922
<v Adrian Cockcroft>Ah, good to know.

00:26:38.922 --> 00:26:42.402
<v Adrian Cockcroft>Yeah, it's not the, it's not the best hotel experience or food experience.

00:26:42.402 --> 00:26:49.263
<v Adrian Cockcroft>It's all about the, well, the food's fine, but the, the the overall it was, it was really about the, the ancient ruins.

00:26:49.263 --> 00:26:49.864
<v Adrian Cockcroft>It was very cool.

00:26:50.013 --> 00:26:50.463
<v Jim Hirschauer>Amazing.

00:26:51.003 --> 00:26:54.243
<v Jim Hirschauer>Listen, thank you so much Adrian, for joining us on the show today.

00:26:54.243 --> 00:26:56.493
<v Jim Hirschauer>We really appreciate you taking your time out of your day.

00:26:56.561 --> 00:27:01.902
<v Jim Hirschauer>And to all of our listeners, if you are an SRE or if you're in a related role and you'd like to be a.

00:27:02.376 --> 00:27:03.457
<v Jim Hirschauer>On Ship Talk.

00:27:03.757 --> 00:27:09.096
<v Jim Hirschauer>Please go ahead and send us an email at podcast@shiptalk.io and we'll get back to you.

00:27:09.696 --> 00:27:10.267
<v Jim Hirschauer>That's all for now.

00:27:10.507 --> 00:27:11.767
<v Jim Hirschauer>Until next time, thanks.

00:27:12.037 --> 00:27:12.606
<v Jim Hirschauer>Thanks, Adrian.