Aleks Rudzitis (00:00):
It is painful, I think, to start. Engineers who are not used to thinking about these things will be frustrated that their work is being interrupted. I joined a startup. I'm excited about the product. I want to spend all my time heads down on the product. I don't want to be told to stop working on something to fix this boring thing where only 20 errors are thrown because why does that matter? You need engineering leaders who I think understand their team's best and the personalities on their team to guide you in this process. And I don't think there's a one size fits all solution.

Ganesh Datta (00:36):
You're listening to Braintrust by Cortex, where we explore how engineering leaders blend AI, platforms, and culture to build high-performing software teams. I'm your host, Ganesh Datta, CTO and co-founder of Cortex, an engineering operations platform designed to help organizations continuously improve their operational maturity and reduce developer friction. In each episode, we go deep with CTOs, VPs of engineering, and technical leaders who've been in the trenches, navigating the tension between speed and quality, building reliability at scale, and figuring out how to lead through major platform shifts. Whether you're running a team of 10 or 1,000, this is your space to learn from people who've made the hard calls and live to talk about it. Today, I have with me Aleks, who's a principal engineer at AWS. Aleks, great to have you on if you wouldn't mind introducing yourself.

Aleks Rudzitis (01:33):
Hi, great to be here and thanks for having me. So my name is Aleks Rudzitis. I'm a principal engineer at AWS. I work primarily in the cryptography space. Prior to AWS, I did a stint at Stripe as a staff engineer there, also working on infrastructure related to protecting our credit card data. And before that, I was at AWS again, still working on cryptography.

Ganesh Datta (01:55):
It's great to have you on. I'm very excited for this episode because one of the things that's on top of everyone's mind right now is how AI is changing software development. And we are seeing a lot of data, both our own, the recent DORA report that along with the increase in delivery, there is an increase in incidents. The quality is degrading across the board. And one of the things that we've been spending a lot of time thinking about is where do humans live in the loop today and how do we improve quality and our ability to deliver software as an organization in today's world? And while I was doing some research on this, I came across a series of posts by you around operational excellence reviews and ops reviews both at Amazon and Stripe. Would love to talk about that. So maybe if we can kick it off with just a high level overview of what an ops review is, why they exist and your early experience with them.

Aleks Rudzitis (02:50):
Yeah. So I spent most of my career in AWS where there's a very strong culture of every week at the AWS level, you get everyone together in a room and you look at effectively things that have gone wrong. You look at your critical metrics, you look at how things are performing. You take the opportunity to deep dive. Usually generally at the company level, you can't investigate everything. You kind of do a sample to see how services are behaving against some definition of correctness. And within AWS at least, we see kind of this structure, this meeting format replicated down the organizational tree. So you'll see it in, for example, we do the same in AWS cryptography. We have a weekly meeting and that one becomes a little bit smaller, we do have a chance to kind of look at everything. But again, I see it the ops review is basically it is two important things.

(03:53):
It is a way for the company to communicate to its employees the importance of delivering reliable software. Secondly, it is a place that serves as sort of a backstop to things going wrong. It is the place where you will look at the graphs, you will look at the signals, you will look at incident reports depending on however you may do that in your company and you will try to extract learnings from them. So it's kind of the place I see where this is the buck stops.

(04:27):
It is easy to ship out software that breaks, quietly fix it and maybe no one notices besides that team, but the goal is that you want to have some process where you all get together and kind of agree on what the definition of correct is and really figure out if we are meeting that standard we have set for ourselves.

Ganesh Datta (04:49):
You described the ops review as a backstop, but it sounds like it's just a meeting. So how does the ops review function as a backstop? What gives it the authority to maybe change things in the organization that way?

Aleks Rudzitis (05:03):
So I think that's a really important point to touch on is that as individual contributors, it's a very difficult process to implement from the bottom up. If the company does not have a culture where the leaders care about the correct behavior of their systems and usually that means things like, when I say that, I mean general availability, bugs in production, other performance characteristics like latency. If the top level leadership does not care about that, I do not have a good answer for how you force that from the bottoms up. Now you can always do it as a team, but the important thing is that in these meetings that you do generally have some person either that has the authority directly or has been delegated the authority to say, one, this performance that we saw is not unacceptable and it is now a priority to fix this.

(06:04):
And this might mean an example of things you might do is you might see that the number of errors we have returned to customers over the last, the number of API calls that have failed over the last month is unacceptable. We are therefore pausing new feature development, new changes until we understand what is going on and have remediated that and feel confident moving forward. So I would say the meeting is not something that has authority on its own, but I would say it is a mechanism for that authority to be exercised by people who do have that authority.

Ganesh Datta (06:41):
So it's kind of like a short circuit mechanism for things that like otherwise would propagate as like, oh, we have tech debt and the

Aleks Rudzitis (06:48):
Yes

Ganesh Datta (06:49):
Tech debt is causing X, Y, and Z, and it's a product prioritization roadmap and the usual things like that.

Aleks Rudzitis (06:53):
And I think the value is in doing it in a meeting format. So you could certainly imagine other mechanisms. For example, a CEO sends an email to his reports who send an email to their reports, that kind of thing. But I love almost the theater of doing it publicly because I don't know if this is what you meant by short circuit, but it's a way to send that signal directly to the individual contributors because I do think even though individual contributors do not necessarily have the authority to change the priorities when it comes to product deliverables, they do have the ability to think about what they're doing in their day-to-day work and the risks they are taking and how the trade-offs they're making are valued by the company. I think it's important for engineers when they're writing software to understand that reliability is important for my company, that it does matter to someone if...

(07:51):
I'll take a very simple example. If a common mistake I'll see in kind of naive software stats is that you'll experience a spike of 500 errors while the service is deploying because you have this kind of old version stop, new version, start problem, database flips over something like that. And I've seen again in newer services where that have not established this culture, people say, "Yes, we fail 5% of our APIs during the deployment window, but that's fine." And then I think for those folks to understand that, no, we actually do care about that. Yes, that does count against us. Yes, our customers do notice this influences the decisions they make and their priorities when they're developing software or thinking about the infrastructure choices they're making.

Ganesh Datta (08:32):
That makes sense. So kind of going back to the earlier point, it's not that the ops review itself as an authority, but it sounds more like a cultural thing in a lot of ways. We care about talking about these things, we care about the outcomes here and we care about in some ways, even interrogating the reality of our software stack across these different dimensions. Is it possible to build this... If you're an organization that maybe does not have this culture of reliability from the ground up, is like a chicken or the egg thing,

Aleks Rudzitis (09:02):
Yeah

Ganesh Datta (09:02):
Can the introduction of the ops reviews create that culture or do you need that existing care to exist first before people will care enough to run and attend these ops reviews, if that makes sense.

Aleks Rudzitis (09:14):
I do fundamentally think that the decision has to be made by someone who has the authority to influence the technical roadmap. If it's not the CEO, maybe it is the CTO or director of engineering, whatever the title is. But then the question then is for them is now if I am in that role and I notice maybe I've joined a company that is still early but has not figured these things out and I see this gap, is it possible for me to change the course of the ship so that it's become part of our culture? And I think the answer is as long as the right people are bought into it at the top, the answer is yes, but it is painful I think to start because engineers who are not used to thinking about these things will be frustrated that their work is being interrupted, especially if it's common that I joined a startup, I'm excited about the product, I want to spend all my time heads down on the product.

(10:11):
I don't want to be told to stop working on something because engine X throws, sorry, not to pick on a particular open source technology, but you know what I mean? I don't want to be interrupted to fix this boring thing where only 20 errors are thrown because why does that matter? So you kind of have to break that normalization of deviance from what we've kind of defined as, or what we want to consider as acceptable and that's going to take a while and you need engineering leaders who I think understand how their team's best and the personalities on their team to guide you in this process. And I don't think there's a one size fits all solution that I could give you except let's start small, let's make the problem visible, let's make it... I mean, one of the most basic things is if you don't have the measurements, if you don't have the graphs, you can't see the problem, then nothing is possible.

(11:08):
So maybe you just start with a process where you make this visible and then you kind of start ratcheting it down and say, "If the performance drops below this threshold or the error spike above this threshold, then we're going to want to stop and look at it. " I think one of the ways that is most interesting to me that I read about in the Google SRE, the site reliability engineering book, though I haven't had a chance to implement quite literally myself, is this concept of an error budget. And I think that makes the situation very legible to, I think, a wide variety of people that it communicate and for those who may not be familiar, the idea is that over some time period you say, "We will allow this many failures of requests and just like a budget, I'm allowed to spend this many dollars in a time period." And now if you start getting close to the end of your budget, you now have to change how you're doing things so that you don't burn through that budget in your time period, which generally means shifting away from new features which may break things and focusing again on reliability, which hopefully fixes things.

(12:24):
I think that's really clear because it communicates that the standard is not perfection. And I think that is the biggest risk of when you're trying to introduce this culture into an organization that is already used to shipping in a different way is that they'll take away that, well, now the standard is perfection and perfection's not possible and everyone just gets frustrated. I think you have to communicate a reasonable bar and that'll depend on what field you work on. In Stripe, it was a lot easier to say to have a really high bar because every failure is potentially someone who was not able to make a purchase. The consequences are a lot easier to grock. But if you are working on a social media website and someone has to just click refresh again to see their feed, maybe you don't want people to have the perception that your app is unreliable, but the consequences are a little bit less dire.

Ganesh Datta (13:18):
Yeah, that makes sense. The error budget thing is interesting because you mentioned earlier that in some way just the theater and having the meeting itself is valuable. And I was talking to somebody recently who was asking, "Hey, why can't we just do this async? Why not just create a report and we can look at the report and have this conversation?" Or to the point of error budgets, why not just force everyone to set up SLOs and you have error budgets and that is the equivalent of you're healthy or you're not healthy. What is the purpose of the meeting versus systems like SLOs and error budgets?

Aleks Rudzitis (13:53):
Yeah, that's a really great question and it's a piece of criticism I've heard several times while trying to implement these processes. One is can we just do this async? Second is this also happens to maybe you start out doing an ops review, but you get so good at your job that you don't have any errors to talk about because everything just works and you've become so good at your jobs and people are like, "Well, why are we coming to this meeting every week?" So in the spirit of this being theater, I think there's two questions here. The first one is, why not just do an async? And this is part of the human driven infrastructure piece that I find really interesting because in theory that sounds good, but observationally I have found that the async process turns into an email which no one ever reads and then the culture slowly atrophies and whether this atrophies in a world where everything is good or whether atrophies in a world where everything is bad, I think both are problematic and for better or for worse, getting everyone in a room together, even if they're on their laptops half the time looking at Slack, it is useful anecdotally again, I don't know how a good way to quantify this.

(15:23):
I find people take away the message better when they are in that room where their leader is asking questions about how the service is doing because there's something about that that is harder for the human psyche to ignore than another email update and I don't really have a better answer than that.

(15:45):
Now one way you can avoid this temptation to make it async is to bring content that can't just be async. A meeting where you are just looking at graphs where yep, everything is okay does become very dull. And I sympathize with the engineers who do not want to sit that. So one thing we've been experimenting with recently is can we kind of combine this ops review meeting with a learning session? So one thing we do is because we generally do not have many issues to talk about, a good problem to have, is that we use the time to talk about things engineers care about. Is it a form where... Because sometimes what happens is the metrics look good, but the engineers are going through extreme heroics to make it happen. People are working super late nights, overwhelmed with tickets are getting paged on the weekend just to keep the ship afloat.

(16:39):
So I think giving them a forum, a way to kind of talk about their experiences being on call is one way that can make the content a bit more dynamic and also make it something that engineers are interested in participating in. Because it's not just a voice for the leadership, it can be a voice for that engineer to their leaders as well to say, "Hey, yes, I know this is great, but look, I was paged 10 times on Saturday to put out the smoke before it turned into a fire." And that's where these good metrics are coming from. We've also used it as an opportunity just to socialize learnings like tell us about how you are testing, tell us about how you are doing your local testing on your laptop before you push it into the beta environment, making your software development experience that much smoother.

(17:31):
And that's just a way, I think, to kind of still stay within the spirit of operational excellence and reliability while making the content a bit more valuable to engineers to kind of sip through.

Ganesh Datta (17:45):
That makes sense. I was talking to somebody on the podcast recently, Nathan Harvey, who runs DORA at Google and he was mentioning that one of the recommendations they have for folks for when to review DORA metrics is an existing forum. So if you already meet once every two weeks for your sprint retros, just tack that on as a way to start. And eventually if it's interesting enough, it'll become its own thing. And so it sounds like maybe some variant of that, I think what you're describing is like the flip of start the ops review and then tack this on at the end, but you

Aleks Rudzitis (18:14):
Yeah

Ganesh Datta (18:15):
You could potentially do it the other way around of starting to create that practice. But I think what you're talking about is also, people talk about that idea behind motivation versus discipline, right? It's like you may not always be motivated, but you need to have the discipline. Even if you don't work out, just go to the gym, sit in your car and it's fine. And so it's similar to that is what it sounds like. I

Aleks Rudzitis (18:36):
Was about to say, make almost the exact same analogy. So I found the same thing with exercise. So I like running. I can't run every day. Someday I have a cold, but maybe I can go for a walk that day and making sure that that 30 minutes is still carved out and maybe I don't even feel well enough to leave the house, but I could do some sit-ups, just something so that time is reserved for me and my mind is there at least for a few minutes every day, even if it's not everything I wanted to be, is useful for maintaining that discipline.

Ganesh Datta (19:07):
And I find, and this is, like you said, it's very anecdotal, but I find that if you do get into that forum, something tends to come up. There's somebody with an interesting topic that they want to share and it turns into a whole conversation and I think it's reserving that space for it. You mentioned a phrase while you were talking about the meeting, human-driven infrastructure.What does that mean? That's an interesting phrase, especially now when everything is agents, agents, agents.

Aleks Rudzitis (19:34):
Yes.

Ganesh Datta (19:34):
What does human-driven infrastructure mean?

Aleks Rudzitis (19:36):
So a little bit more about my background. So I graduated with both a degree in computer science but also in informatics, which you could sort of think of analogous to information science. And in that degree, there's a real focus of how humans interact with technology, not just the technology itself, but what does it mean to be a human in a technological world? And this has kind of stuck with me throughout my career. And I find this fascinating because when I come to work, I don't think of myself as a person who writes code to deliver a feature. I think of myself as a person who works with other people to ultimately deliver a product that's hopefully valuable for even more people and technology is just a means. So human driven infrastructure are all these mechanisms that we build for people to ultimately be successful in this task and the tools, the technology, the compilers, the AI harnesses are ultimately just one aspect of that.

(20:49):
But I believe even with AI, software will remain a team sport. And I think the mechanisms that we use to accomplish these tasks as a group of people are interesting, even if our precise relationship with that underlying technology changes, because ultimately this is for people. At least as far as technology stands today, we're still building this for people and as long as that remains true, we'll remain an essential part of the loop.

Ganesh Datta (21:17):
Yeah. And even if you're building for agents, it's for agents that are for people in some shape or form and eventually it stops at a person somewhere, but I think that aligns well with the idea that our ops reviews are cultural and culture is a part of working with other people on teams and aligning on shared values and things like that and reliability being one of those shared values.

(21:42):
Do you see that idea of human driven infrastructure changing? Has that evolved in the past year where primary consumer of some of these things were humans in the past and some of those things are... I think about, for example, like platform engineering principles like Golden Paths and things like that, where a lot of what we did was to make things standard and easy for people to do things in a repeated way. Maybe some of those things are changing, maybe they're not. But how do you think this idea of human-driven infrastructure is changing, if at all?

Aleks Rudzitis (22:15):
Three things I'll say. The first one is I think it has only become more critical. The faster you're shipping software, the more critical the backstop becomes to understanding where your software may be misbehaving or you might be kind of losing sight of the more important targets. The second piece that has been great is like the AI tools have given us the opportunity to digest a lot more data. We see this in operational reviews where before we would have to sort of... One thing we've sometimes introduced in these forums is the opportunity for the person who was on call the previous week to talk about their most interesting tickets they worked on or their most impactful tickets. And that doesn't scale very well because you're limited to kind of how many tickets that person can write about or report on or summarize and AI gives us the ability to look at a lot more tickets and synthesize a lot more data, especially when that data is, I don't know, for lack of a better term, more squishy.

(23:29):
I think before we could look at metric trends, but now we can kind of look at more sentiment trends in tickets. There's just more ways to analyze that data and summarize it and present it succinctly in a meeting. And I think the final trend I've noticed is that I think one thing that's fascinating to me is suddenly APIs and text interfaces because they work so well for LLMs suddenly became a bigger focus for everybody when before we're doing a lot of things on websites, even a lot of internal tooling might have been websites, but suddenly there's a huge demand for having an API thing or a command line interface for something or just some text driven mode. And that's great for LLMs, but it's also great for people too. I think that the difference is that we're almost back to the old Unix philosophy where we have all these tools we can tie together.

(24:25):
And yes, maybe much of the time it's an LLM doing it, but I can also do it as human as well.

(24:31):
So there's actually almost more pressure to automate our infrastructure in ways that was before required what something called the click ops model of you go to a website and click on things. So I think that's really interesting and I think that just because those tools which building an API is just a much bigger opportunity now because it's not only going to be a few people like me who live in the terminal who are going to use it, but now everyone's LLM is going to use it too. So those are some of those big changes I've seen.

Ganesh Datta (25:07):
Yeah. On the point of exposing some of these tools that are consumable by both LLMs and humans, the other thing that I think about sometimes is, like you said earlier, the human isn't out of the loop, we're just in different parts of the loop and human driven infrastructure in some ways could also be now less about... The task has changed, like giving people visibility into what their agents are doing or giving them the right abilities so that they can wire their agents off in a specific way. And so I think the level of abstraction is just changing, but the concept of human-driven infrastructure I don't think is going away for that reason.

Aleks Rudzitis (25:44):
Yeah. And I think that another space is a testing where I think before people could justify to themselves in many cases that I will do some manual testing on this in a pre-production stage to verify that it works. With the rate of change we see in AI that no longer becomes scalable, but writing tests has also become cheaper, building those test harnesses that automatically test features for you has become cheaper. Even qualitative testing where you would have maybe opened something in a browser and AI can open that and do a certain level amount of value judgment about how something is rendering and whether that looks okay. So this is a great time to want to have a high operational bar because the same tool, the same LLMs, which build all these new customer facing features can also be used to build testing infrastructure. And that's just amazing because it doesn't require much extra expenditure to build super strong guardrails.

Ganesh Datta (26:46):
That makes sense. And on that note, with the way LLMs are changing both their ability to invest in the operational bar and new classes of types of issues, is the idea of operational excellence or the things that fall into operational excellence changing? Is the scope of ops reviews different than they used to be because they're in new failure modes or is the current shape and format of those ops reviews enough to kind of capture the downstream impact?

Aleks Rudzitis (27:16):
I think the primary metrics that we've looked at, at least availability, latency, that doesn't change. Those are kind of the bedrock of whatever your foundation is of correct behavior for your system, that remains ultimately the same. I think where it potentially changes might be secondary metrics. So metrics that you might look into when you're trying to understand how we got to those availability metrics, certainly we... And the problem here is making about a general statement is this probably, this varies a lot depending on what you're looking at. I have looked at worked in organizations where that one of the secondary metrics might be number of human interventions you have to do. Interestingly, that has become a more important metric I think at the age of AI is because people are looking at it and understanding, is there now an opportunity where I could remove a human intervention to make things better where that simply was not possible before.

(28:21):
But yeah, I think it's actually interesting that it hasn't changed much because I think ultimately the definition of correctness hasn't changed. It's more about your opportunity to do things about it has changed.

Ganesh Datta (28:33):
Yeah. It's interesting. I mean, Dora says this in their report as well, it's like, what's old is new, what's news old. Those aspects of shipping a product to customers have not changed. And especially if you assume that there's a human somewhere in the chain, correctness is the same. Your product does what it says it does, and we can define that in certain ways and we want to guarantee those things. Maybe how we got to that end state is potentially different. And LLMs wrote that code instead of a human, but I think that's absolutely right from my perspective. It ties into another series of articles you wrote a while ago regarding preventing incidents and kind of standardization of practices and principles and their application for reducing incidents. Has that changed now? You mentioned one of the things that you cover in ops reviews is learnings from incidents past and Amazon's correction of errors concept and whatnot.

(29:27):
Are the practices that you recommend now for preventing incidents changing because of agents or investing in different types of verification systems or what does that look like today?

Aleks Rudzitis (29:37):
I think the biggest thing that changes is that testing has become a lot cheaper. The ability to go and either backfill test integration tests where they didn't previously exist is a lot less labor intensive than it was. So I think one of the challenges about being in a team where previously where you wanted to ratchet up the bar in terms of quality, what you'd often find is you'd start with a place where there's no testing in this code base or very little testing. And it's always hard to be the first person on a team to decide I want to do this well because the coverage is already so poor. But now with LLMs, the cost to backfill that just to get to a good starting point is a lot cheaper even if it's not perfect. An interesting development has been in the field of formal verification of software.

(30:34):
This won't be of interest to a lot of people. This is kind of a level of rigor I think that I expect most folks are not really interested in, but there's a whole field of formal verification where you use these tools to speaking a bit colloquially, almost in a mathematical sense, prove the correctness of software. This is obviously very critical in cryptography in certain fields like the flight computers on airplanes and things. But applying these writing proofs in these systems, annotating your code with it, verifying that your code checks out against a proof has always been incredibly labor intensive, which is why it's rarely ever applied. And what we're starting to see is that LLMs don't get tired. They don't get bored trying to implement these. And you can now apply some of these formal verification techniques to a lot more software that wasn't possible before.

(31:35):
And you can have these proofs that are very large that no human would have feasibly wanted to write before but are now feasible. Now, of course, you have a problem where the LLM writes the proof. How do you know the proof is good? And I think that that's not a problem that's completely solved, but I'm kind of excited for developments in this space. I think there's a lot more different types of testing that didn't scale with humans before that now may scale with AI.

Ganesh Datta (32:06):
I think it all goes back to even testing, for example, there's only so much time in the day and capacity for humans to do this kind of thing. And while it has gotten easier to do this type of work, it still implies that there is somebody who cares enough to steer their clot or whatever to go and do this kind of work. And again, kind of loops back to the cultural aspect of this is a thing that we care about because there's some sort of impact on the customer. So therefore, we're going to invest in this kind of verification. And so maybe this ties back into the secondary metrics piece that you mentioned where it kind of ties into why is our availability the way it is and it's because, okay, we don't have ways to verify this and that kind of leads to testing.

(32:49):
And I do feel like more organizations are heading down this path now because it is easier to make room for reliability earlier on in their life cycle in a way that maybe wasn't before.

Aleks Rudzitis (33:02):
And I suspect teams or companies are going to find that to move at the speed they want to with AI, these guardrails need to be in place because the unit tests are not testing in general is not only useful for preventing an outage in production or misbehavior in production. It is a useful feedback loop for AI to prevent it from simply hallucinating the way a system works. It's not perfect, nothing ever is, but the engineers I have worked with have certainly found that having robust testing in place allows them to move faster with AI and kind of reap the benefits of investments we have made into this bringing this LLM technology to developers.

Ganesh Datta (33:51):
Do you find that there's more of an emphasis on reliability now and maybe this is not applicable given that you work in organizations where reliability is one of

Aleks Rudzitis (34:02):
Yeah

Ganesh Datta (34:02):
the features in some way, but are you finding that there is more of an emphasis on maintaining reliability because we understand less of the inner workings of our code? Or where do you see reliability changing today, if at all?

Aleks Rudzitis (34:17):
I appreciate what you said. I believe that's correct that generally the places where I have worked, I've had the benefit of having leaders that already put a huge value on this. I think there is certainly anxiety that AI could make this worse or we are bringing new risks in, but I think there's also a lot of confidence that the mechanisms we have built are still ultimately the right ones to prevent catastrophe.

Ganesh Datta (34:46):
Do you feel like the ops review process is maybe helping ease and anxiety? Because if you have this cadence of already, hey, we're looking at our metrics, our metrics are healthy, because you're not breaking into unexpected ways. It's like, okay, well then what is there to have anxiety about if things are going in the right direction? So do you feel like that process has actually helped stave off some of the anxiety that maybe organizations are feeling?

Aleks Rudzitis (35:06):
Absolutely. And I think I didn't mention this benefit of this process before. We've talked about it as a way for leaders to kind of communicate their priorities team, but if I'm a leader, I'm in meetings all day. I only have so much time in my day to think about operational excellence. I do believe it is a channel for me even as an engineering leader to feel that the right things are happening, to sit there and have my engineers in my organization show to me that this is why we believe our systems are still healthy. Whether that is a purpose you want this meeting to serve will depend a lot on the tastes of the size of the organization, the preferences of your leadership and the alternative ways they have to assuage these concerns, but I think that is certainly a possible one. And I would recommend for anyone who is looking to develop similar mechanisms for their organization is to certainly speak with the leaders that will be attending these meetings to understand what their anxieties are because this is a way for them to have content presented to them to convince them that the right protections are in place.

Ganesh Datta (36:22):
You touched on exactly what was going to be my last question here, which is if folks are in a place where maybe reliability hasn't always been the focus, but there is this growing unease of anxiety and concern that maybe reliability is starting to degrade and they want to think about ops reviews as a way to improve the currency of the world, but also ease that anxiety, where should people start? Is it better to start with individual teams, do the org wide ones, start with leadership? What advice do you have for people that want to put this kind of process in place?

Aleks Rudzitis (36:56):
So it depends on where you sit in the organization where you're going to start. I think ultimately you need the blessing of some engineering leader within your organization that has the ability to exercise the authority to drive priorities to make improvements and also send a message that this process is indeed going to be one of your mechanisms now. So you need that buy-in because otherwise like the most motivating thing that can happen is you create this meeting and simply no one shows up to it. So you need the blessing from your leadership. I think if you can get that at the top, that's great. If you can only get that within your team, that's within your two pizza team with 10 people, that's better than nothing. And then you need to understand, I think both what are the concerns of your leadership, what are their anxieties, what does success look like to them?

(38:02):
Because this isn't always the same as it is for engineers.

(38:06):
And then I think you also need to ask engineers, what do they want leaders to know about their service? Because sometimes you will hear engineers, they know where the bodies are buried, they know which parts of their systems are fragile, they know where we're kind of close to an edge. And I think it's useful to understand what their anxieties are because ultimately if you're having an ops review, it is a form, it is a space for these two sides to connect. And I think the input you solicit, the more input you get from both sides about what is valuable to them, the more likely you are to create a meeting that everyone finds productive and that has the impact to change the trajectory, or at least maybe if you're already doing well, just maintain the trajectory you're on.

Ganesh Datta (38:57):
Kind of tying that back to the first part of the conversation, like you said, maybe I'll rephrase the concept of theater and the authority, you want to be able to have the people in the room that can reallocate time and resources towards fixing the things that this is surfacing. And the second thing that you're describing here is like you want to have the right discussion. It's not just about looking at numbers, but it's prompting conversations and it's a forum to tell that story from both ends. These are the things that I'm worried about, this is the things we're worried about. And digging into the discussion itself is valuable because it leads to the outcomes that you want in some ways.

Aleks Rudzitis (39:33):
You used a word earlier, which I thought was incredibly valuable and that was space. And when I try to develop these mechanisms or maybe change the mechanisms we already have, I think of it as a process of creating space I'm not there as a kind of a sponsor or champion of these meetings to fill it with content. I am there to create a space for conversations to happen because ideally you may start, you may be looking at a graph or a metric on there, but what's really interesting is the conversation that kicks off. When we look at that and go, "That doesn't look right. What can we do about it? " So I think it's very important to not lose the forest through the trees by creating a meeting where you're just scrolling through data. You are there to inspire, to have a conversation about it.

(40:30):
And I think when that happens, that's when that process is really successful.

Ganesh Datta (40:36):
Well, Aleks, thanks for joining me on the podcast. Of course, thanks for sharing this with the world in your blog as well. I think this is very timely for folks who are grappling with the effect of AI on their own software development life cycles and degradation of reliability and things like that. So it was a pleasure having you on. Thanks for joining me.

Aleks Rudzitis (40:54):
I appreciate it. Thanks for having me.

Ganesh Datta (41:02):
Thanks so much for listening to this episode of Braintrust. If this resonated with you, do me a favor, share it with another engineering leader who's wrestling with these same challenges. And if you want to continue the conversation or learn more about how we're thinking about engineering operations platforms at Cortex, reach out to us at cortex.io. Thanks for listening and we'll catch you on the next one.