Ganesh Datta (00:00):
What is the piece in the system today that you have the least confidence in?

Martin Nettling (00:04):
Human.

Ganesh Datta (00:05):
Okay.

Martin Nettling (00:06):
Humans are pretty lazy, right? And this is a good thing, but we tend to always overstep something. For example, at the end of the week, we want our engineering leaders to write a report about what they have achieved this week. It's an exercise for the team to reflect what they have done, to feel proud about their accomplishment. You know what happens? They're using AI to generate this report.

Ganesh Datta (00:32):
You are listening to Braintrust by Cortex, where we explore how engineering leaders blend AI, platforms and culture to build high performing software teams. I'm your host, Ganesh Datta, CTO and co-founder of Cortex, an engineering operations platform designed to help organizations continuously improve their operational maturity and reduce developer friction. In each episode, we go deep with CTOs, VPs of engineering, and technical leaders who've been in the trenches navigating the tension between speed and quality, building reliability at scale, and figuring out how to lead through major platform shifts. Whether you're running a team of 10 or a thousand, this is your space to learn from people who've made the hard calls and live to talk about it. Today with me on the podcast, I have Martin from Telium. Martin, great to have you on. If you don't mind introducing yourself.

Martin Nettling (01:29):
Yeah, thank you very much. Martin Nettling, as you said it. I'm a senior director of engineering at Telium, also kind of head of QA. So I took this over our head of QA left. My background is interestingly in bioinformatics. So I studied bioinformatics and molecular genetics and then went to the computer science industry, if you want to say it this way, and software development. Yeah, I think my top period was my last company, Datamere, when I was VP engineering. I started there as a normal engineer and then growing, worked with Apple employee number 10, Ken Grugler, a pretty interesting experience and he taught me a lot. And I was hired at Telium two years ago with my whole team. So I brought six or seven people with me at Telium, which is a very interesting experience because you now have your own culture, your own ecosystem at Tealium.

(02:22):
And Telium has its own ecosystem. It's a worldwide company. We have a team in Poland and the US, and now you have this at once a German team that has an opinion, which creates friction but also creates interesting interactions in the company.

Ganesh Datta (02:36):
I can only imagine, and I think it's probably relevant to the conversation we're going to have today about trust within the systems, but it's great to have you on. We'd love to hear a little bit more about what that transition was like. How do you make a team like that successful as they're embedding into a broader organization where there's already established culture? Was there a secret sauce? How did you adapt your team or did the broader organization have to adapt or was it a little bit of both?

Martin Nettling (03:02):
I think it was a little bit of both and it was not always easy from all sides. So at the end, I would call it a win-win-win situation. So my team I brought in and they are happy that I am the shield and translating stuff to them. And they trust me. And I mean, I think we'll speak later about trust, but they trust me. So it was not hard for them to come to the company. And the other way around, I always knew that I had the support from them if something is going off. And so Matt Hausler and Mike Anderson, those who hired me, they gave me the trust to bring in a team without interviewing them even. So we started there and they also won something because they had already a functional unit in the company that directly could jump on. So we were very cross-functional.

(03:49):
So DevOps, UI, backend, whatever you can think of, this team was able to do it. Yeah, but this also created frustration because of course we had our own ways of work. So we had our own way of thinking, developing and deploying software, and especially releasing software. And this was not in sync with at this time with in sync what Tealium did. And it took us some while to figure out that Tealium is coming a little bit our way and we are coming a little bit their way. So it's not there anymore, it's now in us. It was challenging and it have not been always quite discussions. Sometimes it was a little bit louder.

Ganesh Datta (04:36):
Well, you mentioned trust and that's one of the things we want to talk about today, which I think is relevant here. Obviously we've been having a lot of conversations about AI and coding assistance and how it's transforming software engineering as a practice. But specifically you've been talking about the human element of it, which I think is particularly interesting. I think there's a lot of talk about the systems and the automation and whatnot, but you being a prolific writer recently wrote about this idea of the difference between trust and confidence. So maybe if you want to just describe the premise of this, what you were trying to talk about and what the difference is between trust and confidence in your perspective.

Martin Nettling (05:17):
I would start with some very easy comparison. It's like when you order an Uber driver, he's coming with a car. So at the moment when you see the car, it's probably a very nice car or it's a very shitty car. This is confidence. You have confidence in the car or you don't have confidence in the car based how it looks like and if you enter it, if the doors make a strange noise. And then when the driver is starting and the first time he's already not hitting the right gas or the brake, then you lose the trust in the driver. So this is a difference. You have confidence in a tool, but you have trust in a human person.

Ganesh Datta (05:56):
Can you have trust in the tool as well? I mean, you think about the reliability of a Toyota. Is that trust in the brand? Where does that -

Martin Nettling (06:05):
Yeah, trust. I would say trust in a brand is also something, but you still have confidence in the car. But the thing is, if you have people, so if you have a construction worker at your home and you see, oh, he has good tools and you see how he's using the tools, of course, confidence in these tools can be an amplifier of the trust and the other way around. It can also be that if you have the awesome Uber driver that it's an amplifier of the confidence in the car. I mean, this was only the comparison to make it easier to understand for people that are not in software development, but in software development, we have the same situation. And to have this story, it was like a month ago or two months ago, or could also be longer ago. But I had this conversation with one of the engineers and I reviewed a pull request and he did it with AI, so full thing.

(06:57):
So it was AI generated, then there was an AI code review and some other stuff. And it was not the first time and he's not the only one. We are seeing it everywhere. And it's also not a bad thing. So I definitely don't want to be an AR here. So I say use AI, but use it responsibly. And he came over and I was really bad with him. So I was not yelling at him, but I was like, "Man, this does not work." And I was not able to separate my feeling for him from the tool that he used. And this was the moment where I had the idea, "Man, you have to communicate it differently. He tried to use AI and I produced something that he thinks is correct, but you are not bad about that he's using AI.

(07:52):
You have some missing confidence in the tool that he used and how he use it. And now you have translated this into mistrust that he can't use the tool correctly. And after separating this message, and probably later I can explain a little bit better, but later after I have separated it, it was also easier for me to communicate this. I was like, " Hey, what have you used? Show me how you used it. Can we improve this usage? And if I see that this guy is now trying to improve it, updating the prompts, changing the H&M Files or whatever, then my trust increases. And in the same way, also my confidence increases because I now see that he can use the tool correctly.

Ganesh Datta (08:32):
How do you disconnect those two? Because it seems like those two things go hand in hand. There's somebody's ability to use user tools over time builds your trust in that person as well. Going back to that construction analogy or the Uber analogy, those two things are in some ways not mutually exclusive. Or am I thinking about that the wrong way?

Martin Nettling (08:51):
No, I think you're right. They are definitely not mutually exclusive. I mean, it's like with the example of the construction broker, you have a human that's using the tool. If it's acting fully autonomously, then it's only confidence, then it's not trust. So I have this automore in my backyard mowing the lawn. He's fully automated and there is no human involved anymore. So I cannot say that I trust the engineers from this company. There is no feeling for them. It's only this thing. I only have confidence in this automatic piece of technology that is working in my backyard. No, it's strongly connected and especially in software development. And I think one of the, I would not call it mistakes, but I lack a better word right now, what we are making is we only want to go faster. And I think it more from, hey, we now can deliver a higher quality in the same time.

(09:46):
So in one of my stories on Medium, I call it the augmented author model. You use AI and it could not only be AI. It could also be just a search engine or it could be a consultant or whatever, but you take this interaction with this tool and you are now responsible for how it's transported and how it's communicated. And I think there is a gap at the moment. So often people just do something with AI and it's not only that they're writing code, they're also writing Jira tickets, they're also writing Confluence pages. They're creating a bunch of artifacts and just hand it over without anything, without any comment, without, "Hey, this is the part where I used AI because I struggled formulating this. Or I'm not this firm in the English language, so I used AI for this reason, but this is a German way I wanted to formulate it or the Polish way or whatever.

(10:44):
This is missing, I think.

Ganesh Datta (10:45):
How practical do you think it is to create that culture more broadly? It seems like maybe that's runaway from us where people are, it's part of everyone's daily life.ou can never escape these AI generated Confluence pages and it's all over the place. And I totally agree with you. I think there's the moment you smell AI for certain types of work, and I think this is where the confidence things comes in, we naturally are building our own confidence with our own work in certain types of things. Okay, if you're using AI for this thing, I've done that before too, so I have confidence in the output of the tool. But when you smell certain things where you know it's not great at, you immediately lose confidence in the artifact and then you start to degrade trust with the person that produced it because you're upset that they sent you this thing that you know is of lower quality.

(11:35):
But do you think it's practical to unwind this runaway train that we've let loose now? How can we do this? Maybe focusing on co-generation specifically. You talk about the review process of that one PR. Is it possible to go back to a world where we can build that trust more broadly within an organization?

Martin Nettling (11:55):
I would say yes. So I would like to go a step back. I think the problem why we... I can't say I, because there are a lot of people that don't mistrust it. And I don't trust those people that at least don't challenge AI. So I think that one of the problems is that in the early days it was really bad. So AI had only 50% of correct answers and there have been a lot of errors and still there a lot of people were using it and then telling us wrong stuff. So our initial confidence and trust decreased heavily. Now those models are better. And if you think back and try to imagine, hey, how would it be if we would have directly started with those models? How would trust and confidence look like if we directly started with OPOS 47 or what the current version is for eight?

(12:50):
Because it's much better now. I mean, it's super expensive, but it's much better. And I think the world would, not the world, but at least in our company, it would look differently. So there would be trust and confidence. And I think if we are starting now to the pull request or Jira review or whatever you want to review, if we start separating the perspective of a review or of consuming an artifact, it's not only reviewing an artifact. Reviewing has always this touch of you have to give feedback.

(13:25):
But you are consuming an artifact. And if we are separating it like, okay, we are ignoring white spaces, for example. This is something that can be fully automated or linting or we are ignoring testing because we started to have confidence that an AI model can write the right tests. And we are then looking only at the things that matter for us as humans long term, which is or could be the main question could be, do you want to maintain this code in five years? Do you want to touch it again or not? So if we start separating those levels and I described it as a review pyramid, I think then we can start to get confidence in those tools. Because at the moment, this is a general problem of the world. We are always throwing everything into one bucket and trust it or don't trust it, have confidence or don't have confidence.

(14:21):
It's the same lately I had a discussion sorry to distract you a little bit about, "Hey Martin, AI is now taking over your job. So it's writing code and it's working." I said, "Nah, this is not how you can say it. I mean, there is definitely things that AI can do and can write code. It can probably write an app that is helping you to track your weight or your calories or whatever. This is definitely possible. And it's also not problematic because in a year you delete this app because you're writing the next app or you're not using it anymore. There you can use it and I would also use it. But if you want to write a code that is used by an insurance company or by a bank and you know this code has to live there for 10 years or 20 years or longer, like Kobe, if they have known, they probably have invested a little bit more.

(15:08):
Then you have different demands. This is changing. And this is the same with the review. If you're reviewing a certain code, then okay, there is code that doesn't have to be tested. It doesn't have to lift for 10 years. And there's code that has to live for 10 years. It's on a critical path. It has to be high performance. And there you're looking if you're using the correct hash map, if it's concurrent, if it whatever. So there's also the problem that we are throwing all the stuff in one bucket. Yeah,

Ganesh Datta (15:39):
That makes sense. I mean you mentioned linting as an example, and that's the most obvious one where even 10 years ago we were still having the tabs versus spaces conversation and do you put brackets on a new line and all that fun stuff? And it feels like those conversations are practically dead. Even before AI, they started going in the way of the dodo just because of the prominence of linters. Every team is using a linter of some kind and it's just like, don't argue with the linter. It's going to format things, don't worry about the formatting. And we just ignore that. Then the hot take question becomes why is code any different? Are we heading down a world where code is no longer the artifact that we need to be reviewing? Is it something else that builds a trust? To your point, it's like the human's intent, what were we trying to do?

(16:30):
And so is there a different artifact that we're going to be reviewing instead? Or is the code review still going to be important from a trust and confidence building exercise? How is that going to adapt from what you're seeing? This

Martin Nettling (16:42):
Is a very great question and I have a lot of discussions around this topic. And if I'm looking at how I'm looking at the different teams also at Tealium, I think it's not only the trust in this one artifact, this pull request or this piece of code that is created. It's artifact in how we got there to create this. And I like to call this, I read it somewhere so it's not from me. This unbroken why. And if it's broken somewhere and I come to what I mean with unbroken why, if it's broken somewhere, then I stop mistrusting the outcomes of it. And what I mean is, so imagine you're CEO, you have a strategy for your company and you are explaining your strategy to your directors and VPs. So I don't know the structure at Cortex. You're explaining the strategy and they are trying to translate it into tactical units.

(17:38):
So they split it. You have a marketing piece then and you have an engineering piece and a program management, whatever. And now the managers take the stuff and even slice it further so that the engineers can take over the work. And the engineers, they are producing now the code. And this was a short path of the value chain. And they're writing code and now they are delivering the code to production and the user is using it and creating a log file and entry in a log file or a metric. So the unproven why is that I can go back the full way from this log entry and this file to your strategy. So if this is possible, if I can do this by, oh, now I'm going back to the JIRA ticket. Oh, the JIRA ticket is linked to this strategy document, then this is creating trust for me and it's also going in the other direction.

(18:28):
Does it make sense? Thats a little bit crazy. If we have this, so let's assume we have this unproven why and we have proven that we can work with this unproven why. I can imagine that we stop looking at certain things in this chain. I can really imagine that, oh, if I know that you are writing 100% good JIRA tickets, they are always understandable. They are always well-formed. They always have the right acceptance criteria and they're always small enough. Yeah, then I stop reviewing it. It's like the same with my automow in the backyard. As soon as this thing is not stopping anymore, I will not watch it anymore. But at the beginning, the first two weeks, I don't know how you are, but if I have a new grownup toy, then I am, is it really working? Is it really working? And it takes two weeks, three weeks until I trust it.

Ganesh Datta (19:14):
Yeah. No, it makes sense. There was a previous episode actually on the podcast with Steve Evans, and we were talking about how to debug this chain within an organizational context. And his idea was you do these spot checks where you mix and match people from different teams and then you put them on a round table and you ask them questions to see if there's understanding of the context. And if there isn't, then you can figure out which chain to break in. And you can continue doing that up the chain until you figure out where the context fall down. And this was more from an organizational perspective versus a systems perspective. But to your point, it's the same thing in the path to code as the organizational context and it breaks down somewhere and you could figure out where in that chain it breaks down. I think the thing that I struggle with a little bit is now to your point, that entire chain people are using AI for.

(20:06):
And so it's like aside from the trust and confidence level itself, there's this element, and this is kind of a side tangent, there's this element of you did not put in the effort and you're making me putting the effort to review things. And this is where I think the frustration with AI-generated writing comes from is like, just give me a prompt. If you are giving me something AI generated to read, just give me the prompt that you wrote and I'll read that instead because clearly that has the context that you want me to have. Don't make me read your AI slop. And so I think that applies even in this chain where if people are using AI to write the Jira tickets and the text spec and all the entire chain of wise, where in this chain does the human governance and the human in the loop come in and how do we make sure that that doesn't become yet another artifact as like, okay, well now this is AI slop and so we're going to go up the chain and eventually there's nowhere further up the chain to go.

(21:03):
How do we prevent this from happening?

Martin Nettling (21:06):
This is culture. I mean we had this problem even before AI. I remember being in, I don't know how many Slack journals where people just posted articles. Hey, read this. Martin, read this without anything. I don't know how many select journals you are aware of this happening. And after the 10th time I said, stop posting this stuff without adding a message what you liked about this thing. What was the interesting piece? Why are you posting this? So what is your intention sending it to me? Should I understand something? Am I doing something that is wrong in your eyes or where can I improve? Why should I read this? Not only because it's interesting. Sure, everything is interesting in some way. So I would say if an easy message, and then there is something that is called reasoning bias, human are more satisfied if they always get a reason.

(21:56):
And you can even say it's so because it is so. Even then humans tend to be more satisfied than you're just giving them something. So I think if we only start adding one sentence to this AI litter or AI slop, by the way, also two interesting words, then it would already improve. And if people start using it this way, they're probably also expecting it now from others. So I hope in my perfect world with perfect humans, they start to write more about why they have generated this stuff. And regarding those two words is anecdotal because we are discussing a lot about words. A slop, our CEO said or CTO said, "If there's a slop, then there must also be an AI slip. Slop and slip." And I said, "Ah, this is interesting." So we started separating those words. Because at the beginning, an AI generated document is not slop.

(22:54):
Let's call it AI litter or it's an AI generated document. Slop already has this negative sounding, this connotation. But if I'm then starting reading it and the first thing is some code pieces and there's no opinion, then it's becoming AI slop. So as soon as I'm not happy with what I'm reading and now I'm the reviewer. I'm giving feedback, "Hey, what you have produced here is not good," then it's AI slop. Does it make sense? I mean, it's a little bit nitpicking on the words, but

Ganesh Datta (23:27):
Yeah, that makes sense. I agree with you that I think it's cultural. Even going back to pull reviews again, the idea of not taking code reviews personally and it's about the code, not the person. Again, it's not something new. Organizations have always tried to shape that where it's like we together are critiquing the code and the code is the artifact that you produce and we're trying to do that in a logical way versus you as the human that produced that code. And I think it's like that was a very cultural thing that you had to shape in a lot of ways and being very intentional about coaching people and pushing organization to that. And I think that's probably similar with this kind of world where as we review different artifacts, we'll have to create that culture around it. And maybe we're just in this middle era where we have not all aligned on what the expectations are within each organization for what humans can be given to review.

Martin Nettling (24:23):
So I mean what was the intent of a code review? What was the intent? Is it really to get good code or high quality code into the code base or is it to do the review? And I think there's also a little misunderstanding or a misconception. So some people really call it I'm doing a code review. No, this is not what you're doing. You are making sure that our code base is only filled with high quality trusted code. This is what you're doing. And the code review is only a tool or a thing that is a starter for producing this high quality code. Yes. And if you are good or you're using the right agent, then you directly have the high quality code. But what the code review should do if you start commenting. I always tell my people, if you write the second or the third comment on a pull request, then call the guy, do a mob review, talk about this stuff.

(25:18):
So for me, a lot of things that we are writing down are communication starters. They are the foundation for a meeting. They are foundation for one-to-one communication for a discussion. And this is I think what we start forgetting. So we are trying to avoid communication. So someone is producing a document. Why? Because he doesn't want me to tell all the stuff himself or he cannot focus or whatever. Or the document is for multiple people. I mean, certainly there are different aspects of it. But the main point is he wants to transport something that he doesn't want to tell me, but he wants to read me.

Ganesh Datta (25:59):
Yeah. And we've talked about this internally a lot. Doing code reviews are a way for new people to get up to speed on the code base and the patterns and the trade-offs that people are making and why they're making those decisions. And reviewing the code itself was not the intent like you described. It's something much broader than that. But that takes me to the question of as we're moving things more quickly through the system, how much of that matters today? We talk about maintainability of the code base and making sure that it's quality code mix it in. If all we care about is the chain of wise, does that matter? Is that something we can control? Does the verification process change over time? Because it kind of goes back to the idea of trust and confidence. You look at a code base, how do you build confidence as we know there's going to be some level of degradation over time?

(26:57):
I think we need to accept that fact to some degree. And can you maintain that confidence in the code base? Can you trust people? How much of this process matters in this new hour? What do you think we're going to throw away?

Martin Nettling (27:13):
This is a great question. So I'm a super fem of automation. I always was. And back to your example with those white spaces, I remember the times also when we discussed tabs versus white spaces. And then we went through the code base and replaced everything manually or just by copy pasting it. And the next one, no, I have replaced it again with taps. So now we have the linters and so on. So with this example, I think we can automate a lot. And now we are going another circle. I think if we are separating the concerns of writing code, so this is a code that really updates only a thing, updates a library for example, or this is code that only refactors something as a preparation for something else. So no new tests needed because it's only refactoring. I think we are getting there that we can fully automate this stuff.

(28:09):
Probably we can let it write on the refactor by an AI with this demand and we can also let it review by an AI. But okay, done. Is it off the code that we are writing? I don't know. I know for my guys that worked at Datamere, we strictly had this policy. If you want to rename something in a pull request, then create a separate pull request because this is only distracting. Yeah, this is fully automated. And I would say it's 50%, 60% of the code is always prepare the code in a way that you can ad the new functionality. Okay. So I don't know if this answers your question completely, but it gives us the direction of what can be automated with confidence back to this word where at least I have confidence and this could work. I cannot imagine that at the moment at least, that we are changing or we are adding a piece of code that is performant, that is functionally working, that is also from a reader perspective, it's high quality, it's maintainable.

(29:21):
So there are so many different angles and aspects of good code. I don't know if currently an agent or an AI is able to do all of this stuff.

(29:34):
Quick example, one experience I had last month, we updated the library from version four to five. I don't want to say what it was because I don't want to create mistrust or misconfidence in this stuff. We increased library and we had a bunch of very expensive load tests. So they are running for a week and so on. Those load tests figured out that only the upgrade of this version decreased the performance by 50%. So we don't know why. We have not investigated it yet. Could be they're fixing it later, but we rolled back. This was so expensive an agent cannot do it at the moment. It's not a performance test that was able to run in five seconds. So now imagine the agent is now running it for a week and then coming back and rolling back. It's probably possible, but I don't trust it at the moment that it's possible or that it's doing the right thing.

Ganesh Datta (30:25):
That makes sense. And I think what's implicit in that example about renaming or preparing the code base is that there's a level of confidence in the existing system that you will catch something that you inadvertently break by just renaming things or getting the code base ready. And so that implies an existing level of confidence that that new change, even if it's just preparation is not going to break anything. And so is that where we're going to invest more of our time is those kinds of confidence building exercises where we're going to observe the system and say, "Okay, the system is doing the things we expect it to do and so therefore we have confidence in it. " And it's kind of like the idea of SLOs to some degree where it's like acceptable levels of bad or good. And this is what we define good to be.

(31:11):
There's some level of error that is baked into the system, but overall we have confidence in the system that it's doing what it should be doing. So it's almost like we're going back in time to best practices that we should have been doing anyway, but it's all just confidence building exercises to some degree. Do you think that's where we're going of humans spending more time in the confidence building exercises on either end? Hey, the code that we're going to produce follows this set of standards and it's going to be performant because we've reviewed the intent as humans and we know that it's going to be true that we can validate those assumptions on the other side once the code is produced. Do you think that's where we're going? Or what are you seeing as the ability to build that to continue maintaining confidence level over time?

Martin Nettling (31:56):
So I think we agree and we all agree that you're using AI, that we cannot in the future review everything. That's not possible anymore. So we have to start to work on this confidence and this trust. So I would take it on a higher level. So we have a tillium about, I don't know, 500, 600 repositories and 200 active services, which is absolutely fine, but we only have around 10 engineering teams. So if you now ask for ownership of a service, then there is sometimes no mapping anymore because the team was already laid off. They won the lottery and are now doing something else so they are not available anymore, which is okay. The problem is that we don't have visibility in this stuff and observability on this stuff. So if there's a service and we are using Datadog for reboarding for CI stuff and so on, and this service is not reporting its test coverage to Datadog, how could I then say I trust this?

(33:00):
Or I mean this is only one piece. When you open it then and you see, oh, the whole CI workflow is two versions behind our other workflows. Then you think, oh, is it still working or should I first update it before I change a line of code? So all this stuff, it doesn't have a read me. It has still the default read me from two years ago. So I think if we would start not only at Tium and also at other companies to have standards and then also follow them and have high hygiene. So there is in every repository read me. I mean, this is the easiest thing. But also the CI stuff is on the latest version that we are sharing. And yeah we are following pull request hygiene standards. So every pull request has a why and a what and a good title so that we can dig into something when something is breaking.

(33:55):
So we see why have we changed this line of code? So then we probably also understand better why it's broken. So it's going back to code, but it's an overall thing. There's observability and visibility in our processes. And this is not only for GitHub repositories. I mean, we are engineers. We are talking always about code and repositories, but this is also for Jira projects. If I'm opening JIRA and I am entering a project and it has a backlog of 1000 tickets and the oldest ticket is 10 years, meaning it was created when the company used the JIRA the first time, then I say, okay, okay, then we should do this first.

Ganesh Datta (34:31):
You've talked about this idea of windows of observability, and I think it was specifically around more for developers in their inner loop and not just looking at stuff on the outside as the code is generated, but more visibility into the loop itself. Do you think this idea of observing the organization, the organization's mechanisms for validating this stuff to be part of the windows of observability that you've described or how do you validate this at an organizational level?

Martin Nettling (35:05):
I cannot answer this question. I can tell. So what we started is we are using on the pull request, and I would like to have the same for Jira. On the pull request level, we are start using labels. So for example, we have a label that risk if a pull request is bigger than 1000 lines of code. It's an arbitrary number that came out of someone's head and nobody has adapted it. So 1000 lines of code. And everyone has a chance to remove this risk label by just splitting a pull request into two pieces. But going this way, you can define risk on multiple levels. So you can also say, okay, if someone is touching this file that has no tests and no tests are written, then it's getting a risk label. Or if at all, if a pull request doesn't contain tests, then it's getting a risk label.

(35:59):
Or if a database change is made. So you can add different risk labels or can also add a comment. So at the end, you probably only review the risky stuff or the stuff that is not following your standards. But you should always give the people and the engineers or the product managers, whoever is creating an artifact, the chance to mitigate the risk to say, "Okay, why is it risky? Oh, the JIRA ticket is too comprehensive. Oh, then let's split it in two. Or it has more than 10 acceptance criteria. How can I mitigate this bad label now?" I can imagine if we start having this bottom up, so starting on the smallest units, let's say pull requests and JIRA tickets, but then bubbling up on projects like a traffic light system. Then I can imagine that we can say, okay, this part of the company or this piece here that is above is green light.

(36:59):
I don't look into this. Here's a yellow light. I will call someone. Here's a red light. We have to open the box and investigate.

Ganesh Datta (37:05):
I totally agree. And for what it's worth, that's the exact approach that we've taken internally is we started with just code review augmentation. So we had a custom cloud agent that would review PRs. It would bucket it down into areas that we know are important. So performance and maintainability and security and things like that. And then once we built confidence in that, then it was easy to go in and say, okay, well, hey guys, the code reviews are good. Why can't we now use this to label PRs? Because if we trust, you have confidence in it, so therefore you're building trust in it over time. So we can then use labeling. And so labeling was actually the second thing that we did after the initial code review phase. And then so labeling worked for a while. And that also I think built confidence in the tool, which was, okay, well, if I see this label, it aligns with my general mental model of how these things work.

(38:00):
And so that worked really well. And so, okay, well, if that works well and we have confidence in the tool to do that, then we can build confidence in the system's ability to verify itself. And so if we started adding, we want to write a blog post about this, but we use a tool for our front-end changes that can kind of diff the current state and the new state and it does snapshot diffing with live traffic. And then we did the same thing for the backend where we used AI to build this harness where it runs a bunch of production traffic through it. We can clone entire workspaces, anonymize them, and diff two shots against each other. And so now what we're trying to get to a world is like, okay, well, if we have a review that is labeled as low risk, can we get to a world where human does not have to review that at all?

(38:43):
Because we have confidence in the review, we have confidence therefore in the system's ability to label a PR and then we have confidence in our verification harness. So therefore that entire thing can just go out to production. And so it's kind of this, to your earlier point, it started with culture and the culture feeds into the confidence and then now we've built trust in its brand. So that's kind of the distinction.

Martin Nettling (39:07):
Yeah, you have this evolution everywhere. I mean, we have this evolution or I observed this evolution twice already in two companies where we had this release meeting before you release the software. Everyone said together, can we release it? Do we trust it? So the human translated now their confidence into a shared trust. So after we have created a ton of tests, end-to-end tests, we saw that, oh, we have not discovered new bugs with those end-to-end tests or with the human on the untested stuff. Then at some point we said, why are we doing this meeting? Why are we having 30 people sitting here and discussing? Let's try it for the next two times without this meeting. Okay, no production incident happy. So this was the next step in evolution in confidence in your own SDLC process.

Ganesh Datta (39:56):
I know we're coming up on time. I'll maybe ask one final question before we wrap up. What is the piece in the system today that you have the least confidence in that you think still can be automated eventually?

Martin Nettling (40:09):
Human. This is the human. I use this answer as a little segue into something else. So I think humans are pretty lazy. And this is a good thing. If we would not be lazy, then we would not automate all the stuff. We would not use AI. We would say, oh, I love typing. This is a good exercise for my fingers. But we tend to always overstep something. So there are certain things like a code review that is not only purposeful for making sure high quality stuff fits the code base. It also has a learning aspect. If you're doing a code review, you have the chance to learn something. So probably we should also already take merge code and say, "Hey, you five guys look at this pull request and give me your comments." But it's also that we have writing exercises. For example, at the end of the week, we want our engineering leaders or our teams to write a report about what they have achieved this week.

(41:12):
This is not a thing for management. I mean, this is a nice side effect that we know now what the teams have done, but we could also have looked at the tickets as managers. No, it's a tool. It's an exercise for the team to reflect what they have done, to feel proud about their accomplishment. You know what happens? They're using AI to generate this report. So now this full exercise is done. And there are certain aspects in software development where humans are doing this where there was a reason why I asked my guys to do it. Another example, quick thing. If I'm telling my old team that I brought with me, I'm working by the way for 20 years with them. And at some point in time I decided to stop writing Jira tickets. It's not because I'm late. Okay, I'm also lazy, but it was not the main reason.

(42:05):
The main reason was because I could not validate if they understood me correctly. So we had a meeting sitting all together and I wasn't writing at the end the Jira tickets and they produced something different. So why? The problem was I could not validate if they have understood me correctly. So what I started is after this meeting, "Hey Marco, hey Andre, please write the Jira ticket and then let's review it together if it reflects what I have said or what we talked about. " And though the exercise writing the Jira ticket is already kind of a validation procedure, at least in my team, that makes sure that we are at least having the right starting point before writing code, which is I think often a little bit underrated that you start at the right point. So now, and I see it in other teams where I tried to establish it, people are using just the output of our Zoom meeting, taking their part and asking, "Hey Claude, can you write a Jira ticket of what Martin has said?" This is again, not the right thing.

(43:07):
So the answer stays, it's a human because the human can misuse anything, anything. And at the moment we had a lot of, I would not call it, I don't have a better word for misuse, but we are interpreting the results of AI not always as critical as we should.

Ganesh Datta (43:24):
Yeah, I totally agree. I mean for what it's worth, this was my philosophy for better or worse, even when I was writing code by hand, was this idea of maybe it's not the right term for it, but idiot-proofing my abstractions. I want to make it very hard for somebody that's new to this code base. And I would always say it's not for the current team, it's for the new guy that comes in that doesn't know the reason why we built it a certain way. It should be very difficult for them to break a certain thing. And that was the initial intent, but it was still baked in with this idea of zero trust, if you will. There's organizations like the Googles and the Amazons of the world which have had to figure out both process and system and culture to validate changes at scale. If you're hiring a thousand interns every year and they're just coming in and writing code, how do you verify that at scale? Clearly there are systems there. So then maybe there's interesting things that we can learn from those organizations and feed those back into our systems today.

Martin Nettling (44:18):
For sure. But also they are optimizing a much bigger group. If you only have to have 10 people, it's probably not the best thing to optimize them with the same tooling. But I know what you mean. We definitely can learn.

Ganesh Datta (44:31):
Exactly. Well, Martin, this was a very, very interesting conversation. I'm very glad to have you on and thanks for coming on and talking about trust and confidence and everything in between. It was a lot of fun.

Martin Nettling (44:41):
Thank you very much.

Ganesh Datta (44:49):
Thanks so much for listening to this episode of Braintrust. If this resonated with you, do me a favor. Share it with another engineering leader who's wrestling with these same challenges. And if you want to continue the conversation or learn more about how we're thinking about engineering operations platforms at Cortex, reach out to us at cortex.io. Thanks for listening and we'll catch you on the next one.