{ "version": "1.0.0", "segments": [ { "speaker": "Jack", "startTime": "0.0", "endTime": "18.335", "body": "Hi, everyone. We're AI Engineers Europe in London, and I'm joined by Lawrence from incident.io. Hi. It's great to be here. Lawrence, I know you're doing a talk about this is my butchering of your title, but it's like AI against AI, more or less." }, { "speaker": "Lawrence", "startTime": "18.415", "endTime": "41.205", "body": "Yeah. Yeah. So I think this is a theme that's coming up more and more both, like, actually in the products that people are producing. So for example, we are building an AI SRE quite fundamentally on the basis that when people are shipping a ton more of this software, they need AI to actually help them manage that stuff in production. The more software that they put out, the more problems that you end up hitting." }, { "speaker": "Lawrence", "startTime": "41.205", "endTime": "76.424995", "body": "So if you can get AI to help you debug those issues faster and hopefully tackle them before they even land, that's something that can help manage the complexity of shipping all this software. But my talk is specifically about what we are doing internally when we're building this AISRE tool. So we are obviously running these investigations to try and root cause your incident. And when we're doing that over hundreds or thousands of customer accounts and doing thousands of investigations a day, Actually understanding how the system is performing is not something that a human can do. So for our internal tools, we've actually leveraged a lot of AI to try and understand how it's performing." }, { "speaker": "Lawrence", "startTime": "76.424995", "endTime": "85.07", "body": "If it's getting better, if it gets worse, why has it got worse? And I think this is just gonna be a consistent theme, like fighting fire with fire or AI with AI." }, { "speaker": "Jack", "startTime": "85.39", "endTime": "90.11", "body": "And how how is it, like how does it perform? Has it has it been helpful?" }, { "speaker": "Lawrence", "startTime": "90.43", "endTime": "121.409996", "body": "Yeah. So, I mean, like, we we have a ton of tools to tell us how this AI SRE is performing. So, like, we're working with several design partners now, so anywhere from, like, really big companies through to, like, even just running this on our own account. Like, when you've got the system configured well, like, we're seeing, like, 85, like, 90% accuracy on the incidents that we're running this root cause analysis against, which is really mind blowing and not something that we really thought that we were going to get to. But the real hard part is it's such a nondeterministic system." }, { "speaker": "Lawrence", "startTime": "121.814995", "endTime": "140.15001", "body": "We are plugged into logs, metrics, traces. Sometimes your logging system may just be down. So we might run the same investigation one day, and then the next day, it doesn't do quite as well. Maybe that's us. Maybe that is variability in the AI, but potentially it's even that, like, your logs aren't responding or, like, your application is misconfigured." }, { "speaker": "Lawrence", "startTime": "140.55", "endTime": "158.475", "body": "So trying to dig through all of those all of those all of the needles in the haystack to try and figure out is this actually us, like, what's actually going on? Are we getting better? Has it got worse for a reason that we can control? That's what all of this tooling is is kinda built to do. And, it is making making it a lot better." }, { "speaker": "Lawrence", "startTime": "158.475", "endTime": "159.835", "body": "Yeah. I can believe that" }, { "speaker": "Jack", "startTime": "159.835", "endTime": "170.25", "body": "you wrote a very good report on this as well because I I read I read a certain blog post that you wrote probably a couple of years ago now, the laptop the new MacBook." }, { "speaker": "Lawrence", "startTime": "170.25", "endTime": "172.97", "body": "Oh, yeah. No. Of course. Yeah. That was really funny." }, { "speaker": "Lawrence", "startTime": "172.97", "endTime": "213.07501", "body": "We had, like I was trying to I was I mean, I we had made the case to upgrade everyone's laptops, but it was just I was trying to use some of the new OpenAI stuff because they were the first ones to put together this, like, a like, data analysis platform. So I was trying to use all the data that we had from build times and everything that people are running internally on our dev laptops to justify the ROI on, like, upgrading everyone's fleet. But I think, like, the interesting thing is, like, yeah, it's kinda funny you mentioned that because I'd almost forgotten. That is kind of the genesis of a lot of this stuff, having, like, analysis playbooks that you can use to pull through all of this data. AI is really bad at doing math." }, { "speaker": "Lawrence", "startTime": "213.07501", "endTime": "236.03502", "body": "It's, like, very bad, in some cases, at drawing conclusions about things. But if you can put your human expertise into a runbook and have the AI run that, it actually does really, really well at drawing meaning from a lot of data. And that's exactly what we're doing in the moment to analyze at scale how AISRE is running. It's not getting us new MacBooks that I know, but it is helping us ship a really, really great product for our customers." }, { "speaker": "Jack", "startTime": "236.115", "endTime": "253.88", "body": "Okay. And then the question that's we're gonna ask the dumb, naive question. So how how does it why can't I just, you know, chuck my error logs into court? And what's where is the kind of where are you seeing, like, the special source and what really makes the difference?" }, { "speaker": "Lawrence", "startTime": "253.88", "endTime": "268.535", "body": "Yeah. So there's a there's actually a bunch of reasons. So like naively, if you think about like telemetry data and logs, like Claude and all of these AI tools are really context limited and logs can be gigabytes of logs. Like you've immediately blown the context window. Yeah." }, { "speaker": "Lawrence", "startTime": "268.615", "endTime": "295.64502", "body": "So when it comes to interacting with telemetry data, we have like, very specific ways of formatting and structuring and summarizing that data so that the AI can get, like, a summary and then dig into the interesting parts. So context management is a big thing. But the biggest reason is that if you just open up Chord and you go, hey. Figure out what this problem is, you're essentially taking a really smart engineer from a different company. This is Chord in this case." }, { "speaker": "Lawrence", "startTime": "295.64502", "endTime": "308.74", "body": "And you're asking them to debug your production incident. And, like, we know that this doesn't work because we don't hire engineers and put them in our team and go, hey. Good luck. Here's a SevZero. Like, the best people in your organization are both extremely smart." }, { "speaker": "Lawrence", "startTime": "308.74", "endTime": "335.14", "body": "They know how to leverage the context in the organization. They have so all of your history, but they also have memory, they know how your organization works. And that's exactly what our tool does. So our tool is continually building an understanding of your organization, of your production infrastructure, and then it's using that and feeding that to the AI so that we can actually figure out what's going on. If you just take Cord and you say, hey, give it a shot, It will spiral off and it will go do things, and who knows if we'll find the right answer and you can't really trust it." }, { "speaker": "Lawrence", "startTime": "335.14", "endTime": "366.1", "body": "With our system, we're grounding everything that we say and what we've seen happen in the past. We have, like, why a lot of structures in place to try and, like, grade our confidence and kind of align ourselves with what's going on in the instant channel. And then there's a bunch of, like, incident there's product packaging with this. So, like, we're shipping a desktop app so that you can jump directly into cord code or codex or whatever you use, where we'll pull down all of this investigation data, and then you can pair directly from your cord session with our centralized agent. And if you're doing that in team, then maybe you have several different engineers, maybe you're even remote." }, { "speaker": "Lawrence", "startTime": "366.1", "endTime": "385.615", "body": "Now you suddenly have these agents are helping people on their laptops, but everything is being fed back centrally. And if someone finds something on their machine, we'll we'll prompt you to say, hey, do you wanna share that back into the main incident back to the people who need it? And suddenly someone gets a notification that goes, hey, like, actually Lawrence has found this thing. It probably looks like it's something else. And then you can redirect what you're doing." }, { "speaker": "Lawrence", "startTime": "385.93997", "endTime": "387.86", "body": "There's a huge amount of coordination, and there's a" }, { "speaker": "Jack", "startTime": "387.86", "endTime": "399.3", "body": "huge amount of organizational context, like, that that is exactly what the product is. Yeah. That that makes total sense. Okay. And then what have you been excited about at AI Engineers?" }, { "speaker": "Lawrence", "startTime": "399.625", "endTime": "424.63998", "body": "Honestly, like, just speaking with so many of the people here, like, yeah, like, speaking with actually a ton of customers as well, like Brain Trust and a couple of other people. Of course. It's just like watching everyone try and solve these problems, which are quite novel given the the age of AI and seeing all the really, really cool stuff that they're building that was just simply not possible before. That is actually what's been really energizing and really exciting about this this event for me." }, { "speaker": "Jack", "startTime": "425.255", "endTime": "428.695", "body": "Is there been anything that you're like, I'm gonna take that away and" }, { "speaker": "Lawrence", "startTime": "429.495", "endTime": "463.375", "body": "Yeah. So I think, like, I was speaking with Brain Trust earlier about, like, how their system works and how they how they kind of have their custom tracing solution to try and store all these traces. I think like they have some like ambient, like analysis of everything that's going on in the system and they try and identify patterns that you may not have previously even thought to look for. And I think we are very targeted in all of our internal tools at the moment and we know what we're grading and what we're looking for. I'd love to get something that was a bit more like looking for the thing that we didn't realize was there or trying to identify trends that we hadn't even thought about." }, { "speaker": "Lawrence", "startTime": "463.695", "endTime": "464.975", "body": "That for me was, like, really," }, { "speaker": "Jack", "startTime": "464.975", "endTime": "469.375", "body": "really useful. Okay. Amazing. And okay. Wait." }, { "speaker": "Jack", "startTime": "469.375", "endTime": "473.68", "body": "This is definitely the final question. Okay. What are you most excited about at Incident at" }, { "speaker": "Lawrence", "startTime": "473.68", "endTime": "497.27502", "body": "the moment? I mean, I got I like, I'm really, really, really excited about getting AIS three to everyone. So I think, like, we are properly launching this for general access, like, very, very soon. And, like, watching our design partners who are a total range of companies gradually come in with more feedback being like I think we had someone say the other day, like, just send us a message being like, actually, I'm quite shook. Like, I can't believe it figured this out." }, { "speaker": "Lawrence", "startTime": "497.27502", "endTime": "532.67", "body": "And, like, other people being like, this would have taken four of our engineers days, and I'm still not sure if they would have found it. Like, some of the stuff it does is just surprising, and, like, this is where AI can really, really uplevel your team. We've we've had it find things before where, like, our engineers were debugging an issue with sending pages in China, and they really thought that it was gonna be the global firewall because, like, it normally is when you're dealing with stuff in China. But then the AIS three actually started scanning the documentation for our telecom provider in China, which obviously we can read because it's in Chinese. Yeah." }, { "speaker": "Lawrence", "startTime": "532.67", "endTime": "547.39496", "body": "And it turns out there was a documented thing there that was, like, beyond seven hundred and fifty milliseconds. It'll time you out. And then it correlated that with a trace, and it came into the channel. I was like, guys, I I know you think it's a firewall, but, like, it this is actually documented in these docs. So it stay open and went, oh, god." }, { "speaker": "Lawrence", "startTime": "547.39496", "endTime": "557.31", "body": "Like and that stuff is, like, you you're like, that is just not possible without these tools. Yeah. So, yeah, that's what I'm really excited about. I'm really excited about everyone's reaction when we get when we really release this product. Yeah." }, { "speaker": "Lawrence", "startTime": "557.31", "endTime": "558.58997", "body": "And you think that that's coming soon." }, { "speaker": "Jack", "startTime": "558.58997", "endTime": "564.67", "body": "That is coming as Rob said. That is Emmy Berry's. Everyone, stay tuned. Yeah. Thanks so much, Laura." }, { "speaker": "Lawrence", "startTime": "564.67", "endTime": "565.47", "body": "Thanks for having me." } ] }