00:00:06:11 - 00:00:25:56 Unknown Hello and welcome everyone. Today I'm bringing to you a new format of the merch or podcast here in the San Francisco office. We're calling it the AI newsroom, and I'm joined today by one of my very favorite guests, David Locker. Hi. Thank you for taking the time, David. No problem. David Locker, if you didn't know, is our VP of AI. 00:00:25:58 - 00:00:49:16 Unknown He had code Revit. And I think there's no better person to talk to about this recent massive launch that we saw on February 5th, which was the release of 5.3 Codex. And shortly before that, anthropic opus 4.6 just dropped. So we have a lot of things to talk about. Of course, this hit the news. This hit social media. 00:00:49:16 - 00:01:07:51 Unknown And I think the first question that I want to ask you, David, is what is so special about this release? Yeah. So I think, well, one is the first release in the new year. It's always interesting to see what people have been up to over the over the course of the past couple of months, and I think everybody has this anticipation for the first release that we were all sort of anticipating internally at least. 00:01:07:51 - 00:01:23:42 Unknown When is that next model going to come out? And I think, you know, we saw some improvements. So I think all in all, we've been seeing gradual improvements in sort of this idea of improving coding and how the code generation systems work. And so I think a lot of people are just super excited to see where this space is going. 00:01:23:45 - 00:01:54:55 Unknown Of course, we are developer tool company, right. So we are very interested in to see like you know, how these what these new models can do for us. So let's maybe start at the benchmarks. So there's obviously a lot of emphasis on the different benchmarks that people use. It seems like Checked coders 5.3 did make a massive jump on the S.V. benchmarks, as we know, perform significantly better on, I think 30 points and jump to 5.2. 00:01:55:00 - 00:02:18:42 Unknown And we reported that opus 4.6 tops Terminal Bench. So what do these numbers actually mean? Yeah. So I think these are indicators right. Like these are indicators of the the problem solving ability of these models when it comes to some problem sets that obviously are supposed to be indicative of the real world aren't necessarily always consistent across how you're going to perceive and understand how that model impacts you. 00:02:18:42 - 00:02:36:07 Unknown And so you should take that, and you should be excited about the idea that these systems are getting better. At the same time, real world problems are obviously a lot more diverse than what these these benchmarks can indicate, right? And so at the end of the day, get excited about it, get it on your system and try it out. 00:02:36:07 - 00:02:51:04 Unknown And I think that's how you're going to find out whether or not it's working for you. We also have some personal or internal benchmarks. He had code rabbit. Maybe you can talk a little bit about like how do these models benchmark for code review? What do you see winners, losers or best fits? I've written about this a lot. 00:02:51:06 - 00:03:13:29 Unknown Ultimately at Code Rabbit is very, very important for us to understand exactly how these models behave across a wide array of both repository sizes, difficulty level of bugs, the types of bugs that exist, concurrency issues, memory issues, security concerns and understand exactly how they're performing right, and then even getting it a level deeper. How do they communicate? Do they provide patch notes? 00:03:13:29 - 00:03:33:39 Unknown Do they provide actual fixes to the underlying problems? And really measuring all of these things because at the end of the day. These are indicators of how that model is going to perform for a user on a specific type of code base. Both models made improvements over previous generation models. Right. So 5.3 Codex is a big step up over 5.2 Codex. 00:03:33:42 - 00:03:52:57 Unknown It also I would say is a step up over opus 4.5. And whether you like one or the other in our particular case they're very similar in some aspects. So when it comes to like finding bugs, they're very similar. But the way that they find them and the way they communicate with those things are also very different from each other. 00:03:52:58 - 00:04:10:56 Unknown So for example, 5.3 Codex provides a lot of options sometimes for solving the bug, it doesn't just tell you one way that you can fix something, it will give you a few different options. And so that could come across as being a little bit overly verbose if you don't like it. At the same time, it gives you a lot more freedom of understanding. 00:04:11:06 - 00:04:26:12 Unknown How are the what are the different mechanisms? I can actually address this right? Opus is a little bit more concise in the way that it speaks, but they both kind of had a very high amount of assertiveness. They're very sure of themselves. Both of these models, right. When they make a comment, they're like, this is what it is. 00:04:26:15 - 00:04:50:01 Unknown They're very assertive. And they also this is the most patches provided across all of the different comments that we've seen from any model. So they're clearly geared towards coding, right? They want to find a solution to the problem. So that that is very apparent from our evals that they really like to provide those patches. That's quite interesting because in the public opinions, I've read a lot about this in the last couple of days, seeing people trying it out. 00:04:50:01 - 00:05:08:29 Unknown And it seemed to be kind of apparent that Codex is more optimized for speed, you know, more, you give it a guided task and then it'll execute fastly. Some people even call it like a founding engineer, you know, just get stuff done fast. And then you have opus being a little bit more long running task, high level goal. 00:05:08:29 - 00:05:31:54 Unknown And then it will, you know, do much more. We can also obviously talk about did new agent feature in a bit, but do you see that playing out in code review at all or not? For sure. You have to keep in mind that there are other mechanisms with which we can control some of these parameters on the API level that you might not have access to, or might not configure inside the cloud code tool or the harness itself. 00:05:31:56 - 00:06:01:31 Unknown Right. And so cloud code, their opus 4.6, has this adaptive reasoning where it can reason more or less depending on the type of task that it sees. And so what I saw in 4.5, for example, and the reason why I think that one is so much faster is that it didn't have that. And so ultimately, it thought the number of thinking tokens it used was significantly less for our product in comparison to if I take 4.6 and I tell it, it can reason as much as it wants adaptively for harder reviews, it thinks a lot longer. 00:06:01:31 - 00:06:27:34 Unknown So yeah, it takes it's a lot slower in that sense, but it does a very thorough job. If you look at what it's thinking about, Codex is much faster. Yeah. Ultimately, when it comes to reviewing a PR which happens asynchronously, the user is necessarily going to perceive that when you're at the desktop coding something. Right. And I'm actually doing an interactive experience that might get on your nerves to a certain degree if it's taking a very long time. 00:06:27:36 - 00:06:45:45 Unknown But a lot of people, I think, are starting to use these harnesses because they they've been building on top of them and building on top of them. They've been using them a lot more as background coding agents. If I get the plan of what I want to build really, really good and I'm really precise in how I do it, and I cover all the edge cases and I iterate on that. 00:06:45:47 - 00:07:03:05 Unknown A lot of times I can send an agent off to background code that and iterate on that over and over again and come back later. And so I'm less aware of the speed, I'm more interested in the quality of the end product. And so that's where I think opus is going to shine in that kind of environment. Right. 00:07:03:08 - 00:07:20:11 Unknown So that's actually maybe a good tangent to talk about the fact that we just launched our new planning feature. Right? So it seems like if you were to use a planning feature like the one that we have at Code Rabbit or any other tool, you have a very thorough plan. Maybe opus 4.6 would be the better model to execute on that plan. 00:07:20:14 - 00:07:42:46 Unknown Yeah, so I do think I would recommend trying them both out, see which one works better for your particular use case, depending on what you're trying to build. Because I've seen people give very, very positive impressions of 5.3 codecs. So I wouldn't say that. Just don't just ignore it and go for opus 4.6. But yes, our planning tool is like a deep research agent that again, it's not an interactive tool. 00:07:42:46 - 00:08:06:09 Unknown It happens asynchronously. And so we're able to take more time to find a very, very detailed, more optimal plan to execute on something. And then you can come into that issue and take that plan, and we can give you a prompt that will make one of these systems far more successful. And so, yes, if you have that and use our system for both of these, you're going to be more successful than you would have been prior. 00:08:06:14 - 00:08:30:17 Unknown Right. Let's talk a little bit about the core architectural differences. So opus 4.6 comes out with a much larger context window. I think it's 1 million tokens now. And then they also introduce these parallel agent teams. I think it's what it's called. And then 5.3 we already talked about like fast efficient reasoning more interactive control. How do you play it. 00:08:30:19 - 00:08:46:04 Unknown See this play out. And like what's the difference there from an architectural point of view. Yeah. So I think we need to think about the differences between the model and the harness. Right. And so cloud code has been continually rolling out these interesting features when it comes to the harness. We talked about this before which is skills right. 00:08:46:06 - 00:09:06:46 Unknown For example we have plugins. Now we have this idea of creating teams right. So subjects that each have very specific purposes. And so it's been well established that if you have a specific task and you're you're very narrow and your scope for an agent, it's much more likely to succeed than giving it a whole bunch of things to do and expecting it not to forget any of them. 00:09:06:48 - 00:09:28:06 Unknown Right. And so they've created this idea and sort of embedded it into the harness. So the model is going to perform better in that environment. Even if I was to use an older model in comparison to, to sort of not having it there. Right. And so Codex performs well and it's very interactive. And that's a very interesting workflow of I'm trying to really sort of have that experience. 00:09:28:09 - 00:09:47:36 Unknown Right. But again, if I want a long running task to do a lot of things like planning, coding, verification and kind of go into a loop, then having subjects that do specific things like having Code Rabbit review your code and focus on that and then feed that back in. This idea is very powerful. And so that's the harness. 00:09:47:36 - 00:10:04:59 Unknown And I think that's a great feature. If you were to summarize the trade offs, just to give some more practical tips, what would you say when to use which I think using something interactive makes sense for me. For me, a lot of times when I'm actually trying to engage with the work that I'm doing and trying to really maybe even learn something about the code base. 00:10:04:59 - 00:10:22:39 Unknown And I don't know exactly what I want to do right away. And I'm iterating on something, right? Iterating on ideas. That's a very interactive use case, and it makes a lot of sense for me to then use something like Codex. It's a lot faster that I can get that feedback quicker. I think if I'm going to do something in the background, then you could go either direction. 00:10:22:39 - 00:10:40:00 Unknown And so I would. I like the idea of the harness that has specific tasks that are involved, so that I know that those tasks are being covered, those things that I find as part of the an important step in sort of the software development lifecycle, the planning, the coding, then the verification that I actually did what I wanted to do. 00:10:40:02 - 00:11:02:25 Unknown So that's what I think the differences are. Okay. I mean, some people also argue about, you know, the larger context window of opus 4.6 and then this more senior developer kind of approach of thinking that is, has been called out as being substantially different in terms of how you interact with Codex being better for large codebase exploration. Is that true? 00:11:02:27 - 00:11:23:14 Unknown I think if you're doing a large PR right, if I'm if I'm building something that requires a lot of context to understand, that's when you're going to see a massive benefit, right? So if I'm building something, a large feature, if I'm building something that's going to touch a lot of pieces of code, then ultimately in order for me to do that accurately, I need to have a lot of information available to me. 00:11:23:19 - 00:11:41:22 Unknown Right. And so if it's going to depend on where you're building. So I've seen this a lot when I've been using cloud code, for example, in some cases where it'll just compact at a really awkward time because of the fact that I've been building something that actually is is quite large in a certain sense. It's a big feature, right? 00:11:41:25 - 00:12:02:33 Unknown Especially if I'm doing things like trying to understand the plan and trying to make sure I have all those details correct. That's where that larger context size is going to come into play. And so I think it opens up opportunities for people to use these systems to build bigger things that require that context. Let's talk a little bit about the availability and the cost. 00:12:02:35 - 00:12:30:07 Unknown So at the moment, opus 4.6 is offered by and for API at 5 million per input tokens, $25 per million input tokens, and $25 per output. And Codex, so far, I think is publicly only available via OpenAI's page and the new Codex desktop app, which they're pushing. But we probably also have some insights there. Do you see some cost benefit analysis for using either one? 00:12:30:12 - 00:12:59:06 Unknown I think what I've seen in the past is that they all kind of even out at the end of it, so I don't know what that's going to maintain itself with these new versions, because I'll have to see how 5.3 Codex behaves in the wild, and we've only ever had access to it at this point in terms of evaluation systems, because what I've seen prior to this is that the number of tokens used by GPT 5.2 Codex or GPT 5.2 was much higher than that of anthropic opus. 00:12:59:08 - 00:13:26:47 Unknown And so the cost was actually around the same or the same kind of review. So if it's a very complicated review by 4.5, opus would basically do it with fewer thinking tokens. Right? But 5.3 now uses fewer thinking tokens. So I want to see what happens in real world data before I sort of see whether that cost ends up evening out, or whether GPT 5.3 Codex ends up winning that battle, because that's what I've seen in the past. 00:13:26:59 - 00:13:47:51 Unknown Well, I mean, I guess we're kind of in the era where we feel like LM token cost is just more and more being commoditized, right? Yeah, yeah. Interesting though. Any specific workflow tips that you've seen working for us here internally in at Code Robots? Pacifically. Well, like, how do you see our developers using either one of these models or what do you recommend? 00:13:47:53 - 00:14:06:11 Unknown Yeah, so I see people using more and more this idea of coding in the background and making sure the plan is really great, and spawning a lot of git work trees that are that are sort of coding multiple things simultaneously. And we've been incorporating sort of Korab it into the loop so that you'll get feedback, sort of bring that back in and have AI fix those issues and sort of iterate multiple times. 00:14:06:14 - 00:14:25:36 Unknown Right. And I think as we get more and more into this idea of these systems being able to handle more and more context, handle larger and larger features, delivering code that has a little bit more robustness to it, and being able to incorporate that, that complete flow all the way through the review and back, incorporating that feedback back in. 00:14:25:39 - 00:14:50:01 Unknown I think people should start understanding how to bring that process into their own understanding and their own use of these models. Right, because they're very powerful and it's okay to a certain degree in some cases, where you throw it away because the time sunk into it is almost zero. Right? So the the syntactical generation of code that that cost is very, very low right now. 00:14:50:04 - 00:15:08:39 Unknown So the, the, the thing is like how many ideas can I actually work on? How much how long does it take me? And this is like someone from last week that we met, right? How long does it take me to go from idea to the idea of being implemented in a PR being merged? That's a very different frame of reference compared to how long does it take my PR to be merged? 00:15:08:39 - 00:15:29:33 Unknown How long does it take me to actually just code the feature up? This is a new concept, and so I think trying to figure out how to incorporate that into the way that you use these models, have them implement multiple ideas simultaneously. See how that feels, see how that works for you. Try out different things at the same time and see which one wins, rather than sort of doing things siloed one at a time. 00:15:29:35 - 00:16:01:47 Unknown I think you already touched on on that quite well. What you to kind of wrap it up, what would you recommend people to kind of dive deeper into now that we're seeing these models shift in certain ways, like, should we be learning new prompting skills? Should we optimizing for these workflows? Like what should people expect to learn? I think people should start to think about how do I communicate precisely what I want the model to do when I'm trying to communicate my intent? 00:16:01:50 - 00:16:25:39 Unknown A lot of times we think that we're communicating the context that's necessary, and there's things in our heads that we forget to put out because we think they're obvious. Yeah. And thing is, the models don't necessarily know that information. Right. And so this skill of being very, very precise in how you communicate is a critical skill. It's why we built the planning product is to make people more successful using these models without them having to know all of these things. 00:16:25:41 - 00:16:47:35 Unknown Right. But that's a skill. And ultimately that's the way you're going to get the most from the coding agents is being extremely precise. I think this is a great note to wrap it up. So thank you so much, David. Thank you. Thank you for everyone who's been listening. This is the first of many more newsrooms to come, so make sure to try it in and see you at the next one. 00:16:47:35 - 00:16:48:21 Unknown Thank you.