Rahul Yadav (00:00) Pew pew pew! Shimin (00:02) Hello. Dan (00:04) Yeah. Shimin (00:04) And welcome back to Artificial Developer Intelligence, a weekly conversation show where three software developers chat about all things AI. My name is Shimin Zhang, and with me today are my co-hosts, a Dan AI Hallucination based on descriptions of Russian Winter Lasky and Rahul Duplicate. Payments are not great. Yadav. Gents how are we doing? Rahul Yadav (00:30) Pretty it good. Dan (00:31) Surviving, yeah. I had to put on my echoey sound system today, so I'm a little self conscious about talking. Shimin (00:38) Yeah, the mood certainly has not been amazing in the industry. slash the weather lately. So I'm glad we're gonna still do this. yeah. this week's show, we're gonna start as always with the news thread mill where we're gonna be talking about perplexities, new decider model, and Google's Alpha Proof. Dan (00:55) Then uh Rahuls gonna do a little vibe and tell, which hopefully isn't me this time, but we'll see. You say that, but I'm Rahul Yadav (01:02) not you this time. Hopefully. I got time to send one more prompt. Dan (01:08) And then it's just Dans all the way. ah Rahul Yadav (01:10) Keep everything but make it Dan. That would be there. Shimin (01:14) And then we're gonna move on to post-processing, where we're gonna talk about how AI is affecting writers and readers, how software engineering has changed, a tail of two errors, and a new hierarchical network architecture for large language models. Dan (01:30) And then last but not least, we're gonna talk about well, quite a bit of news in our two minutes to midnight segment where we talk about the financial side of AI through the lens of the 1950s atomic clock because boy, has it been a busy week. yeah, so plenty busy ticking down words. Shimin (01:45) Yeah, my portfolio has been busy indeed. Uh all right, let us get started. And our uh first article this week in the news is from you. Dan (01:55) Yeah, so honestly there's enough content about this model that we could probably have made a deep dive out of it if we wanted to. But since this is a news section, I'm gonna stick to kind of the highlights, which is it's we I think we talked previously about Jav at least once or twice. So that's like a decision model, or some people are calling them system one models. I personally find that a little obnoxious, but whatever it is what it is. And it's pretty neat to see like a large, I don't know if I'd call Perplexity a frontier company, but like, you know, a large, pretty well respected AI company spit out a open source decider model. So Perplexity Decider comes in they I guess they've had a couple variants of it, which I just somehow flew under my radar. So now they're up to one point three. Um and they have one point three, one point three flash. So apparently flash is a lot cheaper to run. but they're both useful for the same type of, you know, prompt where you want to really just get statistical output about what the question is versus like, you know, try to get an actual LM response. So really useful for guardrail systems and stuff like that. And now you can run it yourself, which is pretty neat. yeah, so they have the weights on Hugging Face and then also worth checking out the whole post. You know, it's a lot deeper than I think we'd really have time for on the show, but they there's some pretty interesting stuff in there about how they basically like converted an LLM to a decision model. so yeah. Shimin (03:17) Yeah, they mentioned that they trained it based on uh Qwen three point eight, the twenty-seven Dan (03:22) Mm-hmm. Shimin (03:23) billion parameter model. And I guess the open question here is how well does the decider model compare to the original Jev? Dan (03:31) Yeah, there there there weren't any benchies and I didn't know like really where to find it. Oh, Shimin (03:36) No, there are banshees. Dan (03:37) there are? Shimin (03:38) There are banshees. I'm about to show some banshees. we haven't spoken too much about decision models on the show or s so-called system one models. But one thing one I have up on the screen is the artificial analysis their decision benchmark. And Dan (03:54) Mm-hmm. Shimin (03:55) I just want to point out that Jev Is nowhere on the leaderboard until you actually uh look for it. there we go. Jeff 1.14 is scoring a 66. While there are many others who are scoring much better. So I thought I thought that part is is really interesting, right? Like Jev dominated for for everyone, but Dan (04:22) Yeah, folks were quick to catch on, it looks Shimin (04:23) 'Cause they were the first mover. Dan (04:24) like. Yeah, it's not only that, but I also wonder, like this shows you the raw decision benchmarking, I'm sure, but it'd be interesting to compare that against cost too. Cause I think for particularly for decision models, the whole point of using them really is that like it's cheaper than you know, there's nothing Rahul Yadav (04:40) Yep, and fast. Dan (04:41) capability wise that an L L couldn't do. It's just yeah, cheaper and faster to run it. So Shimin (04:47) Yes. And so if we if we take Dan (04:49) and they've got that of course, of course. Shimin (04:51) a look at the artificial analysis decision index is cost per one million tasks, decider flash uh 1.3 is indeed on the Pareto Frontier. Um Jev on the other hand is not it's actually a little more expensive than decider flash v2. I I'm gonna let my bias be out in the open. I never really thought a lot about about Jev and the and the reason being a few things. One, they never open source their architecture. So we don't actually know how they trained it. I did Dan (05:22) Mm-hmm. Shimin (05:22) listen to a couple of podcasts where the founder talked a lot about how it was mostly synthetic data generation. and since we don't have the recipe, again, I couldn't Actually, spent much time talking about it, right? Because we don't know what the architecture is or how they trained it. it was kind of heartwarming to see all these quick follow-ups from the likes of OpenAI, Microsoft, think Alphabet also has their own Gemini decider model. And what I really like about the perplexity model is they indeed gave us their training and model architecture. As a part of their write-up. And all they did was use existing public data set to train their model. And they achieved both an absolute uh intelligence level and also a cost level that is comparable to what Jev is able to serve as at. So I don't know, Jev has really awesome marketing. I'm not sure how much is under the hood, I guess is what Dan (06:17) Well, Shimin (06:18) I'm trying to say here. Dan (06:19) I it was the first like I think really, I don't know, probably decision models might have existed before that, but that was like the big splash, right? When people realized the Shimin (06:27) Yeah, yeah. Dan (06:27) potential for sure. And like in it also happened to hit at a very interesting time in the the sort of AI continuum where Rahul Yadav (06:37) Yeah. Dan (06:38) like cost was becoming a big factor, right? And then also like people were sort of that are trying to build Agentic systems in the real world are realizing Hey, sometimes determinism is good and you know, these non deterministic systems are hard to control and you need better guardrails than what we've got. So like both of those things kind of Rahul Yadav (06:58) And it was also good, know, everybody was stuck in this LLMs, throw them at everything, regardless of how slow they are or cost. Dan (07:06) Yeah, or cost, yeah. Yeah. Rahul Yadav (07:10) And so one of the things they really contributed to is you can have different kinds of models that do different things and you need small specific models instead of general. purpose models and those are going to, if you put them in the right configuration, you're going to get a better output versus one genius in a data science data center doing everything for you. Shimin (07:32) Yeah, I'm certainly not saying that it wasn't a awesome product market fist story. That it didn't come at the the right time. We realized how they expect. Right. Dan (07:40) You're just mad that it was closed sourced. Rahul Yadav (07:40) If only it could be the best two. Shimin (07:43) That's that's what I'm kind of hoping for. Is like, hey, let's also like you better also be the best thing with your own secret recipe, but that I'm seeing less actual evidence for. Maybe it is the best. I don't have extensive usage of both perplexity decider and Jev to to be able to really say well Just going off benchmarks here. Rahul Yadav (08:00) And it's also interesting, the market is more crowded in the decision models and how quickly they came out, you know, because it doesn't take as much money and time and all that to train them versus, Dan (08:15) Yeah, and the data sets are probably much more reasonable. Rahul Yadav (08:17) yeah, versus LLMs. And so if we go by what happened and what might happen, we might get other models and it might keep getting fragmented where you have tons of competition and tons of small specific models and we might still have the few handfuls general purpose models that we have today. Shimin (08:37) Yeah, that's certainly true. And looking at artificial analysis's decision index, the elephant 9b model scored just as high as JEP 1.14. 9 billion parameters is probably something that is within everyone's training budget. So like I could see a world where every firm trains their own decision model to their specific workflow using proprietary data. And that's very exciting to us as you know, local model, open weight model proponents. Dan (09:05) Yeah. Well, it's also exciting in terms of making agentic stuff work, right? 'Cause decision model good better is custom one that actually underst it's a RLHF almost for your specific agentic flow that you're trying to work on, right? Shimin (09:20) Exactly. Okay. mm-hmm. Rahul Yadav (09:20) Shamin, did you look at the time per task tab in the cost one you were looking at earlier? Shimin (09:26) I didn't outlook at the time. Rahul Yadav (09:29) Right next to it, there's a decision index versus time per task. Dan (09:31) Yeah, there's a tab right there. Yeah. Rahul Yadav (09:33) It's interesting that the decider Dan (09:35) Elephant. Rahul Yadav (09:36) one takes longer, but is cheaper. To me, does somewhat go hand in hand? Dan (09:39) That's not surprising given the architecture to me. Shimin (09:39) Yeah. Rahul Yadav (09:42) Yeah. Shimin (09:43) Yeah, yeah, yeah. It's not surprising to me. Dan (09:46) 'Cause it's probably retained at least some of the thinking architecture I bought from its L L roots. I don't know. about there. But Shimin (09:53) Right. Again, um, it doesn't since it's based off of the Qwen three point A base model, it is still spitting out basically one token at a time, as opposed to I think Jev theoretically the spectral decoding and spits out multiple tokens at one, which partly explains why. But again, we know nothing about its architecture, so we can decide definitively say one way or another, other than This elephant five B, maybe we should do a model deep dive on at some point, because it scored just as well as Jev and is both faster and I believe cheaper. Yes, it is both faster and cheaper. Okay, after spending too much time on that one single news item, uh let's go to Rahul Yadav (10:30) Deep dive. Dan (10:33) So much for not being a deep dive, yeah. Shimin (10:35) let's go to my news item of the of the week. which is from Google DeepMind, because they occasionally write really good blog posts. this article is titled Introducing Alpha Proof Code, a breakthrough. In verifying complex software at depth. So what the headline number here is Google DeepMind created a proof that signals cryptographic protocol is actually secure. I don't know how much we've spoken about formally verified proof of code here on this program. I don't believe we Rahul Yadav (11:09) Very little. Shimin (11:10) spent a lot of time talking about, yeah. I don't believe we spent a lot of time talking about formally verified proof of code. But I think we are this may be a sign that we're starting to see gonna be a a real resurgence or maybe resurgence in interest for formally verified programs. What DeepMind did here is they took the signals and created a set of proofs based on those specs, had those proofs human verified, and then essentially proved it using the same lean framework that OpenAI used to prove their mathematical proofs, and then have a whole bunch of agents work on it until. They can say that hey, based on the signal spec, there's no real security issues in signal that they could find. To prove that this is indeed causal, they subtly introduce two bugs in the signal code And these are two bugs that were previously fixed, and they inject them into an isolated version of signal's code. The system was able to in both cases produce a machine checked proof that the buggy version was insecure. This is big news to me and that's very exciting, even though this probably means that we should all learn some formal verification going forward to check Rahul Yadav (12:22) Hehehehehe Shimin (12:23) the AI's work. yeah, given the cybersecurity attacks that we're seeing every day, this is good news, defender side. Rahul Yadav (12:31) They also took, it said it compressed the time down to four weeks or something and it would have taken years before. So is this, yeah, Shimin (12:39) By him, but yeah. Rahul Yadav (12:42) it's insane. With anyone. Dan (12:43) Yeah, formal proofs Shimin (12:43) Absolutely. And of course this is Dan (12:44) mostly reserved for like really intense like bank software or something where it really needs to be. Or like blockchain. Rahul Yadav (12:51) Yeah Dan (12:52) Typically 'cause of that cost. So yeah. Fact, that was the only time I've ever run into it in my professional career was like at a this company that was trying to make something blockchain adjacent and so they wanted to try to have a formal proof for their it works, Rahul Yadav (13:04) Um... Dan (13:04) it's cryptographically sound. Rahul Yadav (13:06) Yeah. Shimin (13:06) Yeah, but now everybody can cryptographically uh formally verify their own vibe coded video games, right? Like it's cheap enough now. Rahul Yadav (13:14) Hehehehehe Dan (13:15) It's next for Album, yeah. Shimin (13:17) Yeah, speaking of of Vibe games. Dan (13:18) Vib coating it. Shimin (13:19) Rahul what do you have for us this week? Rahul Yadav (13:21) for anyone who has played or is a fan of tar defense games, they might've come across this game called Bad North. Came out in 2018. I think maybe even before that, this might've been a later version. I'll play this one minute video to give you an idea of what the actual game looks like and then talk about where I ended up. And hopefully the... sound would come through as well. Sounds coming through. Dan (13:46) I like I played this. Shimin (13:47) Looks like we have heroes and kings doing power defense and they have a map selection. Right. So like a type tree power defense RTS like game. Rahul Yadav (13:59) That's exactly what it is. Tower Defense RTS, I just love the animation style and Shimin (14:05) Mm-hmm. Rahul Yadav (14:06) how fluid everything is. And it's very unique in its characteristics. So it's kind of like Shimin (14:12) Yeah. Rahul Yadav (14:12) the Vikings are coming in and raiding these islands and you have to defend against them is kind of the premise of the game. So... I was really hooked on to this game a long time ago and I played some other tower defense games, but nothing has come close. so like Shimin, how you have your hamlet on with how Shimin (14:31) Yeah. Rahul Yadav (14:31) each model does, I guess this has been mine, my test of like how good each model is. So I started, this was this from some time. I forget early this year, last year, this was some version of Gemini, not the latest one. And this is what it looked like at that time. And the whole prompt I started with, was something like make a bad North game, but set it in the Pacific, in the World War II Pacific theater. And so you're playing tower Shimin (14:59) Mm-hmm. Rahul Yadav (15:00) defense, but you you have like boats and all that coming. So I'll play this. You guys can watch me play a game that was live coded. Shimin (15:07) So you you didn't give it any um input on the art style or anything like that. You're just assuming that the AI knows what Bad North is. Rahul Yadav (15:15) Yes, and that was one of the big things that I kept trying and trying and, you know, sometimes you know, these things and kind of to relearn them. Where at some point last week, I was using Gemini Hydrogen and I was like, why can't you get the art as good as what Bad North has? And so it was still looking some variation of this versus what you're seeing right now and you can see a big difference. Yeah. Yeah. Shimin (15:42) Right. So go back to the previous one. We're just gonna describe it for the listeners. the original version and which model was this on? This was three five. Right. It's it Rahul Yadav (15:50) 3.5 or 3.8 something Gemini. Shimin (15:53) looks like a checkerboard with kind of rough geometric shapes, I wanna say. Um everything Dan (15:58) Mm-hmm. Shimin (15:58) feels a little blocky and the lighting does not have the same kind of soft focus. Yeah, you gotta correct me here. Yeah. Ex I'm Dan (16:03) Like yeah, ethereal quality. Shimin (16:06) not the animator here. Dan (16:07) yeah, I think it's it's Rahul Yadav (16:08) And so. Dan (16:08) not very ethereal looking. It looks it's it's Lego ish, I guess. Rahul Yadav (16:12) Yeah. um Dan (16:14) Whereas the art style of the bad north is much more um like wispy kind of spirits and like softer colors and softer angles. Shimin (16:23) Most of puppetry feel to it, doesn't it? With like soft chat. Rahul Yadav (16:25) And you can even like in the details here on the side in the stats and everything, you can also see how much more crisp things got. So. ah Dan (16:34) Yeah, the interface in the first shot that he's showing is very very vibe coded, let's put it that way. Like it has that sort of AI Shimin (16:40) Overwhelming. Rahul Yadav (16:39) Yep. Yeah. Dan (16:41) vibes style and the new one's a lot more like I can almost believe a human designer made it, thinner menus, Rahul Yadav (16:47) Yeah. Dan (16:47) sort of more purposefully structured. Rahul Yadav (16:50) so the, was my biggest mistake throughout this. maybe in retrospect, I can see if I still have access to an older Gemini model and how good that does at it. I then gave it a one hour gameplay video on YouTube. And I said, this is what the game looks like and it plays like. And if. paused the video, explored it, looked at the different, captured all sorts of frames and it did a whole bunch of things to make it look much more realistic and much more like Bad North. And so that's, that was one of the biggest things that you're seeing here that, you know, I kept saying make it more like Bad North and the like picture being worth a thousand words. All that was useless and giving it the video did the trick. So I will play the game now and hopefully the sound on this would also come through. So these little arrows are where the boats are coming from and you can move your little guys over. to go and fight them. Shimin (17:50) it's a little I see a little tower defense, yeah. Which which one are your guys, the the red guys? Rahul Yadav (17:53) Yeah, the red guys there. It still needs a little color refinement so that there's these guys and these guys because they represent different whatever the term is, battalions, specialties. How make war? ah And so you can. Shimin (18:09) Yeah. Uh I like the uh a smoke animation, the firing animation. It it's it looks Rahul Yadav (18:15) yeah. Shimin (18:16) like it actually has some I don't know personality is the right word, but yeah. Rahul Yadav (18:21) Yeah, so eh Dan (18:21) So we've got landing craft coming in with with baddies. Your buildings are somehow shooting things and then there's little dudes running around shooting at each other too. Rahul Yadav (18:30) There's little things, but if the baddies get closer, they'll bomb and the buildings. um Yeah. Shimin (18:37) Mm. That's the capture point. Rahul Yadav (18:39) And then you can, here I'll see if I can keep them on the side. One of the things that I've also learned from watching the video was initially the thing was very, so it's burning the building now because no one's going in front me, was The... My guys were so overpowered that even as soon as it came into sight they would just blow it up and then I'd be like, this is not a game, this is just a... You know, as soon as the enemy comes in, yeah, Dan (19:08) Slaughter. Rahul Yadav (19:10) you destroy it from such a distance and so after watching the video it also gave the enemy more power and so it did a bunch of these things that would have been... infinitely hard to describe and I had been trying for on and off for a good while and then just watching a video it picked up a lot of things and you can see the difference in this now. So yeah, if you don't play your guys now it over ran it and took over the island. So that's what I got. Shimin (19:36) Yeah, so giving AI a reference material if you're trying to capture some sort of a style is infinitely better than you trying Rahul Yadav (19:42) Yeah. Yep. Shimin (19:44) to capture, especially in a field that you're not an expert in, right? Like it if Rahul Yadav (19:48) Yeah, absolutely. Shimin (19:49) yeah. I I don't expect a lay person to be able to describe a software architecture at all. So like they just want a thing done. So it'll be much more helpful for them to point to a reference repository. That's truly my Rahul Yadav (20:01) Yes, yes. Shimin (20:01) experience as well. Yeah. Yeah, this is Rahul Yadav (20:03) So the. Shimin (20:04) this this has been the the October of vibe video games. Rahul Yadav (20:08) yeah, we're all warming up for GTA. That's all it is. Point it to... Shimin (20:12) Yeah, that's right. maybe nobody will be listening to this when this comes out 'cause it's I think we're releasing the next episode Dan (20:17) It's too busy playing it. Shimin (20:19) on GTA. Yeah. Dan (20:20) I won't be one of them. Rahul Yadav (20:21) We'll point our agents to GT Shimin (20:23) Me neither. Rahul Yadav (20:23) and be like, can you build me one of these? Dan (20:26) I will still be busy playing the unexpected new Slay character, Rahul Yadav (20:29) Honestly. Dan (20:31) the fool. Shimin (20:34) Taking arrows and uh what is it called when you get the arrow back? The the large the large arrows? Just lodging Dan (20:40) Yeah. Dislodging, retrieving. Shimin (20:42) the arrows, yeah. Okay, well that was really fun. Thank you for showing that to us, Rahul And we're gonna go on to Dan (20:51) Try some trust. Shimin (20:52) Processing. Rahul Yadav (20:53) boy. Dan (20:53) Yeah. Shimin (20:54) first articles from them. Dan (20:56) yeah, well it's not really from me, it's from Rachel Rachel. Shimin (21:00) Brought to us by Dan Who didn't write this? Dan (21:01) I don't know. No, I didn't. writing for Rahul Yadav (21:03) By the time we read it, we'll find out. Based on the headline, it could be Dan, for all I know. Shimin (21:08) That's right. Dan (21:09) Writing for the Atlantic. and the headline is AI is destroying trust between writers and readers. And so this is a little a little off topic for us, but I thought it was interesting because it kind of gets at some of the things that I've been feeling happening to software development too. and sort of enunciates them in a way that I as a developer and not a writer, I'm not very good at doing. the the hook of the article is essentially uh the author is is a writer and was sitting at a book book signing and this couple came up to her and was like super furious and is like I can't believe that in your book the this pivotal character has Parkinson's and you advocated for them to your they they killed themselves in your book. I can't believe that. Now the interesting thing is that did not in fact happen in the book. And so they're trying to figure out like how and had previously talked to what I assume is of like earlier gener generation of Chat GPT. and it responded to questions about the book with the f the wrong plot line where this character like does in fact kill themselves. so kind of interesting that like it could get it that wrong, but then it could also like physically destroy you could argue maybe it's not a relationship they would have had with that that reader because they clearly didn't read it, right? They just asked a chatbot about it. But it definitely injected something into the relationship between two humans that wasn't there before and caused friction that wouldn't have otherwise existed. So that's the headline. But then they they also go in and look at some other examples which are eating away at both sides of professional authors. It's interesting. So one, I'm sure we're all very aware of, even if we haven't really talked about it directly on the podcast, which is like this proliferation of I won't call them fake books, but like entirely AI written books that are on Amazon. And like the cost of generating these things is so cheap that uh People will submit the max that they can in a batch submission. Because if they even get only a handful of readers, then they will have made it worth the generation cost of these things, which is kind of wild. and then on the flip side of it, you have writers who like, you know, for years now have turned to email for their like sort of fan mail. And that 404 article where I don't know if you saw that one. I don't think we talked about it on here, but 404 media is getting inundated with chatbots like Muse, who is writing messages to them on behalf of like readers that have been locked out of things like Gmail, because these like big tech companies don't have a support Shimin (23:53) Oh, right. Yes. Dan (23:55) thing, but you see all these countless stories online of like Well, I got locked out of Gmail and so I posted on Hacker News and then it got fixed because someone that works at Gmail reads on Hacker News, right? So four four we bring media attention to these things occasionally, and when that happens, people get back into their accounts. So of course bots look at this and they're like, well, logically four four media is the best place to go to get this fixed. So let's Rahul Yadav (24:16) Hahaha! Dan (24:17) write them an email about it. So they could Shimin (24:19) Smart Bots Dan (24:20) they literally get like hundreds of thousands of these. So the writer version of Rahul Yadav (24:23) Oh god. Dan (24:23) this apparently is AI bots either impersonating other writers trying to do a a publicity scam, pay us this amount of money and we'll get you featured in these places. Or uh doing these real like tug on the heartstrings, kind of like your book changed my life, and I was headed in this way, and I read it and all this. And then it also turns out to be the same kind of scam. So it's like Shimin (24:46) Mm-hmm. Dan (24:46) filling their inbox with this crap and it's all disconnecting them from their actual readers and reader feedback. which is kinda interesting. So Is that a one to one par analog to what's happening in software? No, not really. But you can see where there's some parallels there, right? Where like Our our relationship to code that we have written is breaking in ways that's like hard to anticipate and I think like sometimes kind of hard to guess what the outcomes are gonna be. Right. So like looking at this Outlook thing that happened, I was genuinely surprised that that didn't get pinned on in some way. Because it feels like like there's this growing gap between what we're writing and what what's happening. Rahul Yadav (25:27) Yeah. Shimin (25:27) I think it directly does relate to programming because like all kinds of open source repositories are being slammed with either AI generated and useless PRs, or they're being slammed with even just vulnerability discoveries which Dan (25:44) that's true. Yeah. Shimin (25:47) aren't the all that meaningful at the end of the day. And as someone who we have a email for this podcast. The number of spam emails I get about folks who are pitching to come on the show is like really, really high. I I guess they're not I can't know for sure they're spam, but they're all AI generated because I can tell. They left the claudisms Dan (26:06) Ha ha ha. Shimin (26:07) in there. yeah, I want a world where it actually costs physical money to write people emails again, which I guess is just the real mail. Dan (26:17) Stamp. Forever stamp. Shimin (26:19) Yeah, the revenge of the stamps. Rahul Yadav (26:20) The, it's a. Dan (26:21) Well, we need to write a formally verified cryptographic system. Rahul Yadav (26:25) Hehehehehe Dan (26:26) FKA blockchain that we c we can use Shimin (26:30) I wasn't gonna say the word. Dan (26:31) the previous fad to to fight the current fad. Rahul Yadav (26:34) It's interesting if you think about why the LLMs today are as helpful as they are. It's because they were trained on useful data that we accumulated throughout our history so far. if you, let's say, know, today it's already flipped to where more than 50 % of the traffic on the internet is agent generated, not human generated. um Dan (27:00) Mm-hmm. Rahul Yadav (27:01) The LLMs of the future will have a very tough task of not being slop. And if they train on more and more slop and everything, and if slop overtakes the world, they're getting trained on slop. And at the same time, more and more of our reality is getting defined by the LLMs we talk to. And so it's also that just scary world where you're talking to sloppy LLMs and they're defining your reality and we're gonna have this, you know, we won't have a shared sense of reality anyways, but then it will be generated on good data either. It would be on slop. So it's just very crazy if things keep going down that path, what that looks like. Dan (27:44) I had a professor in a a drawing class actually that would get really angry if people used computers. And keep in mind this was like not that long ago, Shimin (27:55) Mm-hmm. Dan (27:55) right? So it's kind of odd not to be using computers at that point in time. And he was he was always his whole point was that like if you try to use like a graphic design program, right? Like not necessarily Photoshop, but you some sort of layout thing like Illustrator or something, you will get outlook outlook outlook on the brain, I guess. You will get output that no Shimin (28:15) Is it down again? Mm-hmm. Rahul Yadav (28:18) Hahaha! Dan (28:18) you will get output that at least I don't know, it could be that is good but might be flawed, right? So like even with a human operating it and Because it looks good, the computer covers up your flaws in your thinking. Um, it's much harder to make a like a really truly great design, was kind of his point. So he would always start on paper, always, in sketch, and then take the sketches and translate them into it, which you know, you can argue is maybe a product of the way that he learned to do it. But Rahul Yadav (28:49) Yeah. Dan (28:50) I think there is something to be said for that idea of there's this things that come off as polished that aren't polished, that require a honing process that don't get the honing process. That's like the difference between like agentic engineering and vibe coding, right? Like vibe coding is you you spit the thing out, and agentic engineering is there's a large amount of human effort and polish that goes into it to like make sure that it's a Shimin (29:14) Right, but it's hard to prove that some the friction gone into it. So in an incentive system where output is what is being measured and the effort Dan (29:24) Mm-hmm. Shimin (29:25) is not really measurable, like you're you're skewing towards The thought. Rahul Yadav (29:30) One other thing related to this is if you start, like I'll pick a dumb example. Let's say you're planning a trip. If you start from a blank page yourself, you will explore things much different than if you go and tell an LLM, can you plan a trip for me? And then it kind of has set an anchor off which everything else will. be built. And I feel this more and more where when we're running into problems, if we don't start from a blank page, whatever the LLM sets as our initial anchor is going to define the end solution. And that might be okay in, you know, low risk cases, but if that's the default in every case, it's actually a lot of originality and a lot of like, potential paths that are just closed out because the algorithm said, keep it between the middle probabilities and it doesn't feel good. Shimin (30:27) yeah, and and that bias would A lead everybody to come up with similar conclusions, which is what the frontier models really have their bias towards. and it prevents us from the truly thinking out of the box, which is also not great, right? Like and we haven't figured out what is the right way to To make sure. I think in some in some future world we're gonna have these workflows where you have a gap and you just know like AI can do the research and then you stop and you force yourself to do the initial outline. But we're not there yet. We're still trying to tackle this curious to how that turns out though. Dan (31:04) What Shimin (31:04) speaking of how we're gonna deal with that as a as an industry, as a s as a society, let's go to the next post post processing. To your spiral hole. Rahul Yadav (31:13) So this is um Dan (31:15) Greggerly. Rahul Yadav (31:16) from the pragmatic engineer. think Gurgle is very well known in the software engineering community. They did a contest recently where they invited people to write about how building software has changed and the winner would get $10,000. Dan and Shimin I'm sorry to share you guys did not win $10,000. uh You should Shimin (31:41) Dang it. Rahul Yadav (31:42) have submitted something. Dan (31:41) That's all right, we won a podcast. Rahul Yadav (31:44) Yeah, you're living the dream. um Dan (31:47) Exactly. Shimin (31:49) Living a fantasy. Rahul Yadav (31:52) So the winner of the contest, Neil Kucker, who I've actually subscribed to his blog for a long time. So it's been interesting to see. things changing even outside of this post. He has this whole write-up where he used to work at Bloomberg, then went and worked at Post Hub. Both of these were before the AI Times. then after that, currently works at this company, Tano. um Dan (32:19) Incidentally, that'd be a great newspaper name, the AI Times. Shimin (32:22) Hang on, Tom. Rahul Yadav (32:24) Before the AI times, after the AI times. And because PostHog had a lot of this stuff open source, he was able to pull a lot of the data from that time to inform how many PRC shipped and how much code he used to write and his productivity in that time versus post AI now. And some of the highlights there. he is, posting two and almost two and a half times more PRS per month, compared to, or from Bloomberg to post hog. And then from post hog to Tano, he's gone 11 times in number of PRS merged per month. And if you scroll down to the graph right below this, the, the orange one for Tano is, just, almost a straight line up in about a year. And then what you see in post-hog there comes out to about four years or something like that. So it's just insane the product. Shimin (33:19) He posted sixteen hundred PRs in Tano Rahul Yadav (33:23) Yes. Shimin (33:24) in two hundred days and only eight hundred and fifty nine PRs in thirteen hundred days. Rahul Yadav (33:30) Yeah, I don't know what for you're doing at post-hoc, Neil. Those numbers, they look sus. Shimin (33:35) Rooky numbers, yeah. Rahul Yadav (33:39) But, you know, and then he goes into, as he transitioned to senior roles during his time at PostHog a lot of his time was spent in reviewing code. And he used to review almost every single PR, which is just not realistic anymore. And there are a lot of these old constraints that used to happen in the before AI times. So there were things like communication overhead where you would need to put a team together to work on a whole project versus now you can just have one engineer who can own a whole feature end to end and you don't need all these coordination costs most of the time. Code reviews I mentioned earlier, they're not doing them as much human PR reviews and then in general as an industry almost every day we're farther and farther from human PR reviews because if agents are writing so much code, you very immediately buy human PR reviews as the bottleneck. And so their reviews are getting handled automatically during the CI process. They used to have PMs at Bloomberg. had dedicated PMs post-hoc. Structurally, it has some PMs, but they don't really have PMs embedded in the team. They have PMs who set the direction, but engineers are responsible for doing all the requirements, gathering product ownership part, all of those things. And now at Tano, they're just doing everything themselves. So it doesn't have PMs at all. So his job went from I'm not talking to a PM outside of the sprint planning where they would discuss requirements all the way to you do everything now and just own the whole thing end to end. Yeah. Shimin (35:23) Yeah, this something we talked about, right? developers and PMs and designers become just like Mexican showdown, kind of a deal where Rahul Yadav (35:30) Yeah. Shimin (35:30) everyone can take on each other's jobs. Yeah. Rahul Yadav (35:33) They've done a couple of my large migrations and they previously would have taken a year or more and then they're doing them in weeks now thanks to how fast they're able to move and plan. That doesn't mean that mistakes don't happen. That's where my middle name came from, where if you don't check things properly, you might pay people twice. And so you have to still, there's still that missing piece of. everything could look fine in the code you write and the testing you do and then you ship it to production and these things are complex systems and you end up making mistakes at the end. So that's still something that's happening today. And then old constraints went away but you know constraints are not just something that goes away in life so new ones have popped up. So now, Neal's a lot of his time is spent on building in parallel and they've got four or five projects going on at the same time, which means there's tons of context switching and very high mental fatigue. And I think that's a symptom generally that everyone in the industry is facing where there's, if you're context switching so much, the mental tax is super high and it is just exhausting by the end. the day. On top of that, there's a lot of thrashing. if juggling between four to five projects at any given time wasn't context switching enough, now since the cost of fixing things is so low, before if a customer ticket or something might have come up a small feature request, it would have gone through a long planning. You would have figured out when do we have capacity, when do we do it. Now, theoretically, is just one prompt away from being built and shipped to production. which means you also are, in addition to the projects, you're also spending a lot of time focusing on these reactive things that are coming in, and it's causing a lot of thrashing there. Since there are one person is owning a whole project, there is not much of the concept of a shared context anymore. And this is one of the things that I worry about a lot is companies are, these human systems and projects were a way of working together to get things done. But if everybody's just doing things on in isolation, then it also tears apart another thing in the social fabric that's already been degrading where you don't make friends at work because you don't have any reason to have as much overlap. And so the more you take away all these different things where there's shared interaction with people. The more things would, from at least, personally, it would impact people a lot because they're not getting any social time with anybody else. One thing that Neil called out that was interesting was there's no limit because the tokens you could keep using all the time. And so you have to set your own personal guardrails. And so one of the things I do is leaving the laptop at work and just giving it a couple of tasks that they know would be done within an hour or two, but then coming back to it the next day and getting over the guilt of, I could be spending a lot more time because I have all these tools, but there's more to life than just doing that and avoiding burnout by continuously getting glued to the... the AI agents and the tools we have at our disposal today. And then finally, they test on design as well where if everybody has access to Claude or Gemini or OpenAI, Codex, Pick whatever, everybody's designs are going to look pretty similar. How do you differentiate yourself in a world like that? so design actually becomes much more important and you have to have a personality and post-hoc is probably one of the best examples where you can, if you're looking for a product like that and you look at 10 of them in a row, it's almost guaranteed that it will leave an impression based on its personality compared to other products. Cause the rest of them are gonna be like, we do analytics, we help you track your usage and all these things, but they're gonna have their hedgehog and all these other things that really make it memorable. that personality is going to matter even more in the world versus, yeah, we got some hacky design out of an LLM and let's just run with it and hope that it will actually turn into a brand that we have. Dan (39:44) One question I have is how big are these companies? 'Cause like Bloomberg is huge, but it seems like post hog is two hundred people and Tano's twenty. Rahul Yadav (39:57) Yep. Dan (39:57) So I'm also Shimin (39:58) Seems like all right. Dan (39:58) like I wonder if Rahul Yadav (39:59) 10. Dan (40:00) yeah, I wonder if we aren't also somewhat conflating the s communication patterns caused by the scale of those companies with like AI advancement. You know what I mean? It's like, yes, you can move faster at a startup. That was true before AI too. Rahul Yadav (40:17) Yeah. Dan (40:18) Like uh Rahul Yadav (40:20) But I think some Shimin (40:21) But you Rahul Yadav (40:21) things, you still needed teams a lot of the time, or people had teams, there were still, know, bus factor was a real concern, which now Dan (40:32) Yeah. Rahul Yadav (40:33) it's still a concern, but you could go, well, everything is in the code. Worst case, if someone has to unexpectedly take a few days off, then you can go and point your. whatever to it and you can quickly try and ramp up from there. Dan (40:51) I had kind of a unique experience in for me this week, which was I did my first large scale handoff of a system that I'd been working for on for a bit in the AI era. And it was extremely strange because like a previous KT would have looked like that's like knowledge transfer. Sorry, I shouldn't abbreviate it, but like would look you know, you go through all the How is the code organized and where where do you need to go in broad strokes to find something? And then you jump into the individual details or you might find, point out some warts that about that could bite people or whatever, a lot of hands-on almost code review. But in this, it's just like I need you to understand conceptually what this thing is, where it's moving conceptually, and why uh those decisions were made so that you can basically take it over. Not a single line of code was discussed in an hour long meeting. It was really strange. It's like, do you understand this Rahul Yadav (41:45) Wow. Are you there? Dan (41:47) conceptually and then this and then this? Yeah, Rahul Yadav (41:50) sign of the AI times. Dan (41:52) it was Shimin (41:51) Yeah. Go going back to uh Dan your previous comment, like at a ten person startup, it may be okay to pay your creators twice. Like if you're working at JP Morgan or Bloomberg or any of these other companies, double paying your employees is like the blast radius is humongous. And I I just don't think it's feasible once you get to a certain scale to do so. to move this fast. Now how fast you can go, how cowboy can you be coding at, like that has changed fundamentally. it's Y Wild West out there. But I wouldn't want the developer who was working on the next generation of flight control or you know medical imaging to Dan (42:34) Yeah. Shimin (42:35) just be like, Well yeah, you know what happens? It happens. I'll fix it in two hours. It's all good. It's all good, dude. Rahul Yadav (42:36) What? uh Dan (42:37) Yeah. Shimin (42:40) While the robot surgery Dan (42:40) There's a ten percent chance the engine flies off and just zooms away on its own. Rahul Yadav (42:44) Ah. Dan (42:45) That's okay. Shimin (42:46) I don't know, I didn't read the code. It's okay. And and and you know, Jeff knows about it, but he's on vacation right now. He he's compiling to the subsystem of of the engine control. And that's kind of the other, I mean I think Neil is very courageous for writing out the shortcomings of this new approach on the public internet like this. being both a product manager and a developer and one of your shortcomings is you lose context on what the rest of the company is working on. come on. Does this does being siloed make you a better product manager or a worse product manager? Dan (43:20) Ha ha Shimin (43:21) But now you're also a product manager on top of being a developer. Awesome. So you're doing a worse job at both. Dan (43:27) That's not necessarily true. Rahul Yadav (43:29) The, I would, so people conflate product manager and owner and maybe like it's the product manager and owner are the two well-refined categories. But like a lot of the work that we say engineers would do as product managers, we're actually talking about product owners to a good extent, which is you're not going and figuring out the roadmap for the next n amount of time. You're not necessarily going and talking to customers day in and day out. And a lot of these things that you're not working with marketing and stuff, you're doing more. I know the immediate direction and I am going to make sure the task is broken down and delivered over time. And so product owners more mixes in with project manager stuff. And the companies do... Some companies like, he worked at post hog. They, their executives set the direction and then everybody on like, we'll try out these other product ideas that no one has tried before. It doesn't come from product managers as much. That's my understanding from the outside. And they are in charge of a lot of those like. the company and setting the direction and all of that when it comes to product, not just finance and headcount and all these things. And then product managers are more connecting different pieces that are going on on different teams to be like, we can do all of this in one and providing context where needed. So oh the structures will need to be changed. Shimin (44:57) Right, but like product management product management is fundamentally about making day-to-day trade-offs. You take decision A or decision B. And Dan (45:05) Well, what's engineering? Rahul Yadav (45:06) So is engineering. Shimin (45:08) it was engineering, yes. and it's the same problem with engineering. If you guys are making different trade-off calculations from different parts of the code Dan (45:14) Yeah, that Shimin (45:15) base. Dan (45:15) that is the one thing that I feel like is hard as someone that is an engineer that plays a lot on the product side. Is there's there's a natural and good tension between engineering and product making decisions, right? product should be the voice of the customer, engineering should be worried about the technical side, the scale, whatever else you're gonna have to tackle, right? And sometimes those things will be in conflict and that's like healthy, I think, for the product. And that's what's hard if you're playing in both, is you might Pick the one that you're biased towards. Yeah. Is I think that's what you're saying, right? So it's like, I'm an engineer, I'm gonna do the thing that's more efficient but might be a worse user experience, for example, or something like that. Shimin (45:59) Right. That and you have the left hand not talking to the right hand because everybody's siloed. So you're making different trade-offs leading to incongruities throughout your code base. And now it's even more than just code base. Now Dan (46:09) And your product, yeah. Shimin (46:10) it's the entire product experience. So I could see a world where it's very top down. your CEO just pro the guy at top just transfer all of their trade offs knowledge down to the individual silo workers. That is a buzz factor that I'm not seeing here. So Rahul Yadav (46:25) It's called founder mode, I believe. That's what YouTube tells me. Shimin (46:28) Ha ha ha Dan (46:28) All right. And that's the show, folks. Shimin (46:33) Yeah. Yeah. And clearly this isn't gonna scale. This is not gonna scale to a JP Morgan, a Bloomberg, a Boeing of the world. So what does that transition Rahul Yadav (46:44) Yep. Shimin (46:44) look like when everyone is going eleven X all the time? Like this is the the bargain that we've all struck and we're gonna see how it plays out. Um Dan (46:54) In fact, it wasn't Bloomberg, but someone huge just took away coding agents for their traders. Shimin (47:00) Dsutche bank. Dan (47:01) No, it's someone else. Maybe they did too. But it was like US yeah, it was Goldman I think or something like that. Shimin (47:04) Goldman? Goldman. Goldman, Rahul Yadav (47:04) Uh more yeah Shimin (47:07) yep. Rahul Yadav (47:07) gold goldman. Yeah Dan (47:09) So to your point. Rahul Yadav (47:10) I think, so I see what you were saying, Shemin. I think some companies by the nature of them would be big. And you can also think about, maybe we'll go through this stagnation where all the companies that are going to get started would be small because everybody's starting from scratch with AI tools. And we'll see smaller companies actually winning a lot of big markets. But at some point, you would need, then you apply SpaceX from when it started or whatever, where you go. Well, some things are just too hard and you need a dedicated company going after that. And just by the necessity of that, you might need a pretty large size of company. And, you know, everybody, even if you try and run a very lean company that's trying to go to space and do all these things, you're going to have a large company. You just can't do it as a small company. Shimin (48:03) Yeah, that's the cathedral and the bazaar argument, right? maybe we'll just have more bazaars and fewer cathedrals. And and I I think that is possible. Rahul Yadav (48:10) or more cathedrals, it might unlock a lot of things that we can pursue. Shimin (48:15) Yeah, just tiny cathedrals everywhere as far as the eye can see. Well, but but I s I I I do definitely agree with the author on the fact that like this is a new era. Whether how we adopt to it is is a different question. okay, on to our final post processing article. this one is From me, it is from Cartesia AI. And it is a a new approach about how to design a large language model. Um something that we haven't covered a ton of on this show is how exactly are tokens formed, right? Like we we are aware, you know, like All of our texts are tokenized before they're sent to a large language model. Each tokens about a couple of characters long. And what we haven't talked about is exactly what algorithm is used to create those tokens. the algorithm is Byte-Pair encoding. Um and what it is is you take a large corpus of data and you essentially start merging the most number of um Characters in common until you get to a set size of vocabulary. So think of when you have a corpus of you know trillions of tokens, you can end up having hundreds of thousands of individual tokens from the characters. And this is as far as I can tell, uh how large language models are tokenized from the very beginning. It has a lot of intrinsic downsides. For example, it was well known for a long time that the earlier GPT models are not good at coding in Python. Why is that? Because Python used indentations, and it does not have a good tokenization for how many indents a particular line is. So it would, uh the large language model would read, you know, 16 two spaces, and then it will see like a two space and a four space and a two space. And it gets all conf confused because it doesn't understand that what we actually want is for the next line to have one more indent than the previous line. And this is how also how we run into issues like strawberry. Um large language models are not good at counting characters because they're dealing at a token level and not a Dan (50:24) Mm-hmm. Shimin (50:25) character level. The last example I want to bring up, Dan (50:26) Gosh, strawberry, that takes me back. Shimin (50:28) right, takes you back to March of this year. we talk Dan (50:32) What? Shimin (50:32) about it so longingly. and of course, large language models are not good at doing math because numbers are tokenized. So a large number would oftentimes become three, four separate tokens. And the model does not have a congruent way of knowing, hey, these three tokens in a row, because it those three unique tokens. Often will not come up very often in this training data. So it doesn't have as good of an idea of how large a number is. That is, in my opinion, a fundamental flaw in the tokenization approach. And so what Cartesia and their hierarchical network is trying to do here is they're they started an article talking about the bitter lesson, Richard Sutton's famous essay. about how instead of having specialized learning algorithms with you know human crafted heuristics, you just create a very general algorithm with a learning function that learns through scale that self-improves. And that usually does a lot better when you can give it trillions of tokens as part of its training data. So uh hierarchical network essentially takes that idea and applies It to the token level. What we what they did here is this is a complex diagram, but each instead of working at the token level, text is fed into the model byte by byte. And the model itself then learns to construct chunks from these bytes of data. And if you give it enough data, the model is able to come up with rather good representation of the text based on those bytes of data. So it it will naturally learn that, hey, if you have T H E in space, that should be its own chunk of idea. That is the It also, it has a router that decides when to chunk data and then has also a decoder and encoder that takes the raw bytes of the data and translate them into vector space that then gets fed into uh another network to essentially work as usual. So in the in this latest write-up, they created a new class of models with two of these uh chunking and chunking and de chunking and encoding and decoding layer. So the text gets fed into the model a byte at a time, gets encoded. The router decides when these encodings need to be chunked and fed into the model so ice goes into network and then the reverse process happens. It gets dec the output gets decoded and turned into text. What is really nice about this is it is very general you're dealing with anything at byte level without the intricacies and the errors that tokenization approach well and they ran a couple experiments some kind of headline number these are relatively small in the 10 in the billions to tens of billions of tokens level but but they claim that the uh Through this hierarchical way of tokenization, the models are able to uh better performance view with fewer flops. As well as, and this part makes a lot of sense to me, uh, it does much better at coding tasks in languages that's never seen before. Because you're not dealing with tokenization that are not language specific, and the model itself can learn these chunks on the fly. it is able to come up with much better syntaxtical to work with. Here we can look at some examples of how chunkification happens in this model over time. So when you first start training at step one, essentially you have large chunks that do not make any sense. So you have major progress, but progress gets cut off before the S as your first chunk. But by the end of training, the encoders and decoders essentially learn. What a tokenizer would learn, right? some characters with spaced together represent an idea of a word, but you have fewer words that are split into multiple segments. And of course, on the on the learning theory level, you want to take in one word at a time. You don't want to have three tokens representing one word and two for the other. So that part makes sense to me. We talked a little bit about how regular vanilla large language models are not very good with numbers where it would chunk numbers into multiple tokens this is a good example of it so we have an AP article talking about the movie you can see everything and it In the regular llama three tokenization, the number of the sentence after analyzing 36,431 movies, the 36,431 are actually three tokens, one for 36, one for comma, one for 431. Dan (55:28) Mm-hmm. Shimin (55:28) And in the hierarchical model, the entire word is a single chunk. So I think just like on a gut level, right? Like That's a better way to represent that number than in three separate tokens. And lastly, In the hierarchical model, individual commas in the Python code do not always get their own token, even though the returns and new line characters sometimes do. So something like a variable called in underscore features will be its own token, as opposed to split into in and then another token for underscore features. Now I don't have proof that this definitely works, but just intuitively this seems like a much better representation of textual data to me. I am excited to follow more. Dan (56:12) J like generalized, yeah, but I but it's interesting. as as I was looking at it, particularly the code example you just showed, the first thing that popped in my head is like, How come we haven't tried ASTs yet? Right? Basically make the encoder be well, maybe two two step. One is take raw source and generate a abstract syntax tree out of it, right? So it's basically just like a object representation of the code. Maybe that doesn't matter. Shimin (56:37) Right. And and that's exactly what the bitter lesson talks about, right? Like instead of having human tuned ASTs into the training data, you just give it enough examples of the code base. And hopefully the model should learn something akin to an AST for a specific model. Dan (56:53) C and ASC, yeah. Mm-hmm. Shimin (56:55) And that will work better. And then like intuitively, that may not make sense, but I think Dan (57:01) Well, ASTs aren't for humans, at least not in my opinion. Shimin (57:04) Right. Dan (57:05) But yeah. Shimin (57:06) Yeah, and giving enough training data to be able to figure it out, So they haven't scaled this particular architecture up to something like a frontier level, but Yeah, but I'm excited to see where they go with this. So I will be keeping an eye out for this hierarchical network in action. And lastly, they talked about also using the same approach to train for DNA for image models. Another two cases where tokenization doesn't make as much sense for DNAs. Dan (57:32) Yeah, that's really rough for DNA. Shimin (57:35) Yeah. And And for for vision too, right? how do you decide which areas to tokenize is is questionable. And we usually do a grid based maybe the model should learn what the best way of tokenizing sections of a page should be. Dan (57:48) Well, it's interesting too, 'cause I didn't really think too much about how tokenization is really a compression problem, And I was immediately thinking about different compression techniques that have been tried over the years. Shimin (57:59) Right, yeah. And maybe maybe the model just learned the the best kind of compression already by itself. Dan (58:05) Well, it's always I think gonna be data dependent, right? there's a reason why for example, zip files in use for text because it doesn't matter that much, versus we have very, very specialized formats for compressing music and video. yeah, specialization is really around trying to preserve it's all based on okay, slice it down to only the the frequency range that humans care hear and can care about and then encode that into bands and then take those bands and it's like so yeah. Shimin (58:33) Just have the AI do it for you. Dan (58:35) Maybe we will see some AI driven novel encoding techniques from LisTech, if nothing else. Shimin (58:39) be I'd be one thing, at least one thing that AI is good at. these scotastic, scotastic parrot. so that's my post-processing. Not quite deep enough to be a deep dive, but it's interesting. So if listeners you want to check it out, the link to their write-up is in the show notes. Dan (58:55) Cartesia Shimin (58:56) All right, on to our last segment, Two Minutes to Midnight, where we check on the financial state of the AI industry using the analogy of the Armageddon clock from the Bulletin of Atomic Scientists, where midnight is when the AI bubble bursts. Now, Dan, two weeks ago we talked about what exactly does AI bubble bursting mean? Um We have not yet resolved it, but I believe Dan (59:22) It's true. Shimin (59:22) our tentative definition is an AI stock crash led recession. That's option one, or a fifty percent drawdown in the tech sector's I feels like a fair enough Working definition of it, would you say? Dan (59:39) Okay. I'll I'll need to think about it a little bit, but yeah, that's fine. For for a preprint. Shimin (59:45) Yeah. Rahul Yadav (59:46) Hehehehehe Shimin (59:46) haha And the reason for that is I I looked at what happened with dot com bubble. And that was a seventy seven percent drop in Nasdaq, a recession triggered Dan (59:54) Mm. Shimin (59:55) by it. Now, how do you define a recession being triggered by a stock crash? That's a topic for another day. Dan (1:00:01) Mean we're not economists. Right. Right. Yeah. Shimin (1:00:03) Or financial advisors. Rahul Yadav (1:00:04) Take from Shimin (1:00:07) Right, so our well actually Rahul do you have any gripes with the definition? Rahul Yadav (1:00:11) Tech stocks could be very wide. Shimin (1:00:14) Yeah, na Nasdaq or the S P. Rahul Yadav (1:00:16) I see. Does S&P have a specific category? Shimin (1:00:17) Or Mac or Mag Seven. Yeah, SP is not all tech, but SP has Rahul Yadav (1:00:23) it Shimin (1:00:23) a heavy tech bias. Rahul Yadav (1:00:25) Yeah, well, at that point, you're just talking a general drug. Shimin (1:00:28) Or or just a Mag seven, if that works for you. what what does what does the text start mean? Yeah, it's like pornography. Rahul Yadav (1:00:30) Yeah, we'll know it when we see it is my definition. Dan (1:00:34) Well no I mean that's what we've been operating on and all of a sudden Shim's gotta go quantifying our our segment here. Rahul Yadav (1:00:41) Yeah. Shimin (1:00:43) The market has been dropping. I don't know, man. Anywho, Dan (1:00:46) Yeah, that's true. Shimin (1:00:47) that's that's for later on in the segment. Rahul Yadav (1:00:49) The market is 10 % up in the past year, uh if I may. Shimin (1:00:54) Thirteen percent. Yeah. Rahul Yadav (1:00:56) Well, I was looking today. Maybe NVIDIA is bringing it down to 10. Shimin (1:01:00) Wow, maybe that was year today. Yeah. And I am a little bit behind. Yeah. All right, our first post is actually from me from the register. British fintech firm Revolut slashes AI call coding costs 10x by cutting out Claude and going self-hosted. so yes, Revolute said that they essentially ran into an issue where Claude code was costing too much. So instead they are self-serving GIM53 on 56 Nvidia B200 GPUs. of course, why does this matter? this matters because as we've talked about, tokens are getting more expensive and folks are using too much of it. So more and more firms are looking to self-host their own coding models. That would be an issue for the Frontier Labs and for the likes of well not Nvidia safe, but the frontier labs who May or may not have a moat Dan (1:01:58) I'm also thinking about how much does so that they ran it on how many GPUs? Rahul Yadav (1:02:03) Dan, I did the Shimin (1:02:03) Fifty six. Rahul Yadav (1:02:04) math. I was going exactly where you were going. Dan (1:02:06) Yeah, okay, yes, please tell me. Rahul Yadav (1:02:08) It will cost roughly between 250K to 580K per month in the Cloud And it will require an upfront hardware investment of 2 million to 3.6 million if purchased on-premise. So for anyone who looks at this article and goes, we should do this too. These are not. you models you can run on your local laptop or something. you're basically saying, given that I think that 2000 engineers or something it said in the article, if you do 2000 and they're racking up on thousands every month in token costs, at some point that math makes sense. But for anyone who's not at that scale, it wouldn't be the right thing to try and spin up their own infra. you know, their own models and then you also have to take care of keeping up with the current state of the art and everything if you want to keep pushing the Dan (1:02:59) That's an interesting choice. Shimin (1:02:59) Right. And and the question is, is GLM five point three quote unquote good enough coding model? You don't necessarily need to use table clause. Dan (1:03:08) And if you're if Rahul Yadav (1:03:09) The Dan (1:03:09) if you're not counting really tokens anymore, essentially, then you're in a position to throw it at more things, right? You could start doing crazy stuff like putting it in your CI pipelines and stuff like that. Rahul Yadav (1:03:20) And honestly, to me, the gains would come more from the decision models we were talking about. We were probably using LLMs for a lot of things that should have been decision models, and they're faster, cheaper most of the time. Or they're faster, cheaper definitely compared to LLMs, but even amongst Dan (1:03:39) Mm-hmm. Rahul Yadav (1:03:40) themselves. We were saying they were pushing the Pareto frontier. Instead of first migrating to your own hardware and running LLMs there, the better thing in my opinion would be looking at what things you have that could actually be substituted by either smaller LLMs or by decision models. And that would be a much quicker cost savings that you could realize versus trying to move all your info. or your model 7C Ronin. Shimin (1:04:08) Dan, next article is from you. Dan (1:04:11) Yeah. Shimin (1:04:12) Big news. Dan (1:04:13) I know. So Mr. Leather Jacket no longer at Green Logo. Rahul Yadav (1:04:18) I'm a stay-later. Shimin (1:04:19) That's right. Dan (1:04:19) Pretty wild. mean, I'm sure everybody Rahul Yadav (1:04:20) day. Dan (1:04:21) knows it by now 'cause we're a little late to the punch on on this. Rahul Yadav (1:04:24) Breaking news. uh Shimin (1:04:27) Ha ha. Dan (1:04:27) yeah, so they they well, Jensen stepped down over the stuff that we were kinda talking about last week. but the part that I I really wanted to focus on that's like just wild to me is like I don't know if how widely known this is. I guess it's pretty widely known is like how flat Nvidia is as a is a corporation. there are I think only four levels of management in the whole company. And famously people have something like ten to thirty six directs because of that. But Rahul Yadav (1:04:58) Jensen have 50 or 60. Dan (1:05:00) Yeah. But this is not your average like we booted the CP CEO, okay, cool, new one picked, life goes on situation, right? Like the is kind of the heart and soul of that company since the beginning. So yeah. Shimin (1:05:13) There is talks of him coming back. The board Dan (1:05:15) Right, I saw that. Shimin (1:05:16) says no, but we'll find out. Dan (1:05:18) Yeah, there's a coalition apparently being put together by uh various well known Valley folks that are trying to to bring them back. But yeah, it's like it's really wow. I I I went so far and I'll just I'll put this on the record so we can laugh at it later. I did a hot take on our our little podcast chat where I was like, I think this could be the really Unexpected keystone that I think blows the whole thing up. You can we can all laugh about that when I'm dramatically wrong later, but Yeah. Yeah. Yeah, and Shimin (1:05:46) Yeah. And of course the video stock went down like fifteen percent, you know, since we last recorded. So Dan (1:05:53) it's been ch ch ch ch ch ch ch ever since, right? Still kinda ticking down. Taking a lot of tech with it. Shimin (1:05:59) How's the company's financials looking, Rahul? Rahul Yadav (1:06:02) So to balance what Dan said, the company released its Q3 earnings just this afternoon, three, four hours ago. And their total revenue is almost $116 billion, up from $96 billion, so $20 billion more in Q3 versus Q2. um Dan (1:06:24) A hundred and three percent. Rahul Yadav (1:06:26) Yeah, gross margin is down by like 0.1 % and their operating expenses are higher by about 10 % because they have been doing a lot more spending. Some of this is on stock buyback. I don't know all the financial sharpness that goes into what goes where. So I Dan (1:06:48) Mm-hmm. Rahul Yadav (1:06:49) wouldn't be able to tell by looking at what is an expense and what exactly is counting as income. But assuming all of that is because it has been a publicly traded company for a long time and all of these are, it follows GAP accounting. All of these we can assume are very cleanly reported as much as possible. So the interesting thing to me is they also give their guidance for what they're looking at the target for the. next quarter for Q4. Dan (1:07:22) Mm-hmm. Rahul Yadav (1:07:23) And so their Q4 guidance is they would go up to 130 billion and so another 15-16 billion or so. They said plus minus three percent and so you can make your adjustments off of that but it's still they're going to add over 10 billion in the next quarter. And it's excluding any data center spend that they're going to get out of China. So that was an interesting piece that they added there given all the news that's going on. Dan (1:07:52) Yeah. Shimin (1:07:53) feel like I feel like all of China should be in quotations with a large asterisk. Rahul Yadav (1:08:00) And then also gross margins are still, you know, they were 75 % in Q2. They're going go down to about 71 or 72 % in Q4, but that is still an insanely high margin for this company. And I get the whole, you know, there was a key man in that position and he's gone and so everything is doom and gloom. But on other hand, NVIDIA has 42,000 full-time employees, and they're some of the smartest people, most of them, from all around the world, which means, and companies usually are very resilient. And so I'm not, it's gonna be short-term painful, definitely, but I'm not necessarily buying the long-term argument that without him, everything is going to be just down. I think the market is doing what the market always does, which is just being highly reactive to these things. But the financials. Dan (1:08:54) If you're a financial person you'd say pricing in the uncertainty. Rahul Yadav (1:08:57) That stuff, Shimin (1:08:59) I lost my watching CMBC. Uh Rahul Yadav (1:09:01) if you're a smart person, would watch Jim Dan (1:09:02) I listen to earnings calls sometimes. Rahul Yadav (1:09:05) Kramer wherever he is. And so anyways, to me, the financials, the current financials obviously were reporting all the Q3, but also their guidance based off even all of this is still pretty strong. My guess is long term it's not going to make as big of a difference. Short term it would be painful. So yeah, that's what I would leave it at. No, no, this is financial advice. Please buy and sell and ready Shimin (1:09:35) Thanks for saying that. Rahul Yadav (1:09:36) all other stocks at your own peril. Dan (1:09:38) Yes. Shimin (1:09:38) My my gut actually says completely the opposite. I think short term it's gonna be fine because you don't sink a ship by taking out CEO overnight. I think long term is where they may have issues, but we can agree to disagree on that. all that said, how do we feel about the clock this week? Rahul Yadav (1:09:54) I'd keep it as is. But if you guys are feeling uh doomy, Dan (1:09:57) Keep it as is. Rahul Yadav (1:09:59) you can. Dan (1:10:00) Feel doomy. I just call that Tuesday. Um Rahul Yadav (1:10:02) Hehehehehe Dan (1:10:04) wow, it is Tuesday. Wow, what a weird coincidence. That joke works better when it's not a Tuesday. Mm. Rahul Yadav (1:10:09) Wow, Dan (1:10:09) I Rahul Yadav (1:10:09) Dan's really thinking hard about this one. Dan (1:10:12) I am. I'm struggling a little bit because it's like, the scene is pretty bad. But we haven't actually seen I don't know, like the whole market's down. Shimin (1:10:20) Yeah, the the we have to add in the evidence that the whole market has been down. Nividia has been down percent Dan (1:10:27) Yeah. Shimin (1:10:28) more since we last spoke. That's that's the real kind of the tangible evidence we have. So Dan (1:10:33) Yeah. Rahul Yadav (1:10:33) This is, like I said, not financial wise, but I usually buy things when they go on sale. So, Dan (1:10:39) Turns out Rahul's Shimin (1:10:38) And if you wanna if you wanna hear more about it, go to Rahul's finance, personal finance podcast. Yeah. Dan (1:10:39) been buying NVIDIA. Other podcasts, yeah. Rahul Yadav (1:10:45) Black Friday, Cyber Monday sales are coming Dan (1:10:47) Ha ha. Rahul Yadav (1:10:47) early. What can we say? Dan (1:10:50) It's not you can't afford an NVIDIA card, but now you can buy their stock. Shimin (1:10:50) I am ink. Rahul Yadav (1:10:54) Yeah. Shimin (1:10:55) I'm inclined to move it maybe five to ten seconds forward. And that's that's really Dan (1:10:58) Yeah, I was thinking fifteen. Shimin (1:11:01) what my gut says. Nothing more than yeah, 15. Okay. I can be talked up to a 15. And Rahul, you can disagree and you can tell us why Rahul Yadav (1:11:09) No, it's fine. Shimin (1:11:10) on your own personal finance podcast. Um so Rahul Yadav (1:11:12) Hahaha! Shimin (1:11:14) we're gonna move the clock to two minutes and thirty seconds, and as always, with the setting of the clock. Yes. Rahul Yadav (1:11:18) Entropic, can I just say Entropic will have RSI by next week with its IPO and you guys would look so stupid with moving this clock down. Dan (1:11:29) That's that's that's why we do this. It's not to look smart, it's to look stupid. Shimin (1:11:31) Rahul, you you you forget that if RSI is a thing then money does not matter anymore. This this is this is purely Rahul Yadav (1:11:39) Short-term it will matter. The stock has to go up. Shimin (1:11:44) All right, guys. with a sitting o'clock. that's the end of the show. So thank you all for joining us for our chat this week. If you like the show, if you learn something new, please share the show with a friend. You can also leave us a review on Apple Podcasts or Spotify. It helps people discover the show and we really appreciate it. If you have a segment idea, a question for us or a topic you want us to cover, shoot us an email at humans at adipod.ai. We'd love to hear from you. Dan (1:12:05) Or or or you need help recovering your Google account. Shimin (1:12:08) Don't give the muse agents any ideas. Dan (1:12:09) Yeah, Shimin (1:12:10) if you can find folk show notes, transcripts, and everything else mentioned today at www.adipod.ai. Thank you again for listening and we'll catch you on next week's episode.