00;00;00;00 - 00;00;19;04 Unknown Faster models are smarter models. I think that is true. Trivially, you can imagine that even if the model is not super capable. But it does one thing extremely well, like the faster it can do that, the better. Any time I know exactly what I want, and I'm confident that I can communicate it explicitly and neatly. And I just need that thing done. 00;00;19;04 - 00;00;42;15 Unknown Exactly. Codex is my guy. At any time, I need the model to understand what I mean. That I use bug code. Why would you use a model that's 90% as good as a different model? Yeah. If you have to do one thing with the model, if I need the model to change the color of that, that, that token from blue to red, you know, I don't care if it, if I can count to ten. 00;00;42;17 - 00;01;05;26 Unknown Yeah, I care it does that the best in the world. But we do want to make sure that the model has that smooth frontier, right? In the situations we know the model is going to be in, it's correct to say, that the ultimately we will wind up with. 00;01;05;29 - 00;01;26;08 Unknown Hello and welcome everyone to another episode of the merch. The podcast we bring to you here live from the robot office in San Francisco. My name is Hendrick. I'm a developer advocate here, and today I'm joined by Chris Alexios, product research engineer at Nvidia and, many, many more. The things and professions Chris does, which we're going to learn about in a minute. 00;01;26;10 - 00;01;44;07 Unknown So, super excited to have him here. Chris, how are you doing today? It's been a busy week for you. Yeah, man. It's, it's been good. It's been a little bit, of, of a crazy week, obviously, so. But, overall, it's been really exciting. Sorry. My voice is a little bit shy. We we had some, celebration last night with the team, so. 00;01;44;07 - 00;02;06;14 Unknown Yeah, a little bit scratchy this morning, but it was. I mean, it's GTC, it's the Woodstock or the Super Bowl, depending on who you ask of I am. Yeah. It's just such a pleasure to meet all of the incredible developers. I get to come to the conference and it's been a blast, a whirlwind, but a blast. Yeah, yeah, I was there, yesterday and I think the, the atmosphere, even though was the last day of the conference, was incredible. 00;02;06;20 - 00;02;22;25 Unknown I think to like GDC has a lot of things to a lot of people. And I think but first and foremost, at least, you know, from my perspective, like it is a developer conference, I think there are not as many of those these days. There's a lot of like, kind of enterprise conferences. There's a lot of like research conferences. 00;02;22;25 - 00;02;42;09 Unknown But there there are fewer, conferences that are like dedicated to sharing knowledge with people who are building with tools or building tools. And so, you know, even on the last day, which is like our workshop day, you know, people are learning all day how to build cool stuff. And it's the the atmosphere doesn't really die down. And until we force it to. 00;02;42;09 - 00;03;04;20 Unknown So, you know, it's like, we gotta we gotta shut the doors at some point. And and that's about what kills the vibe. Yeah, yeah. That's true. I remember we had to. You had to force people to leave, the parameters. Yeah, yeah. Oh, great. We can talk a bit more about Nvidia and GDC in just a moment, but maybe for people who are watching this, who haven't heard about you yet, can you give us a little bit of an intro about yourself? 00;03;04;26 - 00;03;27;05 Unknown Your career path, and how did you end up here? I just like building stuff. I've always liked to build stuff, and that's basically my whole career path is it's been like building stuff. And then, yeah, I've learned about it. And then, just to take that to it's right now it's logical, extreme. I think, you know, I have a nontraditional education background, so I don't have a masters or PhD. 00;03;27;07 - 00;03;47;02 Unknown I, you know, not not self-taught by any means, taught by very incredible people, along the years, but, you know, and then got kind of into the startup game a little bit, did some work for as data scientist, which is, you know, obviously machine learning is the absolute best. Learn that lesson very early. 00;03;47;04 - 00;04;06;08 Unknown Computational physics class, when I attended, Trent University. So, you know, just, just been cooking. Cooking ever since, for the startup game on the, on the kind of side of the startup. You know, I'm saying I was doing some YouTube videos and things like that, trying to bring the technology that I love, and that was important to as many people as possible. 00;04;06;10 - 00;04;28;25 Unknown And that's when I interviewed that, good old fashioned Nvidia. And, here we are. We have, you know, a few years later. Yeah. I think, and I watched some of your content and I watched a little bit of your trajectory as well. I think it's it's really amazing. The, if you check out one of Chris's videos, you see the, passion that really goes into what the concepts that you explain. 00;04;29;02 - 00;04;54;27 Unknown Can you tell me a little bit about how that passion, you know, started first? I think it's the word that I'm like that people most used to describe me, which is not the word that I would have thought it was when I was younger. Right. Which is, I'm very passionate guy, I think, very excitable person. I get, I'm very obsessive about the topics that I'm interested in and so, you know, when discussing those topics, you know, I, I lead with all of my energy, right? 00;04;54;28 - 00;05;16;18 Unknown So it's I think because of that, it kind of keeps driving me forward to, to continue to like, either write about, you know, these topics or to talk about them. But so I was in uni the first time that I heard about, machine learning. And it was like this idea that you could write programs that did stuff based on rules. 00;05;16;19 - 00;05;34;06 Unknown Obvious. That's great. That had been great my whole life. I really loved that, in fact. Right. Like, I was in like a code camp when I was like, six or whatever. The universe shut out University of Manitoba, you know, with my with my, my grandparents taking me every day. But then suddenly you learn like, well, we don't actually need to write the program, right? 00;05;34;06 - 00;05;50;27 Unknown We need to set like, kind of some conditions and then get a big pile of data. And then we can, we can figure the program out over the course of, of training. And that really just like expanded my mind like 700 times. Right? This idea that, we don't need to write the rules, we can learn the rules. 00;05;50;27 - 00;06;09;18 Unknown Right? And, I built a, really dumb project, and I recommend this to everybody. Like, my buddy had a bird. We used to play league of Legends, together, and we would be on Skype with each other. So we communicate. And I loved the bird. Okay. It was a cute little, I don't know, like a parakeet budgie or something like that. 00;06;09;19 - 00;06;25;23 Unknown You know, the little the little bird. And I was. I wanted to see the bird whenever it was on camera. But my buddy wouldn't always remember to tell me, so I. I hooked up a visual language to my webcam, called bird, but that would notify me when the bird was in the frame. Right? Oh, wow. Yeah. 00;06;25;28 - 00;06;42;13 Unknown Yeah, it was so dumb. Right? But, like, it got me using TensorFlow at that time. And this is like, old TensorFlow was, like, really rough. And then from that you immediately can see like a world of possibilities unfold before you. Right. Like what if we could do this better and faster? And what if it wasn't like a bird? 00;06;42;13 - 00;07;01;07 Unknown But it also, you know, something more, more interesting, but like that, that is kind of what set the spark to push me down the path that got me to where I am now. I think that's that's really interesting because, you know, you also then participated in other ventures like the AI makerspace, took that passion to, to these type of avenues. 00;07;01;10 - 00;07;26;15 Unknown Plus and also, of course, telling more people about how to use it. How did that actually work under the hood, which I think is amazing. So that curiosity really took you a long path. Yeah, I have like a very specific mission in mind when I, whenever I talk about something about, machine learning or not, what we call AI, which is like, I think that the people we need in the world to actually help us with artificial intelligence, like, don't know what it is yet. 00;07;26;15 - 00;07;43;06 Unknown And there are loads of curious and smart people who have a wealth of knowledge, right, that we that we just haven't, like, rolled in yet. Right. And you see this all the time. And in any kind of like AI research where it's like, oh, this idea from this random field from the 1950s has like made our models better. 00;07;43;06 - 00;08;05;26 Unknown Right? I think that, because I believe this to be true, it makes it really easy to want to talk about it, to everybody and to reach as many people with, with, you know, 100 level, 200 level knowledge so that they can kind of, you know, have the same experience I did with Bird Bot where it's like, oh, like, I could apply this thing that I know to this system and like, maybe it would work. 00;08;05;26 - 00;08;20;18 Unknown And then they try it and usually it does work. Right. So, that's that just keeps me motivated to want to like. Yeah. Promoted all the time and teach as many people as I'm allowed to teach. You know, what this technology is and how to use it so that we can get the advancements, I think are critical for us to move forward in the field. 00;08;20;19 - 00;08;49;03 Unknown What kind of advancements do you think are most critical? We have to make models faster. We have to make systems that sit around models more capable, and, and hopefully leverage techniques that we've picked up from, like, cognitive sciences and like, especially when we talk about things like memory for models. Right. Or for agents more specifically, I think, you know, we have this idea that, like, agents are the models in a loop that do stuff, and we, we like we have like the most basic form of memory, which is like context window. 00;08;49;03 - 00;09;07;11 Unknown And now we kind of like are expanding it a little bit, but I'm certain that's not the most optimal approach. Right. Like our current memory systems are far from optimal. And so we need better ones. The faster the model can run the like, the better it is. I'll say the line because I believe in my heart of hearts, which is like faster models or smarter models. 00;09;07;13 - 00;09;24;06 Unknown I think that is true trivially. You can imagine that even if the model is not super capable, but it does. One thing extremely well. Like the faster it can do that, the better. And then also like, you know, talking about content safety. So security is like a different thing. We need lots of security experts. They're cooking. They're doing their jobs. 00;09;24;09 - 00;09;42;15 Unknown Like content safety is really important. These models are producing a lot of tokens. Those tokens are seen by a lot of people. And like the safer that content is, and the better aligned that content is to our respective values, be they, you know, Americans, since we're in America right now, we're Canadian when I'm back home or whatever other sovereign nation. 00;09;42;17 - 00;09;57;29 Unknown Right. I think that, those kinds of efforts are extremely important to, to nail down, like, I don't want every model to be the same, want it to reflect the people that, that will use it and then reflect the people that made it. I want to put a small pin on something that you said before as well. 00;09;57;29 - 00;10;14;21 Unknown I when I was going through some of your previous work, I saw that you also published at least some, some hotcakes or some, some guide, some pieces about, things like, context engineering. And then the fourth piece from the evolution from prompt engineering there. We obviously had code rabbit care a lot about context engineering as well. 00;10;14;21 - 00;10;39;06 Unknown So I'm kind of curious to hear your your talk, like your thought process on the all we've been doing the whole time is context engineering, right? We just like finally put a word on it. I think all of these is a really interesting thing, to have been in the space long enough to kind of go from, like when we thought LSTMs were good all the way to now that, you know, we're, we're very deep in not just like transformer architecture, but kind of like transformer architecture, right? 00;10;39;06 - 00;11;01;06 Unknown Like the hybrid models. You know, we're we're we're actually, like, really playing around with it, but like, we've always been doing context engineering, right? Like whatever, whatever the prompt engineering boom, right happened. And people were like getting paid six figures to do prompt engineering. Right. That's just context we didn't like. We just we just finally got a good umbrella term because someone came up with, with a smart, with a smart word for it. 00;11;01;06 - 00;11;22;22 Unknown But that's all it is, right? It's like everything is context engineering because everything is context model has a context window. That's how it sees the whole universe, right? Like that's the only thing that the model has to interact with the world is the context window. And so it's all context. We're, engineering all the way down. I think you named it the first class infrastructure. 00;11;22;23 - 00;11;48;18 Unknown Yeah. Before. So what do you see? Or how do you see, for example, some of Nvidia's customers, deploying that idea successfully in production. What do you think is most important if I'm a builder, are listening to you right now yapping about context engineering. What? What is, you know, what is your best practice advice? The reality is, and it's like kind of a cop out answer, but I think it's the real answer, which is you have to learn the model. 00;11;48;21 - 00;12;05;15 Unknown And that means, like playing with the models. So it my I yeah, I know it's not like tactical advice is a little bit cringe, but the reality is you have to put the model in whatever system you're trying to use it in and like, just play around, learn the experience of the model back in the day. 00;12;05;15 - 00;12;25;02 Unknown Like, I think we we've gone through waves of this, and I've personally gone through three waves of this of like, well, product engineering like is just kind of vibe is based like really engineering, you know, and what I'm suggesting now is kind of vibes based, not really engineering, but like, it's going to help you engineer something when you need to write. 00;12;25;02 - 00;12;43;22 Unknown Like, if you understand how the model works or how it reacts, like specific changes in the prompt or changes in like how you define tools or changes in whatever, whatever it happens to be. They're working with the memory system like you. You add some kind of like time waiting to it. And you, you feel the experience, you know, of that of that system. 00;12;43;22 - 00;13;05;23 Unknown It's going to help you, make actual engineering choices in the future. And so that's got to be the step number one. I feel like a lot of people build systems and like, they don't just use them, like themselves to get that to build that intuition. And as models get more and more complex and can do more and more things, I think it's like that's becomes more important because we're all we're going to have is intuition. 00;13;05;23 - 00;13;25;00 Unknown Like, I like, I've met you, right? There's no way for me to engineer you, you know what I mean? Like, there's no way to to to engineer a human being to do exactly what you want. You can, like, work with someone, right? To understand how they tick and like, you know, know how best to interact with them and know the best way to make an ask this kind of thing. 00;13;25;00 - 00;13;43;22 Unknown But like, there's not like, a set of explicit instructions that you can use every time that like, let's you just get the humans to do what you want. And I think we're we're entering the same point with models where is is very intuition based. And, another good practice thing, though, is like you should be leveraging search, for context engineering systems. 00;13;43;22 - 00;14;02;01 Unknown This includes for memory. We want to limit what we shove in context. There's like a business term that's as much as needed, as little as possible. I think for context engineering, that's still true. We should have exactly only the tokens that we need. And exactly. No more or no less than that. And so every decision you make should, should keep that in mind. 00;14;02;03 - 00;14;21;19 Unknown We don't want to. Just as tempting as it is, we don't want to just dump everything in context and pray. Yeah, that makes a lot of sense. Let me dig a little bit deeper, because I also found you in one of your, and mega speakers talk. You talked about, you know, rack being necessary retrieve, augment to generation being necessary, which often is tied to like a vector database. 00;14;21;21 - 00;14;44;24 Unknown You do semantic search and you optimize, obviously, for other putting in context that is not part of the training data or, finding, you know, other, more relevant results. But then there's also famous examples like, for example, Cloud Code, where as the creator stated on ECS, for example, that publicly, we know that they're not using a vector database behind cloud code. 00;14;44;24 - 00;15;05;29 Unknown So, what is your argument? I have the greatest, copout answer in the world here. So we've always been very clear, especially in my work at AI makerspace, of this idea of like, there are foundational patterns in, AI engineering, or what we used to call like, machine learning engineering. And one of those patterns is retrieval, augmented generation. 00;15;05;29 - 00;15;30;25 Unknown And while it's true that when people hear because of a lot of really fantastic marketing, when people hear, retrieval, augmented generation, there is the assumption that there's a vector database, or some other maybe a sparse vector retrieval pipeline, whatever it happens to be. Right. But the reality is, is like, is the pattern of retrieving something and then using that to augment our context window in order to generate a response. 00;15;30;25 - 00;15;59;19 Unknown There's no name vector in Rick, right? Yeah. Yeah. Exactly. Yeah. Like it's like it's it's just such a powerful pattern. I think we need to as a field like focus more on the foundational patterns as opposed to the specific technological implementation of the patterns, because we get kind of locked into a mindset when we when you say rag and everyone assumes you mean I mean, if you look at the original rag paper, like the the image people conjure into their minds when you say rag is not the same as that, that paper even. 00;15;59;19 - 00;16;21;23 Unknown Right. And I think that's that's the thing. And if you use grep with an agent to get context to put it into the context window to generate a great response, like you could make yourself, you can convince yourself you're not doing rag or you or you can convince yourself you are doing rag. But certainly the the pattern that you're applying is, is fundamentally, a very, very useful and very, very powerful one. 00;16;21;25 - 00;16;45;06 Unknown So where would you say that, a vector I use, the usage of a vector database is still meaningful. Listen, we have, petabytes, exabytes, whatever it is of structured data. We have a lot of situations, especially in big enterprises. But but not exclusive to them. Like getting a document that's pretty close to what you said is useful. 00;16;45;12 - 00;17;08;28 Unknown Vector databases. While they may not be the best tool for all cases, and there are even good arguments to say that they are there are suboptimal in a lot of cases. Now, there's still usage that that is important with vector databases. Some of the issue of, vector like dense vector retrieval is that it's kind of slow. 00;17;08;28 - 00;17;35;26 Unknown CVS IVR yeah, and we're in like two Nvidia shirts right now. Right. So CVS is like, GPU accelerated. Vector search I think like that's that helps smoothen some edges out. If someone were to ask me, I have like 10,000 PDFs and I, I want to, build a chat, but I'm almost certainly still saying like, yeah, hook it up to whatever vector database and, have that be the place that you start. 00;17;35;26 - 00;17;59;04 Unknown And then if that's insufficient, you should start reaching for for more complicated tools. Interesting, because there was such, so much engineering that went into tools like, for example, cursor that build very complicated retrieval systems. Merkle trees, and, you know, all kinds of algorithms to improve their vector search because they're using something like puffer. Right. And then you have a tool like Cloud Code taking the market at a sweep. 00;17;59;09 - 00;18;20;02 Unknown Now these days, I most serious developers that I know are using cloud code and they're not using cursor anymore. Why do you think that is? You know, I can't really speak to to what other people saying, but I'll tell you what I think, which is that these tools, especially right now, really come down to like a vibe, which is that like, do you like the vibe of the tool? 00;18;20;02 - 00;18;42;11 Unknown And it is the case that, you know, different strokes for different folks kind of thing here. But the idea is like something like cursor, at least to me, is still useful tool, especially, by the way, for like anything to do with a Jupyter notebook, which is if you're in the space of like creating technical assets that you want people to learn from, still are really important, like format, to to deliver those technical assets. 00;18;42;11 - 00;19;13;28 Unknown And like cursor was pretty hard on notebooks. Yeah. But like it comes down to the vibe and the ergonomics of the tool. One of the things that we saw I know, I know, you guys have been the, been some of the biggest observers of this phenomenon is like, it's I think more lightweight to spin up eight instances of cloud code to rip through a bunch of, work than it is to set up eight instances of cursor and do the same thing I would Chris has got a CLI now, so, you can, you can, you can still do that. 00;19;13;29 - 00;19;30;00 Unknown That's like the idea. It's a viable. Do you like it for me? I like basically every tool because they all are good at different things. It's less than it used to be, to be clear. But, their projects, like I do want to be in the IDE with the code, well, while the agents are tearing away at it. 00;19;30;00 - 00;19;48;21 Unknown But, you know, there's there's there's a time for each tool, especially when it comes to something like Cloud Go versus Codex. Right. Like there's there's a like if I'm looking at a problem, I can I can tell within the first like 2 or 3 minutes is, is this going to be a codex situation or a cloud code situation. 00;19;48;21 - 00;20;15;16 Unknown Right. So which ones which and any time I know exactly what I want, and I'm confident I can communicate it explicitly and neatly. And I just need that thing done. Exactly. Codex is my guy. And any time I need the model to understand what I mean, I use cloud code. You know what I mean? I don't know what it is, but cloud as a model is just super good at, knowing what you mean. 00;20;15;18 - 00;20;37;18 Unknown It's it's great at, determining intent, in a way that, for me, resolves to to the correct intent. And though I, I'm saying like Codex and and cloud code, obviously like open code or kilo code or whatever, it doesn't matter. Whatever shell you use, you know, whatever, two years, I think is, is less important than, you know, the vibe of that Tui. 00;20;37;21 - 00;20;58;28 Unknown So you can apply it to the wrong problem. What does your setup look like at home when you're tackling one hot. Take care. Okay. The greatest invention of the modern age is not cloud code or codex or, you know, these these PhD level models or whatever. It's, tiling window managers where you can just, like, neatly tile all of your cloud code instances or codex and whatever do you use, right. 00;20;58;28 - 00;21;18;03 Unknown Like that has been a godsend, I completely agree, I love it. Yeah. Like there was a, there was like a week before because I, I'm, this is a little bit coerced, but like, I'm primarily on, Windex, windows machine for my developer station. And then I, I asked this agent to like, the computers that do work. 00;21;18;10 - 00;21;41;22 Unknown Right, but I just like the the windows ecosystem for, for, for everything else. And so it's like a little bit of a pain to get a title. I want a manager to work well for windows, but you can do it. And you should do it, by the way. And yeah. So it's like there's I have two vertical monitors and then two horizontal monitors, and I just fill them up with squares of of whatever coding agent, I need for the problem. 00;21;41;22 - 00;21;58;07 Unknown And hopefully at the end of the day, I shot at least some of them down, you know, but, not a not a lot. Tmux is also your best friend. That's sad. That's great advice. Thanks for sharing that. Let's, switch gears a little bit. Let's talk a bit about your work at Nvidia, I think. Yeah. 00;21;58;08 - 00;22;24;11 Unknown People are interested to know what exactly are you working on? Yeah. So I'm a product research engineer, which is, which is like a fun title because it's just three different, like, functions in a row. It doesn't really it's not very self, just self descriptive. A big component of my job is, like traditional developer advocacy. That's something that I'm very happy to be able to do and something I know you're, familiar with, which is that, like, devs are really important. 00;22;24;14 - 00;22;46;26 Unknown Devs are also not like a monolith. There are many different kinds of developers, and we want to make sure we're talking to, devs to understand like what is their experience using our products, what is their experience with our ecosystem, so that when we're making a decision, someone in the room can speak for them, right. And that's that's something that I take extremely seriously. 00;22;46;29 - 00;23;03;18 Unknown And I'm very, very lucky that it gets to be part of my job. Right. Saying, you know, this feature is great, but, like, no one cares about that or, you know, or, like, this feature is great. And like, people have been demanding it for the for the last, for the last month or for four months, right. 00;23;03;22 - 00;23;29;17 Unknown Just making sure that devs feel like they have a voice in the company that that speaks, or is able to speak on their behalf, at least a little bit. I work with the neutron team very closely to, to help make sure that our models are good, which is, which is a overloaded term. I get the opportunity to, like, help them, but they are the masterminds and the orchestrators and the implementers. 00;23;29;19 - 00;23;55;12 Unknown So one like, fun. Meanwhile, I like to describe the role of product research engineering. Is that to product our research engineer to engineering our product research guy and then to, research our product engineer. Right. So, the, the the reality is that, I just synthesize, between a lot of extremely talented groups of people to, to try to tell a story across the whole, technology stack. 00;23;55;17 - 00;24;10;24 Unknown I mean, that sounds like a very fulfilling role, to be honest. Oh, yeah. It is a super fire. Yeah, yeah. I don't know how I got into this, lucked into, into that role. It's quite it's quite literally, at least for me, like the perfect role. So. And they let me do stuff like this for whatever reason. 00;24;10;24 - 00;24;31;23 Unknown So there you go. I think that I've chosen well. Well, thanks to. Yeah. Tell us a bit more about, neutron, the family, the model. What's special about it? Yeah, actually, yeah. I've been a big fan of neutron since I joined Nvidia. Even very slightly before I joined Nvidia neutrons. I actually, like, been around for a long time. 00;24;31;23 - 00;24;56;13 Unknown I think people just, like, have not carried a lot about it, until until recently. This is not because the the models weren't good, but I think it's because we just didn't really know how to talk about the fact that we were trying to build models like, and, there's a, there's a conception, or a perception, sorry, that, Nvidia is, just a hardware company. 00;24;56;13 - 00;25;14;10 Unknown I mean, it is a hardware company. And, at the end of the day, that hardware is, the most important thing that, that that we do. But to make the best hardware, you have to also make, really, really good software. Right. And this is not just because, like, it's good to make good software, though. It is. 00;25;14;12 - 00;25;37;24 Unknown And so that when you're testing your hardware, you are you have the conditions that allow you to explore with the limits of that hardware is. Right. So, Johnson, the CEO, always talks about this idea of extreme co-design, which is that you build a chip and then you use that chip to train a model. And then in training that model, you're gonna learn a ton about what that chip can do. 00;25;37;28 - 00;26;02;00 Unknown And then maybe where needs to change or improve. And then you can bring those learnings to the next layer of the stack. Right. Recently we've been talking about this thing called the five five layer cake. That is, I, look up the article if you want to learn more. But the idea is, we need to go from in these cycles from from metal all the way to end user for metal all the way to end user. 00;26;02;02 - 00;26;25;03 Unknown In order to design the best, most efficient, most, most powerful, hardware. So all of that to say, that's why I never tried. We had never tried. It was doing good numbers. It was actually quite a good model for synthetic, data generation. The first and you were trying to get the chance to work on was, 340 billion parameter model at that time was so huge. 00;26;25;03 - 00;26;43;28 Unknown Right. A dense model, by the way. So like, it's just a just a big guy. And then I think, you know, we we got a little bit better about talking about it. We got a little bit better about understanding what people were using models for and communicating what the message is, which is this idea of extreme co-design and this idea that faster models are smarter models. 00;26;43;28 - 00;27;10;20 Unknown And our researchers have just been cooking like you would not believe, to make these models go faster and even faster. And utilizing Blackwell to do thing cool things like Knvb for pre-training and what Neumann tried is now is, family of models. So, we're currently neutron three. Nano we released a while ago. Super. We just released, and Jensen just talked about ultra finishing pre-training. 00;27;10;20 - 00;27;33;22 Unknown We had some numbers up. It's it's it's doing very well for a pre-trained bottle. Do you want to be very clear to anyone watching you see numbers from the keynote about ultra. Keep in mind that that is just the pre-training. So it's not a finished model. So it's not going to have good results compared to something like, Kimmy K 2.5, which is a fully trained model. 00;27;33;23 - 00;27;56;25 Unknown Right. We are just looking at the pre-training based model before all of the post training stack, which is a an incredibly important part of training model. Explain a bit more why this part is so important. Yeah. I mean, the idea is like the base model needs to be good, but it isn't really good at the stuff that you're familiar with a model being good at. 00;27;56;25 - 00;28;18;01 Unknown Right? It's not good at following instructions. It's not super good at calling tools. It's, all of these things are things that we teach the model how to become very good at during post training. And the idea is, you know, we need a really good base model, so we need we need a good foundation, to to build a great model on top of, 00;28;18;03 - 00;28;42;08 Unknown And we want to make a really good base model that's open, you know, and that's why we've released a base checkpoint for Nano and Super as well. Right. Because if you want to build your own model, pre-training is like the most expensive part. It takes the longest. It is, is extremely resource intensive. Right. And then the post training stack is, is is not. 00;28;42;11 - 00;29;03;07 Unknown It's certainly not inexpensive. But but it is, more affordable than, than pre-training. So we want to have a base that people can train whatever their favorite model is. Right. But what happens is that when people hear there's a new model, they think like, oh, this is going to be like that better than Kimi. 00;29;03;07 - 00;29;23;18 Unknown And it's just like, well, well, what? The post training team finished the jobs. You know, for now it's just the base model. It's it's like the, the, the lump of clay that you can mold into whatever you want, but it's really high quality clay. Yeah. Not. That's great. You also did a lot of work. At least you published quite a few opinions on, evaluation. 00;29;23;18 - 00;29;43;08 Unknown Oh, yeah. Yeah, yeah, man, it's not a good time out there right now. Evals are super hard. How do you how do you evaluate a human being? Right? Like, it's very hard. It's so hard that we've been trying to do it for as long as there've been human beings, which, by my calculations, like, quite a long time. 00;29;43;12 - 00;30;15;13 Unknown Yeah. In fact, for as long as we've been around. Yeah. And I'm not I'm not trying to say that models are human beings, but what I am trying to say is that as models gain more and more intelligence and their capabilities become closer and closer to, you know, the kinds of capabilities that you would expect a person to have, it gets, proportionally more complex to then evaluate them, because what is what is a good model look like when, there's some stuff that like, of course, it's just going to be able to do. 00;30;15;15 - 00;30;36;18 Unknown So how do we really, like, determine how do we fill in all the gaps? I like this, this idea of, like, benchmarks or tentpoles. Okay. If you if you imagine this, what we would love is for our models to have a really, like, a flat capability frontier, right? So no matter where I poke the model, it's as good as it is and everything else. 00;30;36;18 - 00;30;51;19 Unknown So, like my coding model, when I poke the coding ability, I want to see the number be high. And then I want the math ability to also be high. I don't want chemistry ability. I don't want the legal ability and I want the blah, blah, blah blah blah, right? But in reality, models have to be trained on stuff. 00;30;51;21 - 00;31;13;06 Unknown And so we have a very jagged, capability frontier, right? Where, the, the model is like maybe great at coding and great at math. And then it's not so good at, like, writing. You know, writing, writing a story. Right. Or like making beautiful poetry or whatever, whatever it happens to be. And so we need, tentpoles, right? 00;31;13;09 - 00;31;45;00 Unknown To prop that up. We need both a good way to train it and like, any way at all to evaluate what is a beautiful poem, right? Like, I mean, we have opinions about that. He had code Rabbit, as you know. I bet I bet you guys do. I have, contrarian thought, yeah, there's a lot of opinions on the internet, and rightfully so, I think in the community as well, that we should not be optimizing for a horizontal, you know, capabilities curve as you just described it. 00;31;45;00 - 00;32;06;28 Unknown Yeah. We should be training specialists. Oh, yeah. I agree, if you need the model to be good at one thing, you should certainly train it to be good at that thing. I think, though, we don't want that. A lot of people want not just because it reduces deployment complexity or whatever happens to be, but because they want the same. 00;32;07;00 - 00;32;30;07 Unknown Then they want to have confidence in the model, in whatever task they apply it to. Which is why even if and Team Greens publish a lot of papers on like, the future looks like a bunch of specialized swarms of agents, right. Even if that's true, we still want to know how to train. Like the perfect generalist. 00;32;30;09 - 00;32;50;04 Unknown You know, not AGI. Really. A lot of people are the company of said, and I'm sure we'll continue to say, like, we're not worried about AGI. That's like whatever. That that's a good side effect. One day if it if it even is defined. Well, yeah. But we do want to make sure that the model has that smooth frontier, right. 00;32;50;07 - 00;33;13;19 Unknown In this situations, we know the model is going to be in that the capability frontier is is smooth. I think it's I, it's correct to say that the ultimately we will wind up with specialist models. We have to. Right. Like why would you use a model that's 90% as good as a different model? Yeah. If you have to do one thing with the model, I need the model to change the color of that. 00;33;13;19 - 00;33;31;21 Unknown That, that token from blue to red. You know, I don't care if it, if I can count to ten. Yeah, I care it does that the best in the world. Right? So, yeah, I mean, there's also maybe something that people could get confused is with it. If you look, for example, at the architecture of neutron, you have this mixture of experts architecture. 00;33;31;21 - 00;34;01;11 Unknown How is that different from training a special model, having the mixture of experts already baked into the larger model? I was thinking about it this way. Where M0 is a really good efficiency tool. It also gives us like this, this really good capability to like, borrow experts. Right? For, for unseen tasks. So if I have like experts that are really good at a set of things, and I'm, and I'm doing something slightly out of distribution, I might be able to, like, borrow from one another experts expertise. 00;34;01;11 - 00;34;22;21 Unknown Right to, to get the task done. But at the end of the day, it is a unit of model that is good at a set of tasks. And so you have to train only as a generalist, even if in function it can behave like a specialist or perhaps like specific per query experience of the model is more specialist in nature. 00;34;22;21 - 00;34;48;04 Unknown What we what we hope is true with specialist models is that they can be smaller than the generalist model. Right. So, that we don't need as many parameters to be as good at one task as we do to be good at, 5000 tasks. And so that that's the advantage of the specialist, I hope at the limit is that they're just significantly faster than the generalist. 00;34;48;04 - 00;35;06;12 Unknown And if you only need to be good at ten tasks despite deployment complexities, it might still be better to use ten specialists rather than one generalist. Yeah. Let's talk about deployment and deployment complexity for a for a minute. I think a lot of people are hoping as models get smarter, that we can have them deployed more locally on hardware devices. 00;35;06;12 - 00;35;26;01 Unknown Right? Maybe even phones. Yes. What do you see that going? I mean, I'm a big believer, man. So, you know, I don't think it's a hot take. I think that is the future. I don't think we've even begun scratching the surface of how small and efficient and compact and wonderful we can make these models. 00;35;26;07 - 00;35;50;06 Unknown I really believe that we've seen this already. If I were to like, I. If I were to have you chat with ChatGPT on day zero. Right. Like it gives off the vibe of like our like kitty models that we can train in an afternoon. Right. Like literally Karpathy is speed running. You know GPT two level models in less than like three hours, right? 00;35;50;08 - 00;36;19;10 Unknown I haven't checked the leaderboard in a bit, so I don't know if he's proud of Sub2, but the idea is like, you know, we we are going to make models better, and we're also going to make, be able to pack more capability into smaller packages. I don't I do not think we're anywhere close to like a theoretical limit on that yet I do listen, there's some concessions to reality and physics that you have to to make, like we can't make. 00;36;19;12 - 00;36;48;24 Unknown It's unlikely, even in time, that we're going to make a GPT, 5.4, in Pro mode, level experience available in sub 100 milliseconds on your cell phone. You know, but, but most certainly, I, I believe strongly that, we will continue to make model smaller and that in two years we'll look back on, you know, some of these models and be like, I cannot believe. 00;36;48;26 - 00;37;20;23 Unknown Yeah. In what those up in an afternoon now. Right. So it's early Nvidia Nvidia's dedicated to this mission. Yeah. We have incredible quantization teams. We have incredible, you know, doing really cool stuff like, the Cascade models, doing really cool stuff, like, all of the teams that are working on trying to train the smallest, most capable models like, nano, the smallest of the nanos dropped this in this week. 00;37;20;25 - 00;37;40;21 Unknown For even more edge device, you know, experiences. So yeah, strong believer that we will continue to make smaller models better and faster. What do you think about, training challenges that we're having? I think you, you spend a fair amount of time, you know, generating synthetic data for training, doing for a lot of RL reinforcement learning. 00;37;40;24 - 00;38;01;01 Unknown Some people have made points that we run out of data, and that synthetic data will not be enough to unlock the next frontier, that we might even need to change the transformer architecture. Where do you stand on that discussion? Listen, here's here's the reality at NeurIPS. Ilya, of course, famously said that data is a fossil fuel, right. 00;38;01;04 - 00;38;19;21 Unknown That we're going to run out of data and all this other thing. I think SDG at this point is all but proven to be like the renewable energy equivalent. We can SDG our way out of a lot of what is a synthetic data generation. Sorry. Heading up with the, with this three letter acronyms. It's, it's a it's a bad habit. 00;38;19;24 - 00;38;51;08 Unknown But so, synthetic data is, able to to, like, like, really, cover a lot of the, the cases that we need now, is that to say to the to the broader point of your question, no, I don't think we need to change the architecture to to accommodate this. I don't think we need to, to worry we're going to run out of data, but certainly we're going to change the architecture, because there's no way this is the best, right? 00;38;51;10 - 00;39;13;00 Unknown There's no way we we got it exactly right on our first try in all of human history. I think we've done that, like, six times, you know what I mean? Like, we are going to continue to iterate no matter what, you know, even Nemo try and iterates a lot each generation. Like a lot. You know, in terms of the core architecture of the model, I don't think we're done. 00;39;13;02 - 00;39;32;10 Unknown And it's going to help us with data efficiency, and it's going to help us, you know, bridge some of these gaps people are worry about. But certainly I don't think we need we just will. We're we're humans, you know what I mean? We even when we get to what might be literally the most optimal model, I'm sure we will keep customizing for hyper specific niche situations. 00;39;32;10 - 00;39;54;20 Unknown Right. So what do you think about, the observability concern, trust and, yeah, governance as well as responsible AI is that. Yeah. You have on your mind as well? I'm so privileged to work at Nvidia for this, for this question. Right. Like, we open source almost every token. We, we care deeply about security and safety. You write about it a lot. 00;39;54;21 - 00;40;23;04 Unknown You can look at the data that trains the models to determine, if it meets your safety thresholds or what have you. Do I think it's literally perfect now? Of course, AA has not given us a perfect openness score yet. Artificial analysis. So there's work to be done. But when it comes to things like data lineage, when it comes to things like understanding what went into the model, which helps assuage some of the concerns that you're bringing up. 00;40;23;06 - 00;40;48;07 Unknown Just so lucky to be working at Team Green, where we're we're open sourcing a lot of this. You get to make the decision, right? I can only speak for me, write my opinion on this, but like, I don't want to make that decision for people, you know, like, trust me, bro, is not a strategy. You know, like, if you if you, if you want trust, you should be transparent and allow people to look through and trust you because you're trusted. 00;40;48;07 - 00;41;08;03 Unknown Right. Like I think that's a that's something that is so lovely about the culture at, at the company that I'm happy to work at, which is that everyone wants to be transparent and show people exactly what we're doing. Show people exactly how we're improving. Right. Show, show people exactly what fixes we have made to address concerns. 00;41;08;03 - 00;41;25;00 Unknown And we have a whole safety stack and a whole security stack that, of course the teams that, work on those things are, you know, they're they're doing incredible work. I'm super proud to, to be able to point to them in their work to say, like, we have some of the most transparent models that that exist right now. 00;41;25;00 - 00;41;54;05 Unknown And it just does a lot when you're trying to do something in a, in a way that garners trust. So a side of Nvidia and I mean, this is great, great work, that you're doing there for you trying to to family and all other open source models. I think the main concern that people have and we talked about AGI before already, right, is that the genetic systems that we build with these models have inherent biases that might even be manufactured for, you know, oh, not for the best for human and humanity or society. 00;41;54;07 - 00;42;14;21 Unknown Are you concerned I'm not a Duma? Yeah. I'm not, I'm not like, hyper acceleration test or whatever, you know what I mean? Like, I have a I'm Canadian guy, so I'm comfortable living in the nebulous space that is neither, specific. Yeah, but I feel like I have to make a choice one way or the other. 00;42;14;23 - 00;42;40;05 Unknown What I'll say is that, of course, there are things about, you know, bias that concern me. One of the things that I am concerned the, the most about is that it can be very difficult to detect bias, and it can be very difficult to know when you are being biased. And, the using models that have biases are going to leak into the way that people think. 00;42;40;05 - 00;43;08;23 Unknown And I think that potentially could, have long term, you know, impacts, when it comes to like, you know, 0.001% bias over one interaction is like nothing, but over like 10 million interactions is like a lot. Right? Yeah. Like, and, you know, it's just kind of like, I'm not, like, worried about it in a existential, you know, keeping me up at night way. 00;43;08;25 - 00;43;34;08 Unknown I do think it's important to continue to think about it and continue to mitigate. And that's a beautiful thing again, just to to pull the, the, the team green privilege card here. Like, that's the beautiful thing about building open models is that all of the people that I think need to help or are able to, even even if what you can do is look at data and say, here exists some bias, let me help you solve it. 00;43;34;08 - 00;44;01;23 Unknown Like that's hugely impactful and influential, right? Am I worried that every private company has everyone's best interests at heart at all times? Again, not in an existential way, but I think in the long run, even even those companies are incentivized to build models that, that help humanity and, that, that make us better. So I believe that at the limit, we're going to have models that that help us and make us better. 00;44;02;00 - 00;44;21;02 Unknown Yeah. Let's broaden the scope a little bit, because most of the we talked a lot about open source models, of course, most of the. Yeah, most of the open source models currently, AI coming from China. Most of them. Yeah. What do you think about the manufacturers that I'm on or like the models per se that are coming from there, which model maybe surprised you the most? 00;44;21;02 - 00;44;45;05 Unknown Which ones to you, the most most capable? And what do you how do you compare them to, for example, that Nemo Tron family? I think the, the benchmark numbers, are the easiest way to know how they compare on like a capability front. Sovereign AI is is something that's important. Countries should be building AI systems, and we're working with countries like, like India and Korea and, a bunch of others. 00;44;45;07 - 00;45;18;21 Unknown So, that China is also building, good models is, is, is something that you just have to expect, right? Yeah. The most surprising one probably is like, Jimmy Carter, KTLA five. It just makes like, really good websites, like, like it has, like, better taste than some of even the frontier models. Like, I think like, even like if I were to tell you like a clod code web template, like, I think most people would kind of if you've used something like the cloud models to build websites. 00;45;18;21 - 00;45;42;14 Unknown Yeah, we'll know what I mean. Like it has like a kind of samey feeling like structure and theme to it. Kimmy K 2.5, kind of like it's it's not always super different, but like, it generally builds very like novel websites, for, for each different use case. And that's something that I think is, it's not like it's not the most important thing in the world. 00;45;42;16 - 00;46;02;20 Unknown Yeah. Well, like, it's nice that the models are, like, developing some taste, you know what I mean? Like that. That was a love that. I would love to try it out, actually. I mean, I tried, Kimmy k 2.5, I think I don't know which size. I think 14 billion. Okay. For code. Yeah. I was trying it on the plane, back home. 00;46;02;23 - 00;46;34;07 Unknown Oh, yeah. Yeah, yeah, it's a Gemini. Yeah. Just, you know, coding away. And I was so impressed. Listen, every every model builder that's dropping open models right now is like, fucking pretty hard, like. Yeah. I gotta shout out the guys that, like RC and Prime Intellect and, you know, working with people like Datalog and, you know, of course, like Allen Institute for I like everyone who's dropping open models is like, in my opinion, whether you're, you know, one, landmass or a different landmass, right? 00;46;34;07 - 00;46;54;22 Unknown Like, everyone's contributing in a real way to, to the progress. And, and I think it's, there's something to be delighted by from every model. I have never used a single model and walked away saying, this is just trash, you know what I mean? Like, I always am like, okay, maybe it's not like, great for these tasks, but like, look at this thing that I did. 00;46;54;24 - 00;47;18;11 Unknown That's dope. You know, even some, like, weird models, lately, like, this is I'm not trying to roast anybody. So new, new research has, like, a bunch of, really cool models, but they're models for me. Maybe sometimes the capabilities are where I want them for, like, you know, set some side some tests, but, like, they're kind of, like, the funniest models that I've ever talked to. 00;47;18;11 - 00;47;38;11 Unknown Like, they have a very sarcastic sense of humor, which I think, like a lot of models just don't care about. And so, you know, like, even even, things that you wouldn't expect you could be delighted by from a, from a model like the people at new, new researcher cook it. Man is. Yeah. It's dope. 00;47;38;11 - 00;48;01;05 Unknown You can see, like, you know, having a funny model is good. It's good for the world. Right? So, yeah, there's a lot of very interesting stuff on Huggingface. I also. Yeah. The other day. Yeah, yeah. Oh, your face gets weird. Yeah, yeah, yeah, yeah. Let's talk finally about one of them. I think most interesting, for talking points, especially for us here. 00;48;01;07 - 00;48;22;07 Unknown Yeah. Co gen I think. Oh, yeah. So one of the most important parts for, generative AI use cases. Yeah. What do you see? How do you see open models catching up to to that? Because at the moment, I think the frontier is still held by the closed source models. Hot take hot here. Okay. Not really very lukewarm tech. 00;48;22;09 - 00;48;44;27 Unknown But, like, oh, models are very fast a lot of the time, especially like smaller models. So like nano, right. 30 billion, 3 billion active emoji model. So neo returns right now specifically, like it's really fast. So you can verify that code is correct. Yeah. Like I think people sleep on that fact. 00;48;44;27 - 00;49;09;27 Unknown Really like for, like, these sweeping, it for one shot in your code base, like a complex code base and some weird language. Right. Like I was going to cover some. Very sorry. Yeah. You know, PG 13. Let's go. So, the idea is like, if I, if I have a specific function that I need to write, I like Nano could do it 100% of the time. 00;49;10;00 - 00;49;34;23 Unknown It just might take like 200 generations when the model is so small and so fast, like 200 generations is less time than cortex, right? Or less time than like Codex takes or whatever. And at the end of the day, if I have, set of, tests, like I can get a 100% correctness from Nano and less time then, some of the frontier models. 00;49;34;23 - 00;49;56;25 Unknown So what? When people say catching up, I think they mean, like, at everything the whole time. But but even, like, like you guys, you know, I was very lucky to talk, in a session at GTC with, with, David, I think our VP of AI. So he's like, pretty, pretty chill guy. 00;49;56;28 - 00;50;17;08 Unknown And like, what we're building modern agent systems, like, we're not just building based off one model. And so if you have your coding agent and you have, the planner and the reasoner and these kind of things and the frontier models hands, a lot of the code, you can kind of determine complexity, like how complex is this feature, right. 00;50;17;10 - 00;50;41;29 Unknown It just like you would assign, non complex features to a junior dev. Just letting in a riff on it, man. You know, do I see coding models be able to catch up to the frontier? With was open 100% again, code is a solvable problem. We can literally solve code. It is, perhaps going to be difficult and take a long time. 00;50;41;29 - 00;51;02;12 Unknown And, maybe there are some there are some complexity, is there? But like, we can we can absolutely get to the point where the model always produces correct code. So let's talk about those complexities, because for a lot of people might feel like we're already getting there. Now, when you look at things like base 44, you know, just have a vibe code applications. 00;51;02;12 - 00;51;30;02 Unknown Yeah. Well, what is the current challenge in the industry? You know, I don't think we're ready for the slot pocalypse or whatever that's coming, right? Like, I think, Karpathy tweeted about it, something that blew up on Twitter or whatever. But like, the reality is this is this is a necessary transition time, right? In code Gen's, evolution, where every challenge we have now is feeding back into the machine. 00;51;30;02 - 00;51;54;13 Unknown Exactly right. Like, you're, you're you're seeing companies training on, vibe coded projects. You're seeing people take that data and learn from it, try to understand how to mitigate some of the the weirdness that is vibe coding software. Right. And so again, it's like it's just a solvable problem. We see these vibe footed projects taking off and then getting refined over time. 00;51;54;18 - 00;52;16;23 Unknown Right. Like every, every time this has happened, we start with this kind of gross, complex mess. And then over time, engineers guide it back to even over complex mess. But, we we have that, you know, that state evolution from like, absolutely not okay to okay ish. Yeah. And so we can train models to do it better. 00;52;16;26 - 00;52;36;08 Unknown Right. We can train them to do that process. So I if I said the question correctly, I would say like it is a challenge. We got a lot of challenges, but we're also literally creating, software to help us solve the challenge at the same time with the same pile of data. So feel very confident that we're going to get there. 00;52;36;11 - 00;52;58;23 Unknown Yeah. And people like Code Rabbit. Not going to shill, but it is your podcast. We're in your house are working on that. Right. Like, if all code rabbit dead and I know that's not all Code Rabbit does, but if all it did was reduce slot by 10%, right. Like per PR, right. Like over time we would get to a system that didn't have much slop left. 00;52;58;29 - 00;53;19;02 Unknown Right. And I think that's, that there is this competing force in the ecosystem that's trying to hold the, the, the, the vibe coders at bay, or at least help them get to, to a better place means that we'll get to the better place, say, models that are producing slop for the ones we're fighting back. 00;53;19;05 - 00;53;40;17 Unknown They have equivalent capabilities. Should be. Should be possible. Yeah. I mean, it's an exciting time that we're living in. What do you see our role as software engineers developing into that? I don't know, you know, to be honest with you, I think if you asked me this question in 2025, I would have maybe had a very confident and incorrect answer, which was, of course, were necessary. 00;53;40;20 - 00;54;01;20 Unknown Yeah, of course we need to understand these systems. Of course we are integral to the process of writing code. But I think that's because I like writing code. Like, you know, I, I think it's because I enjoy the flow state of sitting at my keyboard and working through a puzzle, and using the tools that I have to do that. 00;54;01;20 - 00;54;26;22 Unknown Like, I like that. Right. And, liking something doesn't mean it's, like, strictly necessary. You know? It's just. You like it. I think we we remain important in the process because, especially right now, AI is not super good at coming up with ideas. AI does like, quote unquote taste, whatever that means to you. It's right. You know, but realistically, I don't know, man. 00;54;26;29 - 00;55;02;26 Unknown Like, I don't know where we sit in in five years from now when it comes to software engineering. Are we just, managers of teams of agents, like, are we still literally writing out, like, if statements? I, I don't think so. Right. But, like, I just don't know. Will we be will we achieve the ultimate nerd dream and, build a structured syntax that lets us, you know, write scaffolding for projects that looks our awful lot like code, right? 00;55;02;26 - 00;55;28;10 Unknown But really, under the hood is organizing teams of agents to do the work. Yeah. So that we're just coding at, like, a much higher, much more, you know, higher altitude, vantage point. Hopefully that would be fire. Oh, I wonder what that multi-screen experience was like. Yeah. Yeah, it's I dude. Exactly. I just don't know. I don't know, I'm hopeful that we'll continue to find, ways to make ourselves useful. 00;55;28;12 - 00;55;53;11 Unknown Humans have needed clever people to work on technical problems since we've existed. We're going to live in a world that's powered by, you know, technology seems to me, for the next forever. Right? And so we're going to continue to need people who are technically inclined and like, working on puzzles. And so we'll do something. We'll find something to do, even if it doesn't look like, what it looks like, this year or last year or ten years ago. 00;55;53;18 - 00;56;06;18 Unknown Perfect. You said I think that's a beautiful closing thought. So, close this conversation. Thank you so much, Chris, for taking the time. Thank you for for having with me today. Yeah, I know it's been a pleasure having you on. We'll see you on the next one. Thanks for joining in, everyone.