Lori MacVittie (00:02.839) Welcome back to Pop Goes the Stack, where the hype cycle is loud, but sometimes it points at something genuinely transformative. Really, sometimes. I am your host, Lori MacVittie, and we're going to separate the signal from the press release today. Joining me as sometimes, ehhh, is our co-host Joel Moses. Ha ha ha. Joel Moses (00:28.177) Ha ha. Yeah, yeah, sometimes-ish. Lori MacVittie (00:30.807) You left me alone for so long, I Joel Moses I am so sorry. Lori MacVittie you know, I just, I just missed you so much. Joel Moses (00:36.851) I know App World Grand Tour. Lori MacVittie That's right, that's right. He was on a world tour. But today we decided we wanted to tackle the all important question? I don't know, debate? I don't know. Anyway, you got AI, there's inference and there's training. And they're the same, but they're different. And they have all sorts of, right, Lori MacVittie (00:58.849) different terms that go with that and different requirements and we wanted to kind of dive into that because the model's not actually learning at real time. I know it remembers your name because it does, but it's not permanent, right? Once you close the browser, it goes away. Okay. It's only there for a while. So we wanna understand the difference between training, which actually teaches it things long term and it remembers, and inference, which is runtime and mostly in memory and quite frankly, it's inference. So to dive into this, we've got Ken Arora. Welcome, Ken. Ken Arora (01:37.821) Hey, welcome. Thanks for having me back. Lori MacVittie (01:40.196) Yeah, well, you know, you're gonna argue with Joel, so I'm all for that. I'm ready. And we've also got Kevin Baughman who runs our actually our AI Center of Excellence. So he knows a thing or two about this subject, which is why we brought him on to keep us all honest. So welcome Kevin. Kevin Baughman (02:00.541) Hey, thanks for having me. Lori MacVittie (02:02.357) Awesome. All right, so I'm just gonna kick this off because, you know, well, I think that's my job here. So, you know, you've got inference, you got training. Like, why does it matter that they're different and how does that impact, you know, the infrastructure, the compute, the stuff you buy? What do you have to have in place? What does it mean? Why do we need to know? Joel Moses (02:26.659) Yeah. Well, Lori, honestly, I don't necessarily think, at least from a mathematical perspective, that there's a lot of difference between training and inference. There's different amounts of math that you have to do in order to do the training. You have to do a backpropagation check, and that is expensive computationally. But if you look at the actual mathematics behind inferencing and training, they're identical. They're just done in different amounts and with different data sets incorporated. You know, training isn't about more math, it's about remembering things. And training, inference can discard like intermediate information or results after they're used, but training has to save every intermediate activation because it has to go and backpropagate. It has to go perform a whole series of checks. The metaphor that I like to use is the one about the piano tuner, which is that you can hear someone play a wonderful composition on a piano, but what led up to that piano being in tune was someone had to go and play a bunch of notes on it and then watch to see whether the notes were correct and correct for any flaws. And so the kit that they may have around that is different. The amount of time that they spend on it is different, but we only hear on the inference side the playing. We don't hear the tuning. Lori MacVittie (03:49.537) That's yeah Ken Arora (03:49.571) Well, wait a second. I was a physics major. Differences matter. Lori MacVittie / Joel Moses / Kevin Baughman Ha ha ha. Ken Arora I mean, fundamentally, we're all six quarks that are put together in different ways. Joel Moses (03:57.448) Mm-hmm. Ken Arora And you know my chemist friends will say we're all a bunch of atoms that are just put together in different ways. So absolutely I mean, at one level you're absolutely right, Joel, right? It is the same compute infrastructure. But at another level, because it's put together so differently, the differences matter. Ken Arora (04:13.286) Yes, you can buy GPUs and in fact you could say, "okay, you know, the GPUs are the same," but even there you see a split between hardware and silicon and ASICs and ASIC data paths that are optimized for one or the other, and that's a whole other podcast topic. But the high level, the high order bit here is that those differences matter. If I try to go into your piano analogy, you're absolutely, you know, you talked about this. The, yes, you're playing a piano and you've got this, I'll make it a digital piano because it might make the Joel Moses Ha ha ha. Lori MacVittie (04:46.113) Thank you, thank you. Ken Arora analogy a little better. Right, a lot of work goes into identifying what the sound is on that digital piano and making, you know, people in a test lab somewhere in Yamaha are working on it and speaker placement, all these great things, and that's a huge amount of time and investment. And then they take that piano and they Ken Arora (05:05.098) replicate it, make thousands of copies and ship them off to wherever. And yeah that's, shipping that piano is sort of like the model and training the model. They did that in lab and then you go off and you play it, and you can play it many different ways and that becomes inferencing. And you can play anything from, you know, hip hop to classical music on it. Joel Moses Sure. Ken Arora Right? And that's wonderful. But, and that's the inferencing part. And, yes, you're right, it's still piano, but the workflows around it, how you go about doing things, what you take as the input--the input for the composition is gonna be somebody playing jazz or somebody looking at sheet music in one case. In the other case, it's some labs and technician using acoustical equipment to see what a piano sounds like. So it's, in my mind, yes, both are pianos, but they're very different. Let's, Kevin, keep us honest. Lori MacVittie (05:53.772) Yeah. Kevin Baughman (05:54.095) No, I 100 % agree. The other thing that I keep in mind is when we're building these systems, for a while we were only thinking in the large language model world about training. Everything was focused on training. And then when it came time to start using these things and deploying them at scale for users, the thought process was very different, right? Versus one big environment versus a bunch of smaller ones. The other thing is how these things are used is training in inference, to me, has gotten blurred very much. Right now with large language models is because people are using context to change the behavior. It's almost like you're performing inference on different data, but that data contains rules on what you want the system to do. And it's ephemeral, right? It's a short-term learning. But occasionally we can get these systems into states where we can never reproduce it again. So it's really, how do we think of Lori MacVittie (06:48.301) No. Kevin Baughman inference in training? Kevin Baughman (06:49.272) Training to me always has been frozen weights. Inference always has been deterministic behavior. And that's all been flipped around. Joel Moses Yeah. Lori MacVittie (06:58.807) So what, but what does that mean? I mean, you know, when you're training, we associate tons of data, like high volume, right, just big chunks of data. So obviously the security around that coming in is going to be different than you know, protecting, you know, from prompt injections. These are two different processes, correct? Joel Moses (07:21.799) Well, to Lori MacVittie (07:22.037) Everybody agrees we're done. Joel Moses some degree. Lori MacVittie That's it. Alright. Kevin Baughman Ha ha ha. Joel Moses To some degree. I mean Kevin Lori MacVittie We're out. Joel Moses Kevin brings up a very interesting distinction, which is that the ever increasing context window means that you're saving a lot more intermediate state and you're reusing it as part of the inference chain, which is more work. I mean it involves more work. It's a little closer to doing the math that you would do if you were creating a model from scratch. Although the scale there is tremendously different. I mean Joel Moses (07:50.031) your training sets are much, much larger than your intermediate context window sets are. Lori MacVittie (07:56.163) Where is that stored? Where are we storing context windows? Joel Moses (07:59.495) Oh, Lori MacVittie Maybe that's, Joel Moses memory. Lori MacVittie you know. Joel Moses We're storing it in some Lori MacVittie Right. Joel Moses memory somewhere, right? Lori MacVittie Well, Joel Moses So, you know, a training Lori MacVittie memory's ephemeral. Joel Moses set is going to... Correct. But a training set, if you're doing a training run, you know, you're talking about, you know, using up to forty gigabytes of memory, you know, attached to your fast GPU complex in order to just hold in state all the things you're gonna need for the gradient. And so that's a lot. But we're not Lori MacVittie (08:28.941) Yeah. Joel Moses really accepting forty gigabytes worth of context window information. That would be, that would be nutty Lori MacVittie (08:34.251) Insane. Ken Arora (08:35.383) Yeah, so Joel Moses and it would drain your bank account fairly quickly. Ken Arora (08:39.331) Yeah. So think about it this way, right? Training bakes things in. It's like baked into the model, baked into the piano you shipped are all these things. The context window is more about a certain level of maybe how I might place the piano in the room to get good acoustics. There's another level of this, which Kevin will probably bring up in a minute, which is fine-tuning. And I, the reason I mention the digital pianos, right, the piano may come, you may be able to fine-tune the piano yourself. It might come with certain presets, right, to say, I want it to sound like a concert piano versus an upright versus, you know, an organ. And then the really fancy digital pianos, you can even upload your own waveforms, your own description of like I want to tweak it a little bit. It should sound a little like this particular style of upright piano or baby grand. And you can tweak that. And that's, and I'll hand it to Kevin here, right? That's kind of like fine-tuning. I'm gonna try to work with Joel's piano analogy here. Joel Moses Ha ha ha. Ken Arora And I apologize to Kevin Baughman (09:39.864) Hehehe. Ken Arora musicians here Lori MacVittie (09:40.578) Keep it going. Ken Arora that we're really bad. Joel Moses (09:41.767) Yeah, Lori MacVittie Keep it going. Kevin Baughman Hehehe. Joel Moses I like to get my Kevin Baughman Hehehe. Joel Moses I like to get my analogies and my metaphors in first so that you know, Ken Ken Arora (09:46.317) Ha ha ha. Lori MacVittie (09:46.434) Yeah, so Joel Moses gets stuck to them. Kevin Baughman Yeah. Lori MacVittie we have to use them. Yes, now we're Joel Moses Right. Lori MacVittie 'cause it was trained in Joel Moses Ha ha ha. Lori MacVittie and it's baked in Joel Moses (09:51.347) Exactly. Lori MacVittie and now we all have to, Ken Arora (09:52.599) Ha ha. Kevin Baughman (09:52.76) Yeah. Lori MacVittie we have to work with it. Kevin Baughman Yeah, so fine tuning, and I'm not going to even attempt to talk about pianos, but Lori MacVittie / Joel Moses / Ken Arora Ha ha ha. Kevin Baughman fine tuning for us historically has been when the context windows were a lot smaller and now they're expanding. And what was happening was we were trying to pass too much information to the model at inference time. And yeah, we've backed away pretty heavily from fine tuning. There are cases in which we use it and there are people that do like to use it. But the idea behind fine tuning is, as Ken said, is you do freeze the weights. So now you're going to have reproducible determinist-, well, not deterministic, in LLMs you never get that, but you get something that is frozen to behave in a way in which you want it. We usually like to use it for very small tweaks. Massive behavior changes, you're probably using the wrong model or tools when you're leading into it. Lori MacVittie (10:47.704) So during inference do the weights ever change? Based on what I'm telling it or how I'm shaping it? Kevin Baughman (10:54.786) The weights are frozen during inference time. The context will, I like to think of it as Lori MacVittie (11:01.93) Influence. Kevin Baughman I'm performing, it influences it, but I'm actually performing inference on more data. Joel Moses Yeah. Lori MacVittie (11:08.685) Okay. Kevin Baughman If I was to try to take a system and I wanted to pass in a fruit and have it classified as a fruit or a car and have it classified as a car, if I say apple to an LLM, it's gonna say something crazy. Kevin Baughman (11:23.018) If I was to give it a few shots, what we call few-shot inference, which a few examples of like apple equals fruit, banana equals fruit, like Hyundai equals car, the system will learn those rules, but you're performing inference on that whole thing. So what you're performing inference on is actually different. People think of that as learning. And it is clever because activation layers allow you to look back at prior data to influence your future results. Joel Moses Right. Kevin Baughman Hopefully that makes sense. Ken Arora (11:50.54) Right. Right. It's Lori MacVittie (11:52.248) No. Ken Arora it's sort of like maybe I have a, you know, I'm talking to a five year old and they know something about the world, but they don't know what's called a fruit, versus say, I don't know, I'm gonna pick an animal, right? Go, "well, that's a fruit," and you point at the banana and an apple and so on. And then you say, "Well, these are animals," you point to the dog and the cat, and you go, "These are animals and these are different." And they'll generalize off of that. Ken Arora (12:19.423) But unlike, an LLM is a little bit unlike a human in the sense that a human will get that from the example and then over time they'll bake it into their model. We don't quite have that yet, but people working on it. Joel Moses Mm-hmm. Ken Arora But it Lori MacVittie (12:34.936) So we're fine tuning children. Ken Arora but it's sort of like what's hard-coded, what's baked in, the fixed weights? Joel Moses Yeah. Lori MacVittie We're fine tuning... Ken Arora What's in the piano you get from Yamaha versus Ken Arora (12:42.731) what are you tweaking by where you're placing the piano and maybe what instruments you're placing next to it? Lori MacVittie (12:51.416) I feel like we're approaching the okay training is an apple but then we get an orange and that's inference and Joel Moses (12:56.776) True. Kevin Baughman Ha ha ha. Lori MacVittie and so it's different it's, Ken Arora (12:59.458) Yeah. Lori MacVittie you know, but so what, you know, you're talking lots of memory. You need it in both, you need GPUs for both. Obviously when you're training you have different--and I hear this word a lot--pipelines that you don't have with inference, right? So what do you need in those pipelines? You have lots of data you have lots of information, Lori MacVittie (13:19.608) you have different security needs, so what makes up a pipeline, Kevin? You've built them. So what goes into a pipeline? Well, I hear you've built them. I could... Kevin Baughman Ha ha ha. Well, yeah. Lori MacVittie Right? Kevin Baughman (13:26.766) Yeah, I mean, for us, it's tons of compute GPUs and tons of data. Now, the Lori MacVittie (13:36.781) That's it. Kevin Baughman security needs of us is we like to think of security as that if I didn't train something into a model, I don't have to worry about it saying it later. Right? So when these LLMs are trained, they're trained on massive amounts of data and Kevin Baughman (13:54.147) you ask it questions and it answers inappropriate things. If, you know, we like to make our system so that it's never been trained on that stuff. So Lori MacVittie So, Kevin Baughman we have to worry about it coming less. Joel Moses Yeah. Lori MacVittie (14:01.934) So the choice of ingesting Reddit was basically the first problem. Kevin Baughman Ha ha ha. Lori MacVittie Like way back that's where it all started. That makes, but that makes sense, right? Controlling the data coming in is actually a security issue. Like don't put top secret stuff in there because it'll spit it out later. I mean it should be obvious, but maybe it's not. So controlling the data is actually a technique. And then on the reverse, it's Lori MacVittie (14:31.456) also controlling the response, which is data coming out. So you're kinda doing the same things but in reverse, isn't it? Kevin Baughman (14:41.39) Yep. Joel Moses (14:42.033) Yeah. Now, Kevin Lori MacVittie (14:42.232) There you go. That's it. Joel Moses Kevin's Ken Arora (14:42.913) Yeah, and that is Joel Moses Kevin's Ken Arora that is where you, you hit it, right, you hit the nail on the head. This is one of the key differences between inferencing and training. Training, you're baking it in. It's in the piano, it's in the model. And it might be hidden, it may be hard to pull out, but it's in there somewhere. Whereas with inferencing, you're saying that's not in there. If I want to, if I have some proprietary data that I don't want to share and I don't want to be baked in the model, but it's still useful to you then I'll do the one shot, the few shot learning, the expanded context window that Kevin talked about to give it that information. So that's an important, you know, an important reason why you might pick one over the other. I need you I need my model to be good at, I don't know, spotting Russian nuclear silo sites, but I don't want it to be put into the model. So instead what I'm gonna do is I'm going to take a model off the shelf and I'm gonna then give it examples of this is what a silo site looks like Kevin Baughman Right. Ken Arora and let it then say, okay, now that you've give me some examples, I can find others. Joel Moses Yeah. Kevin Baughman (15:46.959) Right. And it can be very powerful by attaching it to a knowledge base. And now you can change that knowledge base. You can remove things from it. You can add things to it. The other thing, it allows us to deal with different tenants in the system. So if I want information specifically for Joel about pianos and then Ken about fruit, Joel Moses Ha ha. Kevin Baughman I can have the knowledge base contain relevant information. And that allows us to keep stuff that should be secret out of individual deployments. Joel Moses Right. Lori MacVittie (16:16.632) Right. Well, and I think part of the reason that people started believing or at least have gotten the idea that during inference they can learn is because of the systems around it. When you look at something like ChatGPT, there's also a lot of session data, it saves your conversations. That's not the model. The model's not doing that. That's the systems outside of it, the server and the, Lori MacVittie (16:41.218) you know, everything that goes along with it. And it's starting to sound like it knows you more because it's building more of that context and sharing it every time so it knows, hey, this is who this is. This is who've been talking, these are the key points, right, remember these five things. Right? Joel Moses Yeah. Lori MacVittie Lori likes this. She has fish. She doesn't like dogs. You know, it should remember this, but it doesn't, right? If the entire system went away, the model would forget everything. And I'd have to start over rebuilding that context, right? Joel Moses (17:15.325) Yeah. Now there there is a difference in we've been talking about pipeline and I think that's a good discussion. What we're talking about is an ingestion pipeline for the data for the data set. Now once you get inside the ingestion, you also have a distribution pipeline to take this work and split it apart and do your normalization passes and feed it in to GPU processing, et cetera. And that is also where some differences between training hardware and inferencing type hardware exist as well. The memory needs are much higher on the training side, but also most importantly, the amount of memory bandwidth that you have to feed things into memory and clear things from memory, the faster the memory is, the better. On the inferencing side, you can take a little bit of delay. It's always best if you can get the fastest memory possible, but it's actually less critical. And so if you're looking to save money, you can use lower speed memories with inferencing, but that's the really the only difference in the two pipelines. Ken Arora (18:20.099) Yeah and Joel Moses So, like on the training side, HBM memories are a great place to keep this training data because you're doing a lot of operations and it's about loading and unloading and doing these atomic operations. On the inferencing side, you're basically loading once and you've got an intermediate set and you discard it after you're done. Ken Arora Yep. Joel Moses And so the the memory is different inside the processing pipeline. Ken Arora (18:43.107) Yeah, and speaking as a former hardware person, it's really about that memory working set as Joel said, right? For inferencing, the memory you need is whatever it takes to hold the model, usually measured in double digit gigabytes. Not trivial amounts of memory, but not huge amounts of memory either. Right? You need 10, maybe 100 gigabytes of memory to hold your model. And once you got that, getting more memory is not really all that useful. Compare that to the amount of data that you're training on, which is, you know, petabytes of data. And so you're never gonna keep that right next to your silicon GPU. So that memory bandwidth getting that, those petabytes of everything on Reddit in and out, that's your bottleneck. Joel Moses (19:26.195) Yeah. Lori MacVittie (19:27.832) Well, Joel Moses So from a design perspective, you definitely have to take all of those things into account. So and it's in multiple places. So you've got the ingestion pipeline, you've got the processing pipeline, you've got optimization functions, the whole nine yards. You have to look at each step of the pipeline and choose exactly the right components that give you the most efficient and cost effective way of processing. And that's where the major difference is in inferenc- between inferencing and learning. Joel Moses (19:54.804) Although the training and inferencing have the same mathematics behind them, Lori MacVittie Ha ha ha. Joel Moses the actual hardware is different. I realize I've come around in a full loop there. Kevin Baughman Ha ha ha. Lori MacVittie (20:00.078) You have. You ha-, I mean that's and that's okay 'cause you were so close to like Joel Moses (20:09.942) Ha ha. Lori MacVittie the perfect like takeaway. It was right there, right? Kevin Baughman Ha ha ha. Joel Moses (20:13.842) Yes. Lori MacVittie Like it's different compute. You have to identify the steps and get Lori MacVittie (20:17.656) put the right things and then you right back to the math is the same though. Well of course, like one plus one is two, no matter whether I do it, a five year old does it, we do it on an abacus, Joel Moses (20:27.27) Mm-hmm. Lori MacVittie we count it using M&Ms, or the computer does it. The math is the same. That should be evident, Joel. Joel Moses (20:33.393) Yes. Lori MacVittie It's irrelevant. That's what I'm saying. Joel Moses (20:36.945) Yes, but it let me talk about pianos for a half hour, so I'm good. Kevin Baughman (20:37.88) Ha ha. Lori MacVittie Okay. Well that's Lori MacVittie (20:40.918) true. That's true. I think that is our takeaway. We are way over time. Right, is identify the stage that you're working at. Look at what you need for each in terms of components and put the right things in place to optimize each of these phases because they're both important. I mean you don't train something to not use it then in inference and you can't inference if you haven't trained. So they kind of go together like peanut butter and jelly or pianos and Joel. Joel Moses (21:11.015) Exactly. Lori MacVittie So, that's right. Well, thank you all. That a wrap for Pop Goes the Stack. Please subscribe, so when the good stuffs arrive, you'll recognize it even without a keynote.