Subscribe
Share
Share
Embed
Training and inference get lumped together in casual AI conversations, but they behave differently enough that the distinction matters for cost, architecture, and security. In this episode of Pop Goes the Stack, Lori MacVittie, Joel Moses, Ken Arora, and Kevin Baughman (who leads F5’s AI Center of Excellence) unpack what’s truly different, what’s the same, and where people get misled.
Joel makes the “math is the same” case: both phases run similar computations, but training must retain intermediate activations for backpropagation, while inference can discard them. Ken and Kevin pull the conversation back to practical differences: training is about baking knowledge into the model, while inference is about using a frozen model and shaping behavior with context, retrieval, and few-shot examples. The weights don’t change during inference; the input does, which is why it can feel like “learning” without actually being permanent.
That distinction becomes a security and governance lever. If you don’t want sensitive or proprietary data baked into a model, you avoid training on it and instead keep it in a controlled knowledge base (RAG or similar) that can be updated, removed, or scoped per tenant. Meanwhile, training pipelines emphasize massive data ingestion and throughput, and inference emphasizes responsiveness, session context, and efficient serving at scale.
The practical takeaway is to stop treating “AI workloads” as one thing. Training and inference require different pipeline designs, different tradeoffs in memory and bandwidth, and different approaches to data control. Pick your phase, understand the constraints, and build for it intentionally.
Creators and Guests
Host
Host
Lori MacVittie
Distinguished Engineer and Chief Evangelist at F5, Lori has more than 25 years of industry experience spanning application development, IT architecture, and network and systems' operation. She co-authored the CADD profile for ANSI NCITS 320-1998 and is a prolific author with books spanning security, cloud, and enterprise architecture.
Guest
Ken Arora
Ken Arora is a Distinguished Engineer in F5’s Office of the CTO, focusing on addressing real-world customer needs across a variety of cybersecurity solutions domains, from application to API to network. Some of the technologies Ken champions at F5 are the intelligent ingestion and analysis of data for identification and mitigation of advanced threats, the targeted use of hardware-acceleration to deliver solutions at higher efficacy and lower cost, and the design of user experiences based on intent and workflows. Ken is also a thought leader in the evolution of the zero trust mindset for security, and how that will be applied to increasingly distributed and even edge-native apps and services. Prior to F5, Mr. Arora co-founded a company that developed a solution for ASIC-accelerated pattern matching, which was then acquired by Cisco, where he was the technical architect for the Cisco ASA Product Family. In his more distant past, he was also the architect for several Intel microprocessors. His undergraduate degrees are in Astrophysics and Electrical Engineering, from Rice University.
What is Pop Goes the Stack?
Explore the evolving world of application delivery and security. Each episode will dive into technologies shaping the future of operations, analyze emerging trends, and discuss the impacts of innovations on the tech stack.
Lori MacVittie (00:02.839)
Welcome back to Pop Goes the Stack, where the hype cycle is loud, but sometimes it points at something genuinely transformative. Really, sometimes. I am your host, Lori MacVittie, and we're going to separate the signal from the press release today. Joining me as sometimes, ehhh, is our co-host Joel Moses. Ha ha ha.
Joel Moses (00:28.177)
Ha ha. Yeah, yeah, sometimes-ish.
Lori MacVittie (00:30.807)
You left me alone for so long, I
Joel Moses
I am so sorry.
Lori MacVittie
you know, I just, I just missed you so much.
Joel Moses (00:36.851)
I know App World Grand Tour.
Lori MacVittie
That's right, that's right. He was on a world tour. But today we decided we wanted to tackle the all important question? I don't know, debate? I don't know. Anyway, you got AI, there's inference and there's training. And they're the same, but they're different. And they have all sorts of, right,
Lori MacVittie (00:58.849)
different terms that go with that and different requirements and we wanted to kind of dive into that because the model's not actually learning at real time. I know it remembers your name because it does, but it's not permanent, right? Once you close the browser, it goes away. Okay. It's only there for a while. So we wanna understand the difference between training, which actually teaches it things long term and it remembers, and inference, which is runtime and mostly in memory and quite frankly, it's inference. So to dive into this, we've got Ken Arora. Welcome, Ken.
Ken Arora (01:37.821)
Hey, welcome. Thanks for having me back.
Lori MacVittie (01:40.196)
Yeah, well, you know, you're gonna argue with Joel, so I'm all for that. I'm ready. And we've also got Kevin Baughman who runs our actually our AI Center of Excellence. So he knows a thing or two about this subject, which is why we brought him on to keep us all honest. So welcome Kevin.
Kevin Baughman (02:00.541)
Hey, thanks for having me.
Lori MacVittie (02:02.357)
Awesome. All right, so I'm just gonna kick this off because, you know, well, I think that's my job here. So, you know, you've got inference, you got training. Like, why does it matter that they're different and how does that impact, you know, the infrastructure, the compute, the stuff you buy? What do you have to have in place? What does it mean? Why do we need to know?
Joel Moses (02:26.659)
Yeah. Well, Lori, honestly, I don't necessarily think, at least from a mathematical perspective, that there's a lot of difference between training and inference. There's different amounts of math that you have to do in order to do the training. You have to do a backpropagation check, and that is expensive computationally. But if you look at the actual mathematics behind inferencing and training, they're identical. They're just done in different amounts and with different data sets incorporated. You know, training isn't about more math, it's about remembering things.
And training, inference can discard like intermediate information or results after they're used, but training has to save every intermediate activation because it has to go and backpropagate. It has to go perform a whole series of checks. The metaphor that I like to use is the one about the piano tuner, which is that you can hear someone play a wonderful composition on a piano, but what led up to that piano being in tune was someone had to go and play a bunch of notes on it and then watch to see whether the notes were correct and correct for any flaws.
And so the kit that they may have around that is different. The amount of time that they spend on it is different, but we only hear on the inference side the playing. We don't hear the tuning.
Lori MacVittie (03:49.537)
That's yeah
Ken Arora (03:49.571)
Well, wait a second. I was a physics major. Differences matter.
Lori MacVittie / Joel Moses / Kevin Baughman
Ha ha ha.
Ken Arora
I mean, fundamentally, we're all six quarks that are put together in different ways.
Joel Moses (03:57.448)
Mm-hmm.
Ken Arora
And you know my chemist friends will say we're all a bunch of atoms that are just put together in different ways. So absolutely I mean, at one level you're absolutely right, Joel, right? It is the same compute infrastructure. But at another level, because it's put together so differently, the differences matter.
Ken Arora (04:13.286)
Yes, you can buy GPUs and in fact you could say, "okay, you know, the GPUs are the same," but even there you see a split between hardware and silicon and ASICs and ASIC data paths that are optimized for one or the other, and that's a whole other podcast topic. But the high level, the high order bit here is that those differences matter. If I try to go into your piano analogy, you're absolutely, you know, you talked about this. The, yes, you're playing a piano and you've got this, I'll make it a digital piano because it might make the
Joel Moses
Ha ha ha.
Lori MacVittie (04:46.113)
Thank you, thank you.
Ken Arora
analogy a little better. Right, a lot of work goes into identifying what the sound is on that digital piano and making, you know, people in a test lab somewhere in Yamaha are working on it and speaker placement, all these great things, and that's a huge amount of time and investment. And then they take that piano and they
Ken Arora (05:05.098)
replicate it, make thousands of copies and ship them off to wherever. And yeah that's, shipping that piano is sort of like the model and training the model. They did that in lab and then you go off and you play it, and you can play it many different ways and that becomes inferencing. And you can play anything from, you know, hip hop to classical music on it.
Joel Moses
Sure.
Ken Arora
Right? And that's wonderful. But, and that's the inferencing part. And, yes, you're right, it's still piano, but the workflows around it, how you go about doing things, what you take as the input--the input for the composition is gonna be somebody playing jazz or somebody looking at sheet music in one case. In the other case, it's some labs and technician using acoustical equipment to see what a piano sounds like. So it's, in my mind, yes, both are pianos, but they're very different. Let's, Kevin, keep us honest.
Lori MacVittie (05:53.772)
Yeah.
Kevin Baughman (05:54.095)
No, I 100 % agree. The other thing that I keep in mind is when we're building these systems, for a while we were only thinking in the large language model world about training. Everything was focused on training. And then when it came time to start using these things and deploying them at scale for users, the thought process was very different, right? Versus one big environment versus a bunch of smaller ones. The other thing is how these things are used is training in inference, to me, has gotten blurred very much.
Right now with large language models is because people are using context to change the behavior. It's almost like you're performing inference on different data, but that data contains rules on what you want the system to do. And it's ephemeral, right? It's a short-term learning. But occasionally we can get these systems into states where we can never reproduce it again. So it's really, how do we think of
Lori MacVittie (06:48.301)
No.
Kevin Baughman
inference in training?
Kevin Baughman (06:49.272)
Training to me always has been frozen weights. Inference always has been deterministic behavior. And that's all been flipped around.
Joel Moses
Yeah.
Lori MacVittie (06:58.807)
So what, but what does that mean? I mean, you know, when you're training, we associate tons of data, like high volume, right, just big chunks of data. So obviously the security around that coming in is going to be different than you know, protecting, you know, from prompt injections. These are two different processes, correct?
Joel Moses (07:21.799)
Well, to
Lori MacVittie (07:22.037)
Everybody agrees we're done.
Joel Moses
some degree.
Lori MacVittie
That's it. Alright.
Kevin Baughman
Ha ha ha.
Joel Moses
To some degree. I mean Kevin
Lori MacVittie
We're out.
Joel Moses
Kevin brings up a very interesting distinction, which is that the ever increasing context window means that you're saving a lot more intermediate state and you're reusing it as part of the inference chain, which is more work. I mean it involves more work. It's a little closer to doing the math that you would do if you were creating a model from scratch. Although the scale there is tremendously different. I mean
Joel Moses (07:50.031)
your training sets are much, much larger than your intermediate context window sets are.
Lori MacVittie (07:56.163)
Where is that stored? Where are we storing context windows?
Joel Moses (07:59.495)
Oh,
Lori MacVittie
Maybe that's,
Joel Moses
memory.
Lori MacVittie
you know.
Joel Moses
We're storing it in some
Lori MacVittie
Right.
Joel Moses
memory somewhere, right?
Lori MacVittie
Well,
Joel Moses
So, you know, a training
Lori MacVittie
memory's ephemeral.
Joel Moses
set is going to... Correct. But a training set, if you're doing a training run, you know, you're talking about, you know, using up to forty gigabytes of memory, you know, attached to your fast GPU complex in order to just hold in state all the things you're gonna need for the gradient. And so that's a lot. But we're not
Lori MacVittie (08:28.941)
Yeah.
Joel Moses
really accepting forty gigabytes worth of context window information. That would be, that would be nutty
Lori MacVittie (08:34.251)
Insane.
Ken Arora (08:35.383)
Yeah, so
Joel Moses
and it would drain your bank account fairly quickly.
Ken Arora (08:39.331)
Yeah. So think about it this way, right? Training bakes things in. It's like baked into the model, baked into the piano you shipped are all these things. The context window is more about a certain level of maybe how I might place the piano in the room to get good acoustics. There's another level of this, which Kevin will probably bring up in a minute, which is fine-tuning. And I, the reason I mention the digital pianos, right, the piano may come, you may be able to fine-tune the piano yourself. It might come with certain presets, right, to say, I want it to sound like a concert piano versus an upright versus, you know, an organ.
And then the really fancy digital pianos, you can even upload your own waveforms, your own description of like I want to tweak it a little bit. It should sound a little like this particular style of upright piano or baby grand. And you can tweak that. And that's, and I'll hand it to Kevin here, right? That's kind of like fine-tuning. I'm gonna try to work with Joel's piano analogy here.
Joel Moses
Ha ha ha.
Ken Arora
And I apologize to
Kevin Baughman (09:39.864)
Hehehe.
Ken Arora
musicians here
Lori MacVittie (09:40.578)
Keep it going.
Ken Arora
that we're really bad.
Joel Moses (09:41.767)
Yeah,
Lori MacVittie
Keep it going.
Kevin Baughman
Hehehe.
Joel Moses
I like to get my
Kevin Baughman
Hehehe.
Joel Moses
I like to get my analogies and my metaphors in first so that you know, Ken
Ken Arora (09:46.317)
Ha ha ha.
Lori MacVittie (09:46.434)
Yeah, so
Joel Moses
gets stuck to them.
Kevin Baughman
Yeah.
Lori MacVittie
we have to use them. Yes, now we're
Joel Moses
Right.
Lori MacVittie
'cause it was trained in
Joel Moses
Ha ha ha.
Lori MacVittie
and it's baked in
Joel Moses (09:51.347)
Exactly.
Lori MacVittie
and now we all have to,
Ken Arora (09:52.599)
Ha ha.
Kevin Baughman (09:52.76)
Yeah.
Lori MacVittie
we have to work with it.
Kevin Baughman
Yeah, so fine tuning, and I'm not going to even attempt to talk about pianos, but
Lori MacVittie / Joel Moses / Ken Arora
Ha ha ha.
Kevin Baughman
fine tuning for us historically has been when the context windows were a lot smaller and now they're expanding. And what was happening was we were trying to pass too much information to the model at inference time. And yeah, we've backed away pretty heavily from fine tuning. There are cases in which we use it and there are people that do like to use it.
But the idea behind fine tuning is, as Ken said, is you do freeze the weights. So now you're going to have reproducible determinist-, well, not deterministic, in LLMs you never get that, but you get something that is frozen to behave in a way in which you want it. We usually like to use it for very small tweaks. Massive behavior changes, you're probably using the wrong model or tools when you're leading into it.
Lori MacVittie (10:47.704)
So during inference do the weights ever change? Based on what I'm telling it or how I'm shaping it?
Kevin Baughman (10:54.786)
The weights are frozen during inference time. The context will, I like to think of it as
Lori MacVittie (11:01.93)
Influence.
Kevin Baughman
I'm performing, it influences it, but I'm actually performing inference on more data.
Joel Moses
Yeah.
Lori MacVittie (11:08.685)
Okay.
Kevin Baughman
If I was to try to take a system and I wanted to pass in a fruit and have it classified as a fruit or a car and have it classified as a car, if I say apple to an LLM, it's gonna say something crazy.
Kevin Baughman (11:23.018)
If I was to give it a few shots, what we call few-shot inference, which a few examples of like apple equals fruit, banana equals fruit, like Hyundai equals car, the system will learn those rules, but you're performing inference on that whole thing. So what you're performing inference on is actually different. People think of that as learning. And it is clever because activation layers allow you to look back at prior data to influence your future results.
Joel Moses
Right.
Kevin Baughman
Hopefully that makes sense.
Ken Arora (11:50.54)
Right. Right. It's
Lori MacVittie (11:52.248)
No.
Ken Arora
it's sort of like maybe I have a, you know, I'm talking to a five year old and they know something about the world, but they don't know what's called a fruit, versus say, I don't know, I'm gonna pick an animal, right? Go, "well, that's a fruit," and you point at the banana and an apple and so on. And then you say, "Well, these are animals," you point to the dog and the cat, and you go, "These are animals and these are different." And they'll generalize off of that.
Ken Arora (12:19.423)
But unlike, an LLM is a little bit unlike a human in the sense that a human will get that from the example and then over time they'll bake it into their model. We don't quite have that yet, but people working on it.
Joel Moses
Mm-hmm.
Ken Arora
But it
Lori MacVittie (12:34.936)
So we're fine tuning children.
Ken Arora
but it's sort of like what's hard-coded, what's baked in, the fixed weights?
Joel Moses
Yeah.
Lori MacVittie
We're fine tuning...
Ken Arora
What's in the piano you get from Yamaha versus
Ken Arora (12:42.731)
what are you tweaking by where you're placing the piano and maybe what instruments you're placing next to it?
Lori MacVittie (12:51.416)
I feel like we're approaching the okay training is an apple but then we get an orange and that's inference and
Joel Moses (12:56.776)
True.
Kevin Baughman
Ha ha ha.
Lori MacVittie
and so it's different it's,
Ken Arora (12:59.458)
Yeah.
Lori MacVittie
you know, but so what, you know, you're talking lots of memory. You need it in both, you need GPUs for both. Obviously when you're training you have different--and I hear this word a lot--pipelines that you don't have with inference, right? So what do you need in those pipelines? You have lots of data you have lots of information,
Lori MacVittie (13:19.608)
you have different security needs, so what makes up a pipeline, Kevin? You've built them. So what goes into a pipeline? Well, I hear you've built them. I could...
Kevin Baughman
Ha ha ha. Well, yeah.
Lori MacVittie
Right?
Kevin Baughman (13:26.766)
Yeah, I mean, for us, it's tons of compute GPUs and tons of data. Now, the
Lori MacVittie (13:36.781)
That's it.
Kevin Baughman
security needs of us is we like to think of security as that if I didn't train something into a model, I don't have to worry about it saying it later. Right? So when these LLMs are trained, they're trained on massive amounts of data and
Kevin Baughman (13:54.147)
you ask it questions and it answers inappropriate things. If, you know, we like to make our system so that it's never been trained on that stuff. So
Lori MacVittie
So,
Kevin Baughman
we have to worry about it coming less.
Joel Moses
Yeah.
Lori MacVittie (14:01.934)
So the choice of ingesting Reddit was basically the first problem.
Kevin Baughman
Ha ha ha.
Lori MacVittie
Like way back that's where it all started. That makes, but that makes sense, right? Controlling the data coming in is actually a security issue. Like don't put top secret stuff in there because it'll spit it out later. I mean it should be obvious, but maybe it's not. So controlling the data is actually a technique. And then on the reverse, it's
Lori MacVittie (14:31.456)
also controlling the response, which is data coming out. So you're kinda doing the same things but in reverse, isn't it?
Kevin Baughman (14:41.39)
Yep.
Joel Moses (14:42.033)
Yeah. Now, Kevin
Lori MacVittie (14:42.232)
There you go. That's it.
Joel Moses
Kevin's
Ken Arora (14:42.913)
Yeah, and that is
Joel Moses
Kevin's
Ken Arora
that is where you, you hit it, right, you hit the nail on the head. This is one of the key differences between inferencing and training. Training, you're baking it in. It's in the piano, it's in the model. And it might be hidden, it may be hard to pull out, but it's in there somewhere. Whereas with inferencing, you're saying that's not in there. If I want to, if I have some proprietary data that I don't want to share and I don't want to be baked in the model, but it's still useful to you then I'll do the one shot, the few shot learning, the expanded context window that Kevin talked about to give it that information.
So that's an important, you know, an important reason why you might pick one over the other. I need you I need my model to be good at, I don't know, spotting Russian nuclear silo sites, but I don't want it to be put into the model. So instead what I'm gonna do is I'm going to take a model off the shelf and I'm gonna then give it examples of this is what a silo site looks like
Kevin Baughman
Right.
Ken Arora
and let it then say, okay, now that you've give me some examples, I can find others.
Joel Moses
Yeah.
Kevin Baughman (15:46.959)
Right. And it can be very powerful by attaching it to a knowledge base. And now you can change that knowledge base. You can remove things from it. You can add things to it. The other thing, it allows us to deal with different tenants in the system. So if I want information specifically for Joel about pianos and then Ken about fruit,
Joel Moses
Ha ha.
Kevin Baughman
I can have the knowledge base contain relevant information. And that allows us to keep stuff that should be secret out of individual deployments.
Joel Moses
Right.
Lori MacVittie (16:16.632)
Right. Well, and I think part of the reason that people started believing or at least have gotten the idea that during inference they can learn is because of the systems around it. When you look at something like ChatGPT, there's also a lot of session data, it saves your conversations. That's not the model. The model's not doing that. That's the systems outside of it, the server and the,
Lori MacVittie (16:41.218)
you know, everything that goes along with it. And it's starting to sound like it knows you more because it's building more of that context and sharing it every time so it knows, hey, this is who this is. This is who've been talking, these are the key points, right, remember these five things. Right?
Joel Moses
Yeah.
Lori MacVittie
Lori likes this. She has fish. She doesn't like dogs. You know, it should remember this, but it doesn't, right? If the entire system went away, the model would forget everything. And I'd have to start over rebuilding that context, right?
Joel Moses (17:15.325)
Yeah. Now there there is a difference in we've been talking about pipeline and I think that's a good discussion. What we're talking about is an ingestion pipeline for the data for the data set. Now once you get inside the ingestion, you also have a distribution pipeline to take this work and split it apart and do your normalization passes and feed it in to GPU processing, et cetera. And that is also where some differences between training hardware
and inferencing type hardware exist as well.
The memory needs are much higher on the training side, but also most importantly, the amount of memory bandwidth that you have to feed things into memory and clear things from memory, the faster the memory is, the better. On the inferencing side, you can take a little bit of delay. It's always best if you can get the fastest memory possible, but it's actually less critical. And so if you're looking to save money, you can use lower speed memories with inferencing, but that's the really the only difference in the two pipelines.
Ken Arora (18:20.099)
Yeah and
Joel Moses
So, like on the training side, HBM memories are a great place to keep this training data because you're doing a lot of operations and it's about loading and unloading and doing these atomic operations. On the inferencing side, you're basically loading once and you've got an intermediate set and you discard it after you're done.
Ken Arora
Yep.
Joel Moses
And so the the memory is different inside the processing pipeline.
Ken Arora (18:43.107)
Yeah, and speaking as a former hardware person, it's really about that memory working set as Joel said, right? For inferencing, the memory you need is whatever it takes to hold the model, usually measured in double digit gigabytes. Not trivial amounts of memory, but not huge amounts of memory either. Right?
You need 10, maybe 100 gigabytes of memory to hold your model. And once you got that, getting more memory is not really all that useful. Compare that to the amount of data that you're training on, which is, you know, petabytes of data. And so you're never gonna keep that right next to your silicon GPU. So that memory bandwidth getting that, those petabytes of everything on Reddit in and out, that's your bottleneck.
Joel Moses (19:26.195)
Yeah.
Lori MacVittie (19:27.832)
Well,
Joel Moses
So from a design perspective, you definitely have to take all of those things into account. So and it's in multiple places. So you've got the ingestion pipeline, you've got the processing pipeline, you've got optimization functions, the whole nine yards. You have to look at each step of the pipeline and choose exactly the right components that give you the most efficient and cost effective way of processing. And that's where the major difference is in inferenc- between inferencing and learning.
Joel Moses (19:54.804)
Although the training and inferencing have the same mathematics behind them,
Lori MacVittie
Ha ha ha.
Joel Moses
the actual hardware is different. I realize I've come around in a full loop there.
Kevin Baughman
Ha ha ha.
Lori MacVittie (20:00.078)
You have. You ha-, I mean that's and that's okay 'cause you were so close to like
Joel Moses (20:09.942)
Ha ha.
Lori MacVittie
the perfect like takeaway. It was right there, right?
Kevin Baughman
Ha ha ha.
Joel Moses (20:13.842)
Yes.
Lori MacVittie
Like it's different compute. You have to identify the steps and get
Lori MacVittie (20:17.656)
put the right things and then you right back to the math is the same though. Well of course, like one plus one is two, no matter whether I do it, a five year old does it, we do it on an abacus,
Joel Moses (20:27.27)
Mm-hmm.
Lori MacVittie
we count it using M&Ms, or the computer does it. The math is the same. That should be evident, Joel.
Joel Moses (20:33.393)
Yes.
Lori MacVittie
It's irrelevant. That's what I'm saying.
Joel Moses (20:36.945)
Yes, but it let me talk about pianos for a half hour, so I'm good.
Kevin Baughman (20:37.88)
Ha ha.
Lori MacVittie
Okay. Well that's
Lori MacVittie (20:40.918)
true. That's true. I think that is our takeaway. We are way over time. Right, is identify the stage that you're working at. Look at what you need for each in terms of components and put the right things in place to optimize each of these phases because they're both important. I mean you don't train something to not use it then in inference and you can't inference if you haven't trained. So they kind of go together like peanut butter and jelly or pianos and Joel.
Joel Moses (21:11.015)
Exactly.
Lori MacVittie
So, that's right. Well, thank you all. That a wrap for Pop Goes the Stack. Please subscribe, so when the good stuffs arrive, you'll recognize it even without a keynote.