Subscribe
Share
Share
Embed
Round robin isn’t just “not ideal” for LLM inference. According to recent scheduling research, it’s actively harmful, because it treats inference like stateless, interchangeable API traffic when it’s anything but. In this episode of Pop Goes the Stack, Lori MacVittie is joined by F5's Josh Mendoza, Principal Solutions Engineer, to break down why classic load-balancing assumptions fail under LLM workloads, and what to think about instead.
Josh walks through the evolution from early “spray and pray” distribution to smarter approaches that account for server load, workload type, and state. That history matters because AI introduces the same challenge at a new intensity: inference is a heavy compute-and-memory math pipeline, and conversations accumulate state. Once context and KV cache are involved, moving a request to a different server isn’t a clean failover, it’s a forced cache miss and a recomputation penalty that shows up as slower time-to-first-token and higher cost.
They connect this back to patterns teams already understand: VM migration, session persistence, and why “just move it” has always been expensive when the working set is large. LLMs raise the bar because users won’t tolerate latency, and the payload you’d need to move grows as the interaction continues. You also can’t ignore the request itself, since “summarize this” and “write a full analysis” have very different compute profiles, even if they hit the same endpoint.
The main takeaway is simple: don’t panic, but stop treating inference like generic API traffic. Effective LLM scheduling is closer to dispatching the right resources to the right job, with awareness of state, model placement, cache locality, and the true cost of moving work. The tools exist, but the mental model has to change first.
Creators and Guests
Host
Lori MacVittie
Distinguished Engineer and Chief Evangelist at F5, Lori has more than 25 years of industry experience spanning application development, IT architecture, and network and systems' operation. She co-authored the CADD profile for ANSI NCITS 320-1998 and is a prolific author with books spanning security, cloud, and enterprise architecture.
What is Pop Goes the Stack?
Explore the evolving world of application delivery and security. Each episode will dive into technologies shaping the future of operations, analyze emerging trends, and discuss the impacts of innovations on the tech stack.
Lori MacVittie (00:03.234)
Welcome back to Pop Goes the Stack, where microservices multiply like Tribbles and latency is the only thing truly distributed. I am Lori MacVittie here to count the hops and the regrets. And today we have a very exciting topic to discuss. Because I read some recent research on LLM scheduling--we have to call it something different, apparently, than load balancing.
But this research was pretty extensive and it made one thing painfully clear--not that we needed research to confirm it, but it's nice to have validation--it's that round robin still sucks. Okay? Nobody should be using it. It turns out their research says it's not just not good, but it's actively harmful to inference workloads. Okay, so that's kind of important. And that's because LLMs are, actually they're statefully stateless, which I love.
They're phase-driven systems, they have growing memory footprints, they have uneven compute demands and hard latency expectations. So treating them like interchangeable stateless requests, which we often hear associated with APIs, it guarantees things like cache misses and extended response times and failures. So what we wanted to do today is break down, right, why a lot of our traditional load balancing and distribution assumptions have to be rethought when we look at inference workloads.
Because they are different. And what we've traditionally done to scale things--right, the art of scalability, X, Y, Z, right, you scale things in certain ways--not necessarily gonna work if you want to optimize resources and ensure availability and fast responses. So, today we've got Josh Mendoza, who is a principal solutions engineer. He's actually running the internet out of his house. You can see it in the background. Beautiful, isn't it?
It is the mess that we think it is, but you know, he's got it under control. So welcome, Josh. Excited to have you here today.
Josh Mendoza (02:15.586)
Thanks so much, Lori. It's really a pleasure to spend some time here and share whatever knowledge I do have and share apparently the wall of what happens when you don't load balance properly or when you'd use it incorrectly.
Lori MacVittie (02:28.728)
That's, you get the internet. That's actually how we got the internet. Well it's
Josh Mendoza (02:35.435)
Exactly.
Lori MacVittie
it might be good to start with kind of just, you know, an understanding of load balancing. I like the analogy of it's a table, it's like musical chairs, but that's if you're five.
Josh Mendoza (02:48.44)
Mm-hmm.
Lori MacVittie
If you're five, that makes sense. But in the real world that's not actually how we distribute traffic or do traffic steering today, right?
Josh Mendoza (02:56.48)
Right. So many years ago, for anyone that's been kind of around the load balancing space, we learned, hey, one server can't handle all the traffic load. We've covered this many times. So we said, "Well, let's put a bunch of them together and then start spreading traffic across them." Wow. Lo and behold, round robin load balancing came around because we found out, hey, if we just start you know, praying and spraying, then everything will be fine.
Well, obviously things didn't work out even then. Because what we ended up finding out was when you don't even look at the load and you're just passing packets, maybe one server finished its workload and we're not sending it enough traffic and you're still seeing a misbalancing there. So then we had to get more intelligent. We had to then ask the server, "What is your load? How busy are you?" Or the servers, the load balancers had to keep track of those things.
So we, even at the beginnings of what load balancing was, we were even then trying to figure out how to get away from something as simple as round robin.
Lori MacVittie (04:01.891)
Right. We recognized that not all requests incurred the same, I don't know, you know,
Josh Mendoza (04:13.324)
Work
Lori MacVittie
Work.
Josh Mendoza
Yeah.
Lori MacVittie
Yes, that's the word I'm looking for. Thanks, Josh. See. But it didn't, right? If you ask for I just want this, you know, 1k image, well it's we understand that's gonna be pretty minimal. But if we, you know, if we're sending a request to, I don't know, update a database that has a different, right, work profile, if you will.
Josh Mendoza
Exactly.
Lori MacVittie
And because they're different,
Lori MacVittie (04:40.844)
we have to distribute traffic differently. Or at least consider that
Josh Mendoza (04:45.314)
Yeah.
Lori MacVittie
when we make the decision.
Josh Mendoza (04:45.314)
And what we ended up seeing, at least from my background, was we started seeing that shift even maybe fifteen years ago over to now we're trying to work the workloads into what services are you actually servicing. So when we started to first see what we now call like the microservices infrastructures, those all began as things such as what are you actually running?
You saw early versions of application workload distribution to make sure hey now I wanna separate my stack, it's not monolithic, okay we're getting smarter. But because this workload is very different than this other workload, well now if they're all the same servers, if I still have a load balancer that's saying just, you know, spraying the traffic, again it's gonna be the wrong work path.
So we started to get more intelligent about what workloads can be run on which servers. And as that ended up growing, that's where we start to see now imagine you've got one workload here, like you mentioned, it's just my I'm gonna respond with images. Now I have a business service that's gonna do something major, like it's actually do, you know, information transformation. And then on the other side I've got streaming content.
For example, I'm actually watching something live. Now we're starting to get to more modern applications. Each of those is all having to be treated differently. Because I can rapidly respond over here with the images or static content, but these heavy workloads, I can't just be like, oh, I'm gonna move the same client to another engine or another server that has that workload.
Because you might lose things such as state, where you need to maintain awareness of the session, whatever it may be. So even as I'm discussing these things, you start to think, we're pulling apart how do we load balance. Decause we need so much more information. And what we're seeing obviously with AI is an even further explosion of that going on.
Lori MacVittie (06:58.338)
Yeah, it gets very different. I think that's why we actually moved from calling it just load balancing to things like application delivery to traffic steering or traffic management because we weren't just distributing load anymore. We're making decisions about where to send this work. Right, we're a dispatcher almost saying, "okay, it has these characteristics. We're gonna dispatch this workload over here because this is the best fit."
So we're actually right doing magic. You see that in global load balancing, right? That's really a part of it.
Josh Mendoza (07:33.493)
Exactly. And I love that you said the dispatch, because that's one of the key things. If we think of someone who's doing a dispatch, say a 9-1-1 operator who's dispatching resources. They're not gonna send a police officer to a burning building by himself. Cause I'm, they're, respect to all of our law enforcement out there, they can't put a fire out with waving something at it. You you're gonna need a fire hose, you're gonna need the right tool sets.
You need the right workloads and the right people on site to handle that situation. Now that's the same thing with load balancing. As we get into these situations, you can't just say, I'm gonna send the same traffic that I would for, let's say for example, I have a live streaming encoding service, which for example, the one that we're on right now, the platform that you're watching through.
You're watching it in higher resolutions or lower resolutions. Well, that workload to encode that video is incredibly heavy. But if all of a sudden, as I'm sending the workload there, let's say the fire is here and I'm sending all of the work there, and I decide to move the client to a new server where it doesn't have any context, any of the video encoders, and any of the previous video frames, all of a sudden we're gonna lose, you're gonna see a massive drop.
Because the systems, there's no way to correlate those things together. Well, the same thing ends up happening in AI services and especially when you're talking about global distribution. When you're gonna globally distribute traffic and you start sending traffic to one place, and then all of a sudden you wanna move that workload somewhere else. Now, if we think about AI, when you move that workload somewhere else, the way that AI works is it has context, it has awareness, it has all of its tokens over here, all its memory, let's say,
Lori MacVittie
Mh-hmm.
Josh Mendoza
is over here, and all of that memory suddenly has to be shifted somewhere else. Well, that's a massive payload. And so
Lori MacVittie (09:41.23)
Huge, yeah.
Josh Mendoza
there's so many things involved in that space even. And I know we're, I'm jumping ahead a little bit into that topic, but that's what I keep thinking of is we wanna send the right resources to the right places. And when we start getting to global traffic management
Josh Mendoza (09:57.165)
you start having to understand what is running, not just what's running and what services are available, but how heavy is that compute. How heavy would it be to move if I have to move it? Because obviously there's circumstances where you must. But how do you arrange for that? So, and I'm probably, even everything I'm describing is probably like jumping all over
Lori MacVittie
Ha ha ha.
Josh Mendoza
'cause that's just how the internet works, you guys. So many moving pieces.
Lori MacVittie (10:19.95)
So many moving pieces. Well, and I love that you used the dispatcher, right, you played on that. Because one of the things that AI does that makes it different is really not just about the, right, what's going on and the resources, but then, right, the caller, I mean themselves. So traditionally I send a request. That HTTP request is going to look very much the same.
Every single time. Here's the headers, here's the thing, right? Here's some information. Here's the data inside of it. It's gonna be about the same. There's not a whole lot to differentiate my request to Google from yours, right? Just
Josh Mendoza
Right.
Lori MacVittie
the, right, they're gonna be the same. With AI, that's not true. We not only have to consider the server side, right? Where is the last, well, where's the memory? Right? Where's the KV cache? Where was it last? What model is loaded? Right? You can have different models loaded in different inference servers. So you gotta make sure you send it to the right one. You don't wanna have to sit and wait for a cold start. But then you also have to look at the actual request. Because a request for, "hey, summarize this document," is very different from, "hey, write me a treatise on load balancing."
Like these are going to incur very different amounts of work. Right? One is going to be a lot quicker than I have to actually like research and generate something. So now we've got another variable that we have to also factor in when we start making decisions about where do I dispatch this particular request? And I think that's one of the things that we're missing because most AI today is, you know, it's an API.
Right, there's an API in front of it and we say, "ah, it's just an API." But as we've been learning with security, right? Oh no, we need all these like guardrails and red-, we need all these things to help protect it because it's not the same.
Josh Mendoza (12:26.36)
Yeah.
Lori MacVittie
The same is going to be true when we start scaling it out. Hey, it's not the same.
Josh Mendoza (12:31.54)
Exactly. Exactly. I think one of the key misconceptions is that AI, like the internet, it's just a little box.
Lori MacVittie (12:43.213)
Ha ha ha, just a box.
Josh Mendoza
We can just set it on a table and it just magically does things. Not quite. Right. As you start digging more and more into what AI is doing, is it's just a really complicated math problem. And that complicated math problem involves a lot, as you mentioned,
Josh Mendoza (13:01.0)
of memory, a lot of resources to compute the next thing it's about to add. It's interesting, I recently was reading an article about how when you look at the the math behind it, the word "the" is one single token. But it takes, depending on the model size, billions of parameters that it will chunk through as a math problem to find "the" as a token.
And then the next token it adds, so "the," let's say, "the whale," adding the "whale" has all those parameters that it has to chunk through again. And so math, math problem... Now imagine you fed it a document like a book. Imagine how many tokens it's gonna take to, and all of that is gonna be stored in local memory. Now if you wanna suddenly move that conversation, that entire thing, to another server.
It's not just magically gonna appear on the other server. And so that's one of the key things is that, it's because it's a math problem that involves a lot of compute, a lot of memory, we can now think back to when we had this revolution in virtualization--I'm gonna step back in time a little bit. When we first moved into virtualizing servers, and we had the situation of I've got a VM that's got a workload because that's where my machine is for that particular workload type.
I don't know if you remember, Lori, of doing the VMotions that we had to do in those times.
Lori MacVittie (14:40.086)
Yeah, yeah. That, lots of data too. I mean, just huge.
Josh Mendoza (14:45.332)
And so, now think of in those times for those of us that had to play in that space of your snapshots and your moving and the V motion technologies that were built to move workloads and carry the memory that you're gonna move over, because you have to have the memory, all of these things, from one data center to another, how long it would take, how long we had to pre-plan.
Now think of the fact that if you asked whatever AI agent you talk to a question, and it took longer than three seconds to respond to you after you're done talking. That time to first token, you would not use that service. So that is what's happening with AI, is it's exploding that same conversation we had when we were talking about virtualization is now exploding.
But the data sizes and the way that we have to handle it, it's not changing completely. It's just getting more complicated and we have to do it faster. So then that, going back to like the original topic we were mentioning of load balancing, that's exactly why we can't just round robin through these things. Because that workload is in one place and that workload itself can spin up a whole other workload somewhere else that it needs to spin up depending on how those systems are interconnected.
Lori MacVittie (16:14.412)
Yeah, the KV cache is very similar to, you know, how we think about like SSL persistence, right? We had to basically be aware of what server you were originally assigned to because the cost of calculating and doing all the math for the certificates and to do the, right, the encryption and the decryption was too expensive. So we made sure that we could understand, hey, we're always gonna send you for this session to the same place because all the math is already done.
And what you're saying is AI is the same thing, right? It did all this math to get "the whale." So we want to keep sending you to that same place because the work of figuring out "the whale" was already done and it's keeping it in its cache. And if you just move it someplace else, it's got to do all that work again, which is going to occur more latency. And the thing nobody talks about is that over a session, that keeps growing.
Because it keeps sending that context just keeps growing. So every time it's got to do that again. It's got to do all the math for all that context again. And you just don't want to do that. It's inefficient, very inefficient. So,
Josh Mendoza (17:31.105)
Absolutely.
Lori MacVittie
I'm hearing we've kind of gotten to the end of our discussion, but if you were gonna leave people with like one takeaway, that
Lori MacVittie (17:41.046)
they should remember about, right, load balancing, scale, and AI. What would it be? What's your advice?
Josh Mendoza (17:51.149)
Wow, to leave one single thought. There's just so many that come to mind immediately. First of all, don't panic.
Lori MacVittie (17:57.389)
Yeah.
Josh Mendoza
Right.
Lori MacVittie
Oh, that's a good one, yeah.
Josh Mendoza
It, you know, Adams had it right the first time. Just remember your towel, don't panic. Because the reality is AI, similar to many of the workloads we've had to handle in the past, it's a workload. And what we need to understand is how those things interconnect.
Josh Mendoza (18:20.192)
That's what got us to the place that we are now. Is if you can take the time to understand how do these systems actually send traffic between themselves, which we've covered in many of the topics that we've posted up on F5, many of the things that you've covered, Lori, on how those things function. Remembering that the way that we move forward is understanding the workloads and the more that we understand the workloads, the more that we'll be able to handle how we distribute that traffic intelligently.
I think that everything that we have to do in our space is it's a matter of taking the time not to panic, understand what's actually happening behind the curtain. And sure enough we're gonna find out it's probably somebody just back there pulling a couple of levers instead of some magical Oz type situation. So that's what I would recommend. I think that it's definitely we're seeing the evolution. We've seen it happen many times. We saw it when we had microservicing.
We saw it when we had the virtualization of hardware. We're seeing it with the modernization of applications. And now we're seeing the next revolution of AI. And each time we see it, we grow more and more intelligent in what we can use for making the decisions on how we distribute traffic. And the more intelligent that we are ourselves in how those systems work, the more we'll be able to feed into the systems we build to make sure we intelligently use the resources that we have available to us.
Lori MacVittie (19:56.531)
Awesome. That is great advice, especially the don't panic. And that's true, right? Each generation of applications has a, you know, equal and opposite effect, right, on like app delivery. Right? And we go, oh, we need to consider this now, and then it happens. And this is also true with AI. Right? The base is there and it's a very rich and robust set of different algorithms. Things exist other than
Josh Mendoza (20:24.652)
Mm-hmm.
Lori MacVittie
round robin.
Lori MacVittie (20:25.804)
Okay. And then just the core of, you know, build your own, if you will, right, traffic steering. So all of these things exist. It's just a matter of looking at the technology, looking at your environment and what's there and going, "oh, I need to know this," and then factoring that in and bam, suddenly, right, you're very efficiently distributing traffic for AI. So it's not rocket science, it's computer science. we've got this. So,
Josh Mendoza
Yeah.
Lori MacVittie
we will get there. We'll get there. So don't panic. I like that. So that is a wrap for Pop Goes the Stack. Please subscribe because your next architecture diagram is going to need a legend and probably a therapist.