Lori MacVittie (00:03.362) Welcome back to Pop Goes the Stack, where AI keeps arriving like it's here to replace everything, right up until it needs DNS. Suddenly it's 1997 again. Hi, you're not seeing things. Yes, it's funny hats for Halloween. So welcome! If you can figure out what Joel is wearing, please let us know. We're not actually sure. Joel Moses (00:29.732) I may have forgotten my hat, so I just invented something. Lori MacVittie (00:33.334) It's working. It's just I'm not sure what it's working toward. You know, but that's okay because Joel has to be here because this week we're going to talk about DNS. And as you know, Joel's middle name is It's-Always-DNS. Joel Moses (00:49.478) I don't know how I acquired that name over the years, but it is always DNS, so... Lori MacVittie (00:53.346) Yes, you do. Because that's your first answer whenever there's an outage. It's DNS. Always. It's always DNS. Here to concur in the positive, I'm sure, is Griff Shelley. Hi Griff. Griff Shelley (01:07.136) Ha ha, Lori, Joel, or should I say Mr. Always DNS. You know, you can legally change your middle name. I've worked with people who did it. So Lori MacVittie (01:15.0) Yeah. I like that. Joel Moses (01:16.21) That's a good point. I'll look into that. Lori MacVittie (01:19.222) Right now we wanna talk about DNS and specifically AI. So that relationship, if there is any--I hear there is, not sure--you know, but DNS is foundational. Okay. Lori MacVittie (01:35.891) I guess one of the things that frustrates me about this topic is always like "DNS, what does it have to do with AI?" Well, it's like saying, Griff Shelley (01:43.8) Everything. Lori MacVittie Yeah. Joel Moses It does. Lori MacVittie IP doesn't... What's going on here, right? DNS is foundational. But there are some significant, I think, not necessarily changes, but approaches and perspectives on DNS that are coming from AI and how it's being used by AI. So let's dig into that. Go ahead, Joel. Kick us off. Joel Moses (02:04.227) Sure, yeah. I mean DNS is a foundational piece of AI infrastructure. Sometimes I think that people don't realize just how deeply DNS is dug into the infrastructure. I mean whenever you have a constructed AI pod in Kubernetes, a lot of times those pods find each other, of course, using Kubernetes core DNS. And if you have a delay going to the lookup between pods, then you have a delay across your entire infrastructure. Whether that infrastructure is built for training or built for inferencing, any delay in an AI infrastructure leads to propagated delay elsewhere in the system. So it's not just you know internal, it's also external. It's the way elements find each other. So when you're looking up, for example, your S3 storage, you're looking that up using DNS. That's the initiation. When you have AI agents, the agents also do their own DNS lookups. And they can be impeded by the unavailability of DNS. So it's everywhere within the infrastructure. I think it's really funny that we spend a lot of time, you know, putting up our H100 clusters, you know, putting together our liquid cooling loops, our high bandwidth memory, our multi-terabit InfiniBand fabrics, and then DNS breaks. And that keeps anything, any of that from being relevant. Lori MacVittie (03:30.988) So I think there's something very interesting in what you're saying, Joel, because I'm hearing a theme of like performance and availability of DNS. And I think as AI scales out and we have more volume and more requests, those are things that are being very much impacted. So Griff, you were mentioning this, you know, as we were prepping, kind of touching on the subject. Do you want to talk to that? Griff Shelley (03:50.828) Yeah, it's really interesting with kind of the rise of agentic traffic, right, because all the news releases and tweets, or whatever they are now, talking about "The agents are coming, the agents are coming." And they are. You know, we use agents for a lot of kind of even mundane menial tasks now, but the fact we have to consider is that they still run on that same infrastructure that we would use to, you know, search for a movie or search for travel plans or book a hotel or whatever the case might be. But they operate in very, very different ways than humans. And so we're really thinking more about traffic is operating at kind of software speed. Right? So instead of me going to a website and clicking on a link and then getting distracted on YouTube and then coming back and then doing some work on that site, whatever the case might be, you're more seeing agentic traffic almost as like a fire hose of requests or traffic coming to these sites or these applications or these software. And it still has to run on that same infrastructure. So Lori, to your point about the latency kind of compounding, when that happens at a really large scale, then you can get serious problems with keeping that pipeline moving, keeping the data moving quickly. And that's really how, you know, a lot of customers see value in building out their infrastructure is keeping that data intact and moving as it needs to. And because all of that at a very foundational level runs on the DNS protocol, it's really incumbent on everyone who's in this space to be mindful of it in that if your protocol, if your foundation is not up to handling that level of traffic, you're gonna have problems. Just that's all there is to it. You're gonna have problems managing and dealing with that level of traffic and that type of traffic, for sure. Lori MacVittie (05:50.402) And that's, really we're talking about in the past, DNS pretty much could handle the traffic that it was given and people knew how to scale it out. And there have always been DDoS attacks against every protocol that we've ever built but Griff Shelley Yep. Lori MacVittie we stopped that architecturally saying, Hey, we'll put firewalls in front of DNS and that will stop that. So it kind of insulated it from those attacks. But these are legitimate requests and I think one of the interesting things there is that, like you said, they interact with the same resources, but not in the same way. Joel Moses (06:28.752) Yeah. Griff Shelley Yep. Lori MacVittie Right? They use AI or APIs instead of, you know, GUIs. Griff Shelley Right. Lori MacVittie So it makes sense, Joel, you were talking about some kind of new standard that they're talking about for DNS. Joel Moses (06:42.427) Sure. Yeah, I mean if you think about it, the best and most available global directory that we have is probably DNS--if you're not talking BGP, but let's stay talking about Lori MacVittie (06:55.272) I Joel Moses DNS. Lori MacVittie I warned you. Joel Moses Sorry, Lori. Griff Shelley (06:57.593) Ha ha, Lori said Lori MacVittie I warned you not so Griff Shelley I wasn't allowed to. Joel Moses I mentioned the word, but again, a lot of times, Lori MacVittie (07:02.53) Trigger word. Joel Moses (07:02.643) you know, when you're looking to discover things, when you're looking for AI agents, for example, to publish trust anchors and things like that, so that you can look up the activity of an AI agent and trust that the agent is coming from the organization that you expect, there are proposals that are floating out there that incorporate some of that information being published through DNS. There's something called called AID, which is essentially AI Agent Discovery, and that is on the back of DNS. So as we look forward into the future, we aren't just leveraging DNS and increasing and changing the scale needs for DNS. We are more increasingly reliant on DNS to be the Uber directory of AI networks. Griff Shelley (07:49.524) Right. And Joel, I think you make a really important point there that I would say, you know, we don't necessarily need some sort of new fancy science project technology to keep this traffic moving, but we do need really good fundamentals to help it keep moving. One of the analogies I stumbled on was, you know, the internet traffic now as it exists today in large part is kind of like older, you know, Model T and Model A Ford's driving around on our roads and highways. And that's fine, for the time being. But with this impending surge of agentic traffic, we're going to be going from these more kind of lower impact traffic models to high scale freight moving along Joel Moses Right. Griff Shelley the internet super highway, such as it were. And also to your point about the agentic discovery, it reminds me of the Linux Foundation published a proposal--maybe I'm misremembering a little bit--this summer about DNS AID, which again sounds like a concert John Mellencamp would support Lori MacVittie Ha ha ha. Griff Shelley really having a lot of, you know, giving a lot of attention to the authentication. Right? Because Joel Moses (09:00.656) Mm-hmm. Griff Shelley again, to your point, do we know if this agent is who they say they are? Or are they coming from where they say they are coming? Because you can really amplify the kind of bad actor impacts when you can send a swarm of agents to do, you know, these misdeeds. Joel Moses (09:14.015) Right. Griff Shelley (09:15.554) So the authentication aspect is gonna be hugely important when you're dealing with a large amount of traffic traversing this very kind of base level protocol with some really high impact consequences if it's not taken care of properly. Joel Moses (09:31.43) Yeah. Yeah, the Lori MacVittie (09:32.408) Doesn't that defeat the purpose? I mean, so maybe I'm mishearing this, but it sounds like agents will have to authenticate to DNS. Is that what they're trying to say? Because that makes no sense to me. Joel Moses (09:43.74) Well, DNS will certainly play a role in authenticating agents. Griff Shelley (09:46.654) Yeah. Yeah. Lori MacVittie (09:47.348) Okay, all right, the reverse. Okay, Joel Moses (09:49.404) Yeah. Lori MacVittie yeah, I misheard the relationship there. That makes more sense. Joel Moses (09:52.826) Yeah, don't get me started on DNS Dane and how we should be treating TLS trust anchors. That's a whole other topic for another day. But for AI networking, you know, one of the things that's...you know, there's an old joke in the computer science area that there are only two hard problems in computer science. There's managing caches, naming things, and off-by-one errors. Right. Lori MacVittie (10:15.43) Oh, parody bites again. Ha ha ha. Griff Shelley Ha ha ha. Joel Moses And you know, if you think about...ha ha ha. Joel Moses (10:20.868) And if you think about it Lori MacVittie (10:21.286) Ha ha, Griff's never heard that. Griff Shelley (10:23.256) Ha ha ha. I'm gonna, I'm gonna write that one down, Joel. That's a good one. Lori MacVittie (10:25.222) How's that possible? Joel Moses (10:27.207) If you think about what that means though, let's decompose what's behind a single agentic AI workload. Like in under five hundred milliseconds, you know, a user clicks to start the agent, the app looks up the external LLM provider, that's a DNS query, requires a vector database, so it downstream initiates a DNS query for that. It hits an analytics API, does a DNS query for that, pulls user context from a remote server, begins making DNS requests for each one of the components that it launches. And a lot of times these are brand spanking new pods that are accessing DNS for the first time. And so for the most part, there's no caching Lori MacVittie No cache. Joel Moses involved in this. So it's a fresh lookup, it's a network round trip every single time for every single thing that it makes. Not only does that change the scale needs, it also injects a whole lot of network round trip latency in this transaction set that basically makes the chat bot visibly stutter Griff Shelley Yep. Joel Moses that you're talking to or makes the agent a lot slower in processing things. A user might blame the model's thinking speed, but what they're actually blaming is the speed of the DNS queries Griff Shelley (11:40.972) Yep. No, yeah Joel Moses and the fact that almost none of them are cached. Griff Shelley Yeah. No, you're absolutely right. And you know, when you're working with an application or website, you know, sometimes you expect, Okay, it might act a little slow, maybe there's a lot of traffic, maybe I'm asking it to show me a lot of information. But when you're, you know, engaged with one of these chatbots or you're trying to do rapid research, whatever the use case might be, that latency and that just kind of bumpy interaction is not a great way to maintain, you know, a user connected to that service. And in an attention economy where every millisecond counts or you're going to go to a competitor, it really becomes incumbent to make that transaction, make those transportations behind the scene as fast as possible. Otherwise you're going to run into those problems that you're talking about. And we actually, we had a customer recently pull an incumbent out of their environment because they just could not keep up with the query responses per second. And as much as we don't like to talk about speeds and feeds, it really did come down to the speed and feed of what the incumbent was able to do versus what we were able to do. And when we were able to show that we could do 2x the QRPSs that the incumbent was, that was game over because they need that interaction to be so quick and so streamlined and so efficient because everything else depends on it from that point forward with that interaction with the service or the LLM whatever the case might be. Lori MacVittie (13:14.416) So basically relying on a cache to improve your performance is not gonna be optimal in this situation. We actually have to start thinking about DNS and its performance, its speeds and feeds, and make sure that it's up to the task. Joel Moses Right. Lori MacVittie And maybe remembering that DNS is also highly distributed. Right? The entire, Griff Shelley Yeah. Lori MacVittie right, the design of the system is actually brilliant and maybe we need to architecturally like start exploiting that to our advantage in situations where we don't have we can't rely on the cache, where agents are spinning up a lot maybe we want to be able to, you know, further distribute DNS so there is a cache more locally or, you know, something like that. Joel Moses Sure. Lori MacVittie I, you know, these are not new techniques. It's just we don't typically rely on them 'cause we're so used to, "Oh yes, there's this DNS server, we'll just use that. We won't deploy more." Right? Griff Shelley (14:14.857) Yeah. Yeah. No, it's orders of magnitude more complicated, I think. And then when you add on some of these newer aspects of getting really granular traffic control. so for instance, maybe you're paying for a premium service on one of these chatbots, you know, you've got Claude Pro or whatever the case might be. Well, we're getting to the point where, you know, those users might go to one specific model in one data center, or the free users might go to a different one. Well, at that point, now you're potentially talking about global load balancing. And again, being able to trigger that extremely Joel Moses (14:50.558) Mm-hmm. Griff Shelley quickly, or you know, this model is really good for this type of request--maybe you're doing Claude Code--or this model is better at this request, you're just doing a chat interaction. Again, you need to have those pathways really set in the most efficient manner possible Joel Moses (15:07.572) Yeah. Griff Shelley again, to preserve that user experience and also just keep Griff Shelley (15:11.714) from hotspots and bottlenecks popping up and really wrecking the interaction. Joel Moses (15:17.29) Or even things like geographic partitioning, which Griff Shelley (15:20.066) Yeah. Joel Moses global load balancing can do. You can also relay node picker information up into global load balancing so that Lori MacVittie (15:27.758) Mm. Joel Moses you're picking the most available and quickest GPU Griff Shelley (15:31.106) Yes. Joel Moses resources to serve the problem downstream. So DNS is an important carrier of not just name information, it can be a carrier of performance information as well. Griff Shelley (15:40.524) Yeah, absolutely. Joel Moses (15:41.914) And as we move forward into the future and things begin to discover more information about each other, we're discovering trust anchors, we're discovering organizational attachments and things like that, all using this venerable DNS protocol that's been with us for a very, very long time. It rides the back of that technology. Griff Shelley (16:03.436) Yeah, yeah. And I mean, I think Lori MacVittie (16:04.838) Well I'm Griff Shelley it's kind of, oh, go ahead, Lori, sorry. Lori MacVittie No, I was just gonna say, I mean, everything does. I mean, we get so excited every time there's a new technology, but every new technology that's been radically impacting our architectures and, you know, businesses as it were, ends up being at the top of the stack. Griff Shelley (16:26.764) Yes. Lori MacVittie The stack that's at the bottom, things like IP, TCP, UDP, you know, and the protocols that Joel Moses Ha ha. Lori MacVittie (16:34.523) you know... Did, were those too many Ps? Griff Shelley Ha ha ha. Lori MacVittie P P P P P P Joel Moses (16:37.722) That's a lot of Ps. Griff Shelley Ha ha ha. Lori MacVittie What? What are you, twelve? Griff Shelley (16:40.65) We didn't say BGP, it's okay. Lori MacVittie Yeah. Joel Moses (16:43.258) That's right. We didn't say BGP. Yeah. Lori MacVittie (16:44.006) Ugh, trigger warning, we will mention BGP. Joel Moses Yeah. Lori MacVittie But those, right, those have been around and they continue to be the foundation. AI is even on top of HTTP, which was really the last big, you know, Here, let's make everything different. So it shouldn't surprise us that we have to go back and revisit them though, because they weren't built for this kind of traffic or this kind of, you know, volume of traffic or these kinds of requests. But we always have to go back to the basics and look at them and say, "How does Griff Shelley (17:15.936) Yes. Lori MacVittie this new technology impact that? Do we need better performance? More scale? New security mechanisms? What do we need?" Griff Shelley (17:24.5) Yeah. Lori MacVittie Yeah. Griff Shelley Yeah, I don't think you can deny that a lot of these emergent technologies are revolutionary in a lot of ways. And at the same time, we have to acknowledge they're standing on the shoulders of giants to tap into that old saying. Joel Moses That's right. Griff Shelley And those shoulders have to be able to hold 'em, right? And so it's incumbent on us as the people who provide this infrastructure or our customers who deploy it, you know, to make sure that they're up to the task. Because these technologies are are heavy but they're super impactful and so it's incumbent all the way down to do so. Joel Moses (17:58.122) Yeah. Lori MacVittie (17:59.204) Well that's a Joel Moses You know Lori MacVittie a good, that's a good point. We're starting to run out of time, sadly, but that's a, you know, good... We keep saying we have to do these things, we have to make sure. So I think one of the takeaways that would be really good is well, how does someone do that? If someone's walking away from listening to this, what advice would you give them to go check on their DNS? Other than just, you know, call your mother, you know, call your DNS, Griff Shelley Ha ha ha. Lori MacVittie make sure it's okay. Joel Moses (18:28.411) Yeah. Lori MacVittie (18:30.783) You know. Joel Moses (18:31.491) Well I think the place to start is first--and this is the takeaway that I've got for today--remember that it is the hidden backbone of AI engineering. DNS is kind of that backbone that ties everything together. A microscopic light latency spike at the DNS level can cascade itself very easily into a catastrophic bottleneck when multiplied across a cluster that has 10,000 GPUs using them for distributed training, right? I like to think about DNS in AI infrastructure not being like a dusty yellow pages that you look things up in occasionally, I like to think of it as a high precision timing belt in a high performance car. Griff Shelley (19:10.296) Hm, I like that. Joel Moses (19:11.25) The engine might have a thousand horsepower, but if the timing belt slips by half a millimeter, Griff Shelley (19:15.372) Yes. Joel Moses the valves smash into the pistons, the entire car grinds to a halt. Griff Shelley (19:20.14) Yes. Joel Moses And DNS is kind of one of those things that if it's not high precision, if it's not answering in a small period of time, then everything downstream from that becomes suspect. Griff Shelley (19:31.308) Yeah. Joel Moses That's my takeaway. Griff Shelley Yeah. No, Joel, I think you're onto something there in a really foundational way. I actually started my career as a mechanic. And so I saw a lot of people coming in with high horsepower cars, but they just couldn't change the oil on time. Joel Moses Yeah. Griff Shelley You know, and you might make all the power in the world, but if your oil's not clean and your fuel injectors aren't up to snuff, you're not gonna make the power and your car's gonna stumble. And in a lot of ways, it's the same kind of case for this infrastructure we're talking about. If you can get the fundamentals right and you can make sure that all your records are aligned where they need to be, you don't have any kind of configuration creep across your estate, you're setting yourself up for success so that the bigger rocks down the road are a little less onerous to try to tackle. But you know, cleaning up your DNS is not always sexy, but it's extremely it's extremely important to make sure that you are that those shoulders that you're gonna be standing on are strong enough to handle what you're gonna put on 'em, for sure. Lori MacVittie (20:35.502) I like that. And that would be my takeaway. It's like go find your DNS. Do you know where it is? Griff Shelley (20:41.058) Yes. Lori MacVittie Have you checked on it lately? Like make that an actual priority. Like we are going to actually go and look at it. You know, has it been updated lately? Who knows? Sometimes these things are not. So Joel Moses (20:52.456) You might have to dig a little. It exists in Lori MacVittie Yeah. Griff Shelley (20:54.828) Yeah. Joel Moses different places. It exists inside of Kubernetes. Lori MacVittie That's right. Griff Shelley Yes. Joel Moses Every cloud provider has its own mechanism for internal DNS Lori MacVittie Yeah. Mm-hmm. Joel Moses and record discovery. Griff Shelley (21:01.1) Yes. Joel Moses So you have to dig. Lori MacVittie Yeah. Yeah. Griff Shelley Knowing where it all is. Yeah. Absolutely. Absolutely. Lori MacVittie (21:05.936) Yep. Yep. Go find your DNS and say thank you. That's my takeaway. That's Griff Shelley (21:13.356) I think that's perfect. Lori MacVittie There you go. All right. Well then we'll close it out with that and say that's a wrap for Pop Goes the Stack. Please subscribe because the future may be agentic, autonomous, and full of embedded things, but it still faceplants the same way when name resolution goes sideways.