Subscribe
Share
Share
Embed
Chaos Lever examines emerging trends and new technology for the enterprise and beyond. Hosts Ned Bellavance and Chris Hayner examine the tech landscape through a skeptical lens based on over 40 combined years in the industry. Are we all doomed? Yes. Will the apocalypse be streamed on TikTok? Probably. Does Joni still love Chachi? Decidedly not.
Ned: I am impressed that someone went back and found their VHS tapes of live shows that I was part of back in the early-2000s, and digitized those shows and has uploaded them to YouTube.
Chris: Was this just a way to, like, embarrass you?
Ned: Well, I did have frosted tips, so totally justified.
Chris: I mean, we all had that moment. I just never got the pleasure of seeing mine on YouTube.
Ned: We can change that.
Chris: Nope. No, we can’t. YouTube’s over.
Ned: Hello, alleged human, and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. The limitless potentiality of the many-worlds theory, it isn’t complicated. Every decision we make spawns another branching universe, thus causing millions of parallel worlds to exist, and I just simply observe them all simultaneously, just like everyone does. With me is Chris, who is also here. And there, and there, and there, and there. But not there, because we erased his birthday.
Chris: Nice callback.
Ned: [laugh]. Thanks. If that isn’t part of the intro—like, the pre-roll—it’s not going to make any sense, so I think we should definitely edit it out [laugh].
Chris: [laugh]. I like the way that you say ‘we.’
Ned: When I say ‘we,’ I mean the good editors over at HumblePod, and you and I have nothing to do with it [laugh].
Chris: We, uh, handle that rarefied air called not ‘knowing how to do it.’
Ned: I am personally offended by that. I’m not saying it’s wrong. I’m just offended, Chris.
Chris: How dare you point out my obvious weaknesses and limitations, sir?
Ned: So true, and how dare you? [sigh]. Well, I suppose we should get to the topic at hand.
Chris: Yeah. You know, I’m looking forward to another 45,000 words about DNS.
Ned: Hey, I trimmed it down. This one is two-thirds the size of last week. So, you’re welcome. I actually cut some stuff because there was more. There [laugh] was so much more. When I found myself, at eight o’clock in the morning, reading through a Windows NT 3.5 manual. I realized I might want to stop and reconsider all of my life choices up until now.
Chris: Seek help. The sad thing is I know that you have that on your bookshelf, don’t act like you Googled it.
Ned: Shut up. I hate you. The last week, I spent entirely too much time talking about DNS, so this week, as a fresh change of pace, I’ll be talking about… DNS.
Chris: [laugh].
Ned: You’re welcome [laugh]. Realistically, you should probably go back and listen to last week’s episode, but if you don’t want to, number one, it’s fine, really. We’re not disappointed in you. Really. It’s fine… it’s fine. And here’s the Too Long; Didn’t Listen. DNS was established in 1983 because we needed to have a dynamic, distributed solution for resolving names on the various inter-networks.
The original DNS spec was pretty much unconcerned with security issues around things like authentication, integrity, encryption: not important. So, solutions like DNSSEC and DNS over HTTPS were created to solve those issues. Naturally, this has led to other issues, especially with DoH becoming more popular.
Chris: DoH being?
Ned: DNS over HTTPS.
Chris: Thank you.
Ned: You’re welcome.
Chris: I didn’t forget.
Ned: This week, we’re going to dig into some of the DNS alternatives that sprung up as private networks became a real thing, what type of DNS attacks are out there today, and what the Microsoft ZTDNS solution does to try to ameliorate them. And honestly, I only had enough time to write about one DNS alternative. See my previous note about making this slightly shorter, and you know, the NT book.
So, DNS was created to help with name resolution on networks that spanned multiple organizations. So, we are talking about the internet, but we’re also talking about other networks that existed at the time: ARPANET, CSNET, NSFWNET—you know, the original porn network—okay well, maybe not that last one, but still. The main idea was that you would need a distributed hierarchical name resolution service because you had all these different nodes and all these different networks, and no central repository of all information. That was done by design. We could get into the whole decentralized nature of the internet and how that was a key design choice, but hey—wait for it—that’s a whole other episode unto itself.
Chris: [laugh].
Ned: Jesus, Chris. How many episodes are we up to, spawning just from this one and last week?
Chris: At least five.
Ned: [laugh]. Well, that’s good. We’ll never run out of things to talk about. So, my point is that DNS was really good for the shared or public networks, but it might have seemed like overkill for an internal network, and so companies like IBM established their own internal networking software implementations, including the NetBIOS software interface, you may have come across this semi-acronym once or twice in your life. The name itself comes from the fact that the networking software ran on the network adapter itself in an IBM PC instead of running on the CPU, and the software was stored on a BIOS chip on the network adapter that’s handling the network transport. Hence: NetBIOS. That’s where the name comes from.
Chris: Yeah, I still don’t follow.
Ned: Ah, fine. I’ll draw you a diagram later. That network adapter handled network transport and session layers of the traditional OSI stack. So, programs could interface with the network adapter by sending what were called Network Control Block messages, and receiving responses. I’m not going to get into NCBs. There’s a whole manual on it that I found, and I will [link it in the description 00:06:12] if you care. I’m just going to note that NCBs existed, and that’s how IBM chose to implement it.
Since NetBIOS was handling network communications, it needed a way to assign and resolve names. They decided that each network adapter could have up to 17 names, one of them was permanent, and the other 16 were selectable.
Chris: That’s such a weird number to settle on.
Ned: I agree. As usual, it has to do with the fact that it’s a powers-of-two thing.
Chris: 17 is a power of 2?
Ned: 16 is a power of two, and the other one was made up as kind of like a proto MAC address. So, it used the adapter’s unique identifier, and then padded it with I think ten bits of zeros. So, that didn’t have to be stored anywhere; that was just a fundamental property of the adapter. The other 16 possible names did have to be stored somewhere, and they were stored in a special table.
Chris: Ten bits of zeros.
Ned: Yeah, it’s all ones and zeros, baby. Those selectable names, it could be a name that was unique to that particular system, or—and this was kind of interesting—you could be part of a group. So, you could share a name with other systems, and that was an early form of being able to send broadcast messages to a targeted group, something we later did with, what’s it called, IGMP, or something? I don’t know.
Chris: ICUP?
Ned: [laugh]. Hey. Keep your eyes down and straight, [jerk 00:07:43]. So, it could store these names on the local adapter, and it would also store other discovered nodes, and this was all stored in name tables. But how did a system on the network learn the names of other hosts and their corresponding addresses? By blasting the network with requests constantly, of course. That seems efficient.
Chris: When in doubt, use the megaphone.
Ned: More or less. NetBIOS could create sessions or send datagrams. That would be our equivalent of TCP and UDP respectively. That’s not necessarily what it used because this is outside of a specific protocol implementation, but essentially, datagrams could be used for broadcasts to all nodes on the network, and that was how a name was registered. If a system wanted to use a given name, it simply broadcasted to all other nodes that it was going to use that name, and if no system objected, meaning that it was already using that name, then all the systems on the network would add that name and address to their table. It’s a very noisy kind of protocol.
NetBIOS naming is completely flat and it has no hierarchy, and there’s no configured name servers that function as an arbiter. It’s just a bunch of nodes on a network yelling out names into the void. And those names were restricted to 16 characters, so as you might imagine, NetBIOS did not scale particularly well. And it didn’t need to. We’re talking about the era where a segment of a network might have five or six nodes on it. If you were a really rich enterprise, you might have, like, a room that has 20 nodes on a network. That was getting, you know, kind of big, so NetBIOS only had to deal with that number of nodes on a given network.
Chris: And this is something that you’ll find—and we’ve already talked about it with a number of different processes, protocols, philosophies around computing that came from the early days—people really never thought beyond 20 computers.
Ned: The idea that a single household, as I said last week, would have 17 network-connected devices. Was preposterous. The idea that a household would have a computer, at the time, seemed preposterous. Why would anybody want such a thing? It took awhile for people to get the message that, A, people were all going to have computers at their home, and B, they were going to want to connect those computers to something. And that was, like, late-’90s.
Chris: I mean, philosophically, I still don’t think we know why.
Ned: No, it was all a mistake [crosstalk 00:10:24—
Chris: [laugh].
Ned: And yet, here we are.
Chris: What would Kierkegaard say? Think about it.
Ned: “Someone get me out of this box?”
Chris: Not even close. Moving on.
Ned: Sidenote, buried in the documentation for the original IBM PC network reference from 1984—yes, I read it—is support for boot from LAN. That was a thing.
Chris: Wow.
Ned: I know. An IBM PC could be configured using jumpers to substitute the usual diskette-based boot process for what they called a remote program load request that would be serviced by a reserved name, and that reserve name was IBMNETBOOT. Neat. I bet that’s still buried in a protocol stack.
Chris: Yeah, I was going to say, that still totally works, doesn’t it?
Ned: It probably does. So, to try and handle the scaling problem, RFC 1001 established the idea of NetBIOS name servers and datagram distribution servers. The former would act as sort of a bulletin board of NetBIOS name entries, giving clients that were operating in a specific type of mode, a place to go and try to resolve a name instead of just blasting it across the wire. It also helps with routed networks where two clients might not even be in the same broadcast domain. The datagram distribution servers? Don’t worry about them; they weren’t really important for name resolution. Go read the RFC if you care.
So, the NetBIOS standard also introduced the concept of a scope. So, NetBIOS names could be limited to a scope, and that scope name could be tacked on to the end of the NetBIOS name, thereby creating a hierarchy that is compatible—kind of—with DNS. And here we began the uneasy alliance of NetBIOS and DNS that has persisted for far too long, thanks in part to Microsoft and their insistence on backwards compatibility. Sons of bitches. Sidenote, it really would have been nice if Microsoft had decided somewhere in the late-’90s, well, really the early-2000s to just completely ditch the older versions of the operating system and go with NT and DNS only. They could have done that with Windows 2000, and they chose not to.
Chris: I think they should have just stopped at Windows XP Service Pack 3. We all know that’s where it peaked.
Ned: Wait, they released operating systems after that?
Chris: [laugh].
Ned: Oh, so while the RFC 1001 standard established the idea of NetBIOS name servers, it was up to vendors to actually implement such a thing. And we’re talking this RFC was minted in 1987, so Microsoft was working with PC DOS 3.1 and Windows 2.0. They also had a really tight relationship with IBM, until they didn’t. So naturally, they supported the NetBIOS stack, which implicitly included NetBIOS name resolution.
The implementation of NetBIOS on Windows was called NetBEUI, NetBIOS Extensible User Interface.
Chris: Or ‘net bleah.’
Ned: ‘Net blahh,’ which if you’ve ever been deep in the selector fields for a network card in early versions of Windows, you will have seen NetBEUI. Not only that, but it supported multiple networking protocols, including IPX, token-ring, and—kind of—TCP/IP. It was called NetBEUI over TCP or NetBT because we need an acronym of an acronym of an acronym.
Chris: I mean, if you think of a better way to do it, I’m all ears.
Ned: Just name it something else [laugh].
Chris: [IAE 00:14:05], as the kids say.
Ned: Then we had the release of Windows Server NT 3.5, and Microsoft introduced the WINS service, aka the Windows Internet Name Server, which never resolved things for the internet. I don’t know why that was in there. But okay.
Chris: It was more like LOSSES, am I right?
Ned: Ahhhh.
Chris: Ahh? Ahh?
Ned: You’re funny. You’re a funny guy.
Chris: Stop muting my microph—
Ned: So, WINS was meant to assist with name resolution over NetBEUI by providing their own implementation of the NetBIOS name server concept. NT 3.5 also introduced DHCP as a service, so clients can automatically get their IP address and their WINS server settings from the network. Weee. This was not the first time they introduced domain controllers, I think that was NT 3.1, but this was right around when domain controllers and primary domain controllers became a thing. Now, you know what wasn’t included in Windows 3.5 for either the server or client versions?
Chris: Minesweeper.
Ned: Oh no, I’m pretty sure Minesweeper was on there. DNS. There was no DNS stack—
Chris: Oh.
Ned: Either [laugh]. It wasn’t until they released the resource kit for NT 3.5 that Microsoft had an implementation of DNS, and at this point, we’re talking 1995. Internet connectivity is still far from a given, and Microsoft is just trying to support the internal networks that are flourishing at all kinds of companies. They embraced name resolution offered through NetBIOS and NetBIOS name servers, and they really didn’t give DNS much more thought, right, up until Active Directory and Windows 2000. Chris, have you ever been in charge of maintaining an Active Directory environment?
Chris: I… don’t want to talk about it.
Ned: [laugh]. Spoken like someone who definitely has. So, if you know anything about Active Directory or you’ve managed it in the past, you know how absolutely critical DNS is to the functioning of basically everything in AD. Domains and forests themselves are DNS constructs, and they’re resolved as such. In fact, the way that a client finds a domain controller is all up to DNS and SRV records. I didn’t want to do a whole thing about SRV records, so I didn’t, but you know, look it up.
Chris: Don’t look it up.
Ned: God. God, don’t. If you love yourself, please don’t. But since Microsoft absolutely cannot deprecate anything, NetBIOS stuck around and many components of the operating system would default to using it for name resolution. SMB in particular, was a major offender in this and so many other ways. Listen kids, just say no to SMB 1.1.
So, while DNS was ostensibly the way forward, you still had to deploy WINS servers if you wanted anything approaching efficiency on your network. And don’t even think about disabling NetBEUI, if you know what’s good for you.
Chris: [sigh]. Yeah.
Ned: 20 years later, it’s now safe to kick NetBIOS to the curb, assuming you’re running a fairly modern stack in your internal environments. Make sure you’re doing that. But Microsoft knows that a lot of cruft still remains, like that walled up server that’s behind Fred’s desk, and that’s why WINS deprecation was only finally announced with the release of Server 2022. That’s right: decisions made in 1994 have a direct impact on operating system features 30 years later. And at this point, you can’t possibly be surprised.
Chris: I’m frankly, just impressed that I’m still awake.
Ned: Me too. Well, done. So, like I said, I thought about going into SRV records, and I decided not to because that’s another ten minutes of your life. And I thought about talking about the Name Binding Protocol that was in AppleTalk—that’s how Apple approached it—but we are already in danger of this becoming a three-parter, and so I’m just going to stop there with the DNS alternatives section.
Chris: Good plan.
Ned: Links are in the [show notes 00:18:25] if you would like to pursue further. So, the long and short of it is that DNS did in fact become the de facto name resolution standard for basically every type of network. And due to its complete ubiquity, that means it also became a target for hackers. And there have been several interesting uses of DNS for malicious motives over the years. I’m going to regale you with the use of DNS tunneling, which if you listen to our Tech News of the Week episode, you would already understand. But, Chris, are there any other DNS attacks that you want to bring up before we get into tunneling?
Chris: No.
Ned: Okay. Just checking. I know you’re the security person out of the two of us, so you might have one or two things you want to throw in there.
Chris: Well, this is the big one.
Ned: This is the big one. This is how it’s really being abused. So, if you think about how DNS is typically used, a request is sent via UDP to an IP address on port 53. Assuming a DNS server is listening on the IP address in question, it may choose to respond to that request, or it may decide to ignore the request entirely. Why not? It’s UDP.
Since DNS queries are so common, they are generally not blocked by the network firewall. And even if you do block the client from making a direct DNS request to an external host, chances are your internal DNS servers are forwarding requests in their stead. Otherwise, how in the hell would your internal clients even access the internet? Now, this is a two-part thing. They’ll do the DNS request initially, and maybe you allow that, but then when it comes to actually sending the web requests, you may have a proxy that sits in the middle and intercepts those requests.
So, if an attacker really wants to insert themselves, they have to do that during the DNS request part. That’s where this hacking bit comes in. An attacker starts by infecting a system with malware using any of the commonly known attack vectors: phishing, malicious websites, USB drives strewn across the parking lot. Don’t do that. Don’t pick up those drives and just plug them into your computers.
Chris: They’re super cheap, everybody. They’re just four bucks. They have them at Costco. That’s not a joke.
Ned: Unless, of course, they come prepackaged with malware, which that’s definitely happened a few times. Maybe go with a reputable source, spend the extra five bucks.
Chris: Fair.
Ned: Once the malware is installed, it can send DNS requests to a Command-and-Control Server. I wrote command-and-conquer, which I actually kind of like better [laugh], but command-and-control I think is the actual.
Chris: I spent a looooot of time playing Command and Conquer.
Ned: Yeah. We’ll just call it C2 for short. So, the requests look a lot like a normal DNS query, but ciphered into the request could be data that’s either being exfiltrated, or a request to that C2 server for commands that should be run to establish something more persistent. Essentially, what’s happening on the external half is that this C2 server is being registered as an authoritative name server for one or more domains that are controlled by the hacker. The infected client computer will take some piece of information and base-64 encode it, which makes it safe to use as a host name string, and then that string is broken up to make each segment meet the DNS requirements. And then it will tack on the domain that’s controlled by that C2 server to the end, so it’ll be a bunch of gobbledygook.example.com, let’s say.
And then it’s going to send an A record request to that C2 server using that string with the domain name. That C2 server will interpret that string back into whatever it is. If it’s just data exfiltration, it’ll log that information wherever it needs to. If it’s asking for a reply, to do something else on that client server, then it will respond with a similarly coded response, which in turn will tell that client to do something. So, if you’re looking on the wire or monitoring your firewall, and you see a ton of DNS requests coming from a client from the same domain with gibberish hostnames, that’s probably an infected host. You might want to be quick, though because hackers know that you’re looking for this type of activity, so the DNS tunneling might only last long enough to establish a more persistent session over another protocol.
Chris: Yeah. And also, you have to make sure that whatever protection you have in place is aware that the names are going to change all the time.
Ned: All the time. Yeah, you can’t just blacklist a domain and call it good.
Chris: That’s honestly one of the biggest problems with domains is it takes all of two seconds to set one up.
Ned: And they’re usually not just going to set one up. They’ll set up—
Chris: Hundreds. Thousands.
Ned: Yeah. And because they can place that C2 server wherever they want now, thanks to the cloud, you can’t even block by public IP address. So, how do you protect against this kind of thing? By applying a DNS filtering solution at either the firewall or the DNS server level. By inspecting requests for suspicious looking strings in the name of the query, you can effectively stop most DNS tunneling, assuming that the client is on your internal network and you can intercept those requests. So, problem solved, right? Right?
Chris: Right.
Ned: You’re adorable.
Chris: Oh. So, not right?
Ned: Well, you got two problems here. One, the assumption is that the client is using your DNS servers or going through your firewall when they’re out roaming, out in the greater world, which they’re probably not. So, if they’re at a Starbucks… [sigh] you’re still kind of screwed. So, that’s one problem. The other problem is, last week, we talked about the rise of DNS over HTTPS aka, ‘D’oh.’
DoH provides encryption of DNS traffic by establishing a TLS session with a target server before transmitting the request, and it’s become the standard for most modern browsers. That’s really good for privacy. Yay for privacy. It is less great for traffic inspection because you can’t. Because it’s encrypted.
Chris: You can’t inspection.
Ned: [laugh] [sigh]. If your DNS requests are no longer in clear text, then your firewall cannot filter them, which means malware is now free to use that channel instead. Which actually brings us nicely to Microsoft’s ZTDNS solution, and me finally landling—landing this plane, but not that sentence.
Chris: ‘landling’ is totally a logical word.
Ned: [laugh]. It’s like ladling and landing. If I had some soup that I was ladling into your bowl, I could be landling?
Chris: I think you’re making it worse.
Ned: [laugh]. All right, let’s just talk about ZTDNS. Zero Trust DNS is a new feature for Windows, and it’s currently in Private Preview. So, this is very early days; they’re still testing it out. I think you have to request to be part of the Private Preview at the moment.
The goal of ZTDNS is to lock down network access to only approved destinations while reckoning with the modern standards of DNS over HTTPS, or even DNS over TLS. So, how do you go about inspecting the traffic without breaking the encryption? You don’t. You don’t inspect the traffic.
Chris: Oh.
Ned: Yeah. Microsoft’s solution is to perform kind of the inspection before it happens by leveraging the Windows Filtering Platform that’s integrated with the Windows DNS client. You can think of the Windows Filtering Platform as kind of the equivalent of IP tables, or you know, Berkeley Packet Filtering. It kind of serves a similar function. So, this Windows Filtering Platform will maintain a list of protected DNS servers that it trusts, and by default, access to all other IP addresses is blocked, except for access to those protected DNS servers and a list of other exceptions that are supplied by the admin. So effectively, your client cannot reach an internet address, by default.
Chris: Tell me more.
Ned: How does it work?
Chris: Don’t ask questions.
Ned: Well, I’m going to answer that question. Get off my back. Jesus. When a URL like pickles.cucumbers.com is typed into the browser, the Windows DNS client will send a DNS resolution request to the protected DNS servers. If that domain is in a whitelist of allowed domains, the DNS server will respond with the IP address, which will be added to the allowed list in the WFP filter. Otherwise, the policy is to deny by default.
So, even if it gets back an IP address that comes from some other DNS server, it’s not going to get added into that filter, and so when it goes to try to access the website at that IP address, it’s going to fail. Can’t get there. Protected DNS servers will use certificate-based authentication to prove their identity to the ZTDNS client. Connectivity between the client and server is secured using DoH or DoT, meaning that the server needs to support one of those two protocols. That doesn’t mean you have to use a Windows server for ZTDNS on the protected DNS server side, but one can imagine that they’re going to make things easier if you do.
You might have caught the fact that this relies on the use of the Windows DNS client, and that browsers tend to use their own DNS client ZTDNS will absolutely break those browsers, and you’ll need to update those browsers to use the Windows DNS client instead. But critically, it does mean that you can’t do an end-run around ZTDNS by using a browser with its own DNS servers. The WPF is still going to block that traffic. ZTDNS is clearly aimed at managed devices in an enterprise and not at your home desktop, but I think it’s an interesting approach and could help combat attacks like DNS tunneling, while still making use of DNS protection mechanisms like DNSSEC and DoH. And that’s it. We did it. Two episodes, countless minutes spent on DNS to explain a Private Preview feature from Microsoft. Worth it?
Chris: Ummm…
Ned: The answer is yes.
Chris: Oh, yes. Yes.
Ned: Thank you [laugh]. Just, like, validate me a little bit alright? Hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end, so congratulations to you, friend, you accomplished something today. Now, you can sit on the couch, fire up Oracle Virtual VM, and—OracleBox? VirtualBox. That’s what I was looking for.
You can fire up VirtualBox and spin up your own instance of Windows NT 3.5. You have earned it. You can find more about this show by visiting our LinkedIn page, just search ‘Chaos Lever,’ or go to the website, pod.chaoslever.com where you’ll find show notes, blog posts, and general tomfoolery. We’ll be back next week to see what fresh hell is upon us. Ta-ta for now.
Chris: So, are you going to do a blog post of this whole thing?
Ned: I guess I’m going to have to do [laugh].
Chris: I recommend a diagram.
Ned: Many, many diagrams [laugh].