Hello, and welcome back to CS615 System Administration! This is week 3, segment 3, and we'll try to wrap up our discussion of the Domain Name System. Now obviously this will leave out many discussions, since a topic as vast as the DNS cannot be adequately covered in just a few short videos, but I hope that we'll have provided enough of an overview to ensure that you are ready to troubleshoot DNS related issues with a sufficient understanding of the underlying infrastructure and beyond what most people know about name resolution. That is, as with all the other topics in this series, we're shooting for just enough detail to let you know how much more there is to understand and hopefully whet your appetite for more. --- But let's go back to where we left off in our last video. Remember, we went through the details of a complete hostname lookup from a DNS resolver, starting at the root nameserver, then hitting the gTLD nameserver, then the nameserver authoritative for the yahoo.com zone down to the delegated yf1.yahoo.com zone. We followed the packets for this lookup because we wanted to avoid having to defer to magic steps in between. --- And so we determined that we really need to begin our query way at the top of the domain name space: the root. But let's be honest: we kinda cheated a bit here, didn't we? We said: "Let's ask the root!" --- but... we never specified just how we knew where to find the root. That is, we _did_ defer to an initial step of magic, which seems unfair. So let's talk about the root of the domain name system. We know that the root nameservers have IP addresses, and that they really cannot be all that different from any other authoritative name server: in the end, they are simply nameservers authoritative for a single zone. It just so happens that that zone is the root of the domain name space. But if this is just another nameserver and the root is really just another regular zone, then --- ...we should be able to look up the nameservers responsible for the root zone via the DNS. And look at that, that works. Here we see that dig(1) tells us that when we look for NS records, we get back 13 answers, as well as an additional 27 records the nameserver thinks might be useful to us. The answer section here shows the various root nameservers, named a.root-servers.net through m.root-servers.net. [pause] That is, we have 13 root nameservers in total. Why 13? Well, it turns out that 13 NS records fit neatly into a single 512 byte UDP packet, so having 13 nameservers means you only need to send a single packet back. Now of course nowadays the additional information we're including balloons the packet above the 512 bytes, which is why you'll find in your tcpdump packets that the response was truncated and your resolver will retry the query over TCP. But anyway [continue] in addition to the NS replies, we also get the IP addresses of the nameservers, which of course are all dualstack IPv4 and IPv6. So when we ran 'dig', it queried our local resolver for the root nameserver NS information, [pause] and presumably the nameserver did perform an NS query just like we discussed in our last video. But that means that we have another chicken-egg problem: if we want to ask the root server about the NS records for the root server, we need to know what the root servers are! And so it is indeed the case that every nameserver installation does require this information to be provided so as to be able to bootstrap itself. [continue] And here, under /etc/namedb/root.cache is where we find that information on this particular system. This file is part of the 'named' software distribution -- ie BIND -- but, as it says here, is also available from the internet via FTP from the InterNIC server. In other words, when you set up a new resolver, you need to somehow get this information. Now this information may be outdated, so once you start your server, it will immediately look up the nameserver information for the root again itself and then cache that, possibly overwriting information from this file. But here we have the NS, A, and AAAA records for all the 13 root name servers. Now let's compare to what we find on InterNIC... and there we go. We see that this file is current as of March 17th and looks pretty much identical to our copy. Now this is no coincidence: the root nameserver information doesn't change all that often, since, as you can imagine, that could lead to a number of problems when new nameserver installations come up with an outdated cache file, but either way, we _are_ once again relying on copying around what amounts to a slightly more advanced hosts table, just like when we started out in the early days of the internet. Funny how the more things change the more they stay the same, huh? But let's get back to the fact that we have only 13 different root nameservers. That seems a bit fragile, doesn't it? I mean, sure, there's _some_ redundancy, but if 13 servers are the load-bearing backbone of the DNS for the entire internet, then any sufficiently motivated adversary could try to take them down and bring the internet to a standstill. I mean, DoS'ing 13 servers really shouldn't take all that much, right? --- Fortunately, there are more than just 13 root nameservers. What we saw up until now is that there are 13 root nameserver _authorities_ -- the a root, the b root, etc. all the way until the m root. But each of these roots are comprised of dozens or hundreds of individual servers. If you go to root-servers.org, it'll show you a nice map of just how distributed the root authorities are: For starters, the 13 roots are operated by 12 independent and international organizations, so as to ensure that no single country has control over the domain name space. Each of these authorities then distributed their servers globally, as shown here. If we drill down into the US, we can see, for example, that right here in New York we have 9 root server instances, the A, C, D, E, F, and J roots, respectively. The footprint of the different roots is global, but not all roots have servers in the same locations. For example, up here in Nuuk, there's only a K-root, operated by RIPE, with many more varied distribution again here in Europe. [pause] As you can tell, the different root servers are operated by different organizations, and each organization may deploy their respective root servers in different locations. Based on our earlier discussion around peering, it shouldn't come as a surprise to you that many of the root servers are co-located in the central network hubs, to ensure physical vicinity and thus reduced round-trip time for queries. The E-root, for example, is operated by NASA [continue] and is co-located all across the globe in over 160 locations. The B-root is operated by the University of Southern California, but served from different peering points around the world as well. Note that operating of the root servers is not a business -- it is provided by the root server operators by and large as a public service, even if operated by a commercial business, such as Cogent, which runs the C-root. The F-root is run by the ISC, and if you've been wondering just how exactly we are able to have over 1300 individual name servers but only 13 IP addresses -- well, 26 with each having both an IPv4 and an IPv6 address -- then the answer is given here: the root servers are all using anycast to allow multiple systems to offer the same IP address and to let the routing algorithms determine the closest one, generally as determined by number of hops. Now if each root has only two IP addresses, but may reach any number of different servers, how do you know which one you end up talking to? Well, by and large it doesn't really matter, --- but there are ways to find out. For example, the F-root, operated by ISC, offers a lookup like this: If you ask f.root-servers.net for the "hostname.bind" text record using the chaosnet protocol, you get back a response that includes as the first label a three-letter airport code, telling us that "our" f-root is located at least somewhat near Dulles International Airport in DC, which makes some sense, [pause] since our AWS instance here is in the US-East region, which is served out of datacenters in or near Ashburn, Virginia -- that is, right next door to Dulles International. Now this is result here also illustrates something else that's useful to remember about the DNS: it is a distributed database, so can be used for other things beyond just IP addresses, and, as the use of the "chaosnet" protocol here shows, even in other contexts. By using resource records of type 'text', for example, we can put any information into the DNS we want, really. We've already seen a number of different types of resource records [continue} such as NS records, which again use the internet or IN class of records. Each resource record also has a time to live, or TTL, which describes for how long a caching resolver may cache the result for. The NS records for the root servers are long-lived -- roughly 5 days -- since they are not likely to change all that frequently, but other records, especially those that may change more frequently, often times have a shorter TTL, which you have to keep in mind when troubleshooting why you are getting different results from those that may be in the authoritative zone. Another resource record is the "SOA" record, for "start of authority". This record defines information about the zone, and includes the name of the primary master, the email address of the primary point of contact, the serial number, a refresh timeout for secondary name servers as well as a retry, expire, and TTL number. Note that the authoritative name servers for a domain need not be _under_ that domain, and in fact, as we show here, some DNS service providers even distribute their nameservers across multiple TLDs for redundancy in case one of _those_ becomes unavailable. It's also worth noting that two domain names that look similar and that we associate with one another remain entirely distinct and can have completely different authorities -- stevens-tech.edu is different from stevens.edu. [pause] Ok, what other DNS resource records do we have? Since the DNS is such a conveniently distributed database, people have found all sorts of uses for it. For example, there's CAA records, [continue] which provide information about which certificate authorities are allowed to issue certificates for the given domain. We'll discuss this again when we talk about the https ecosystem -- here, we see that Yahoo only allows DigiCert and GlobalSign to issue certificates under yahoo.com. Or you can just put some humorous text into the DNS for your silly joke domain... ...or you can put your host's SSH key fingerprints into the DNS, so that clients can verify who they're connecting to instead of simply accepting any host key mismatches as just about every single person on the internet does. But of course the most common lookups remain the resolution of hostname to IP address as well as the other way around. [pause before tcpdump after www.cs] But reversing the lookup -- going from IP address back to hostname -- is somewhat interesting, in that it uses different domains. Let's take a look and [continue] go ahead and capture traffic once more: The reverse lookup of an IP address uses the PTR resource record, and the result then looks like this. Note how the IP address was reversed and then looked up in the special "in-addr.arpa" domain. But how would in-addr.arpa know what Stevens decides to name its hosts? --- Let's take a look at the packets here in Wireshark. As expected for any lookup, we begin at the root and ask "Hey, who's responsible for 'arpa'?", with a response pointing us to a.root-servers.net. Next, we ask that server "hey, who's responsible for in-addr.arpa?", and the response includes a list of nameservers called a.in-addr-servers.arpa, b.in-addr-servers.arpa and so on. So far, so good - not much different from the other lookups we've seen. Now we get into the part where we used the reversed IP address and ask "who's responsible for 155.in-addr.arpa?", which tells us some nameserver run by ARIN is in charge. And perhaps this is where we realize that the reversed IP address allows us to effectively include a whois lookup by netblock -- because the 155/8 ip block was allocated to ARIN, it's ARIN that knows where to find answers for reverse names. So of course we then ask ARIN for 246.155.in-addr.arpa, which then tells us it's nrac.stevens-tech.edu that knows more. This is because the 155.246/16 netblock was delegated to Stevens, as you remember from our earlier videos, and so the 246.155.in-addr.arpa zone is also delegated to Stevens, thereby allowing Stevens to add entries into that zone as they see fit. So when we then go up the reversed IP address by octet... we eventually find our final PTR query here, with the response returned by nrac.stevens-tech.edu. So reverse IP address lookups work almost the same as any other DNS lookup, but it requires the reversal of the address to allow for delegation of zone authority along the netblock allocation lines. Neat! --- But we still have one problem left, so yes, here we go again with the packet capture: Let's assume we're looking up the IP address of www.iana.org. If we look at the packets we captured, we of course see our DNS UDP port 53 traffic sending our queries and the servers replying with the various responses. But... now how do we know that these responses are authentic? This is simple, plain UDP, so any system in the middle could have sent us bogus information, right? Let's look at the results dig(1) returned to us once more. Note over here, it includes the "ad" flag -- authenticated data. Somehow the DNS server got assurance that the data is authentic, that it was not manipulated. How did it do that? Let's go again: We capture our DNS packets, flush the cache to make sure we get the whole story and repeat the lookup. We can then --- load the pcap file into Wireshark once more and take yet another, even closer look at the packets we captured. Here's our initial query to the root nameserver about .org, and the response... includes not only the NS records, but also, down here, an RRSIG record -- a resource record signature. This is a cryptographic signature asserting the integrity of the entire NS record set. If we look further down... we see that the other responses also contain RRSIG records as well as DS records -- Delegation Signer records used to identify the signing key of the zone. The final result, then, _also_ includes an RRSIG, but of course now the question is: how do we verify this signature? We'll need to have a public key for this. How do we get the public key? Why, the DNS is a distributed lookup table, let's ask for it! So here we see the lookup of type DNSKEY, which... should be in this response here. But this response was truncated -- the information didn't fit into a single UDP packet -- so we need to try again using TCP here you see us asking again, this time via TCP and the response is then found over here. So now we can use this DNSKEY to verify the signature on the earlier results, but of course we want to have assurance that this record itself was signed, using the parent key, and so on and so on all the way up to the root. This system of signing records by different zone keys is known as the Domain Name System Security Extensions, or DNSSEC, and as you can tell we could discuss that topic by itself in several stand-alone videos. --- But for today, let's stop and just summarize some of the important implications and considerations when dealing with the DNS. For starters, and that may be obvious, the DNS contains an incredible amount of critical information, and is often times used for all sorts of subtle decisions that assume its correctness and authenticity. Unfortunately, the DNS does not provide any such guarantees -- plain text UDP or even TCP without any assurances means that all sorts of things can go wrong. To address this, DNSSEC was developed, which we just saw in use, but - unfortunately it's not widely deployed. There are comparatively few domains that use DNSSEC, in part because of practical deployment concerns, in part due to precisely the importance of criticality of the DNS: if you mess up DNSSEC, your entire domain falls off the internet. That's a pretty big risk, and we need to see more robust deployment practices here. - Next, you may have heard a fair bit of talk about DNS-over-TLS or DNS-over-HTTPS in the last few months -- these address different problems, or, better: threat models within the DNS system, and are topics we could likewise cover in depth for hours. For now, I hope that you will research them on your own, applying what you've learned here. - Another aspect about the DNS is that it's everywhere and seldom restricted. This allows for all sorts of interesting abuse angles, such as data exfiltration via DNS lookups, for example. - Misconfiguration of the DNS can lead to endless hours wasted, and troubleshooting can be tricky due to the nature of the DNS, allowing for delegation and - different TTLs on records. As SysAdmins, you'll soon learn that if everything's all messed up and nothing makes sense, it's probably the DNS. Or more likely: somebody thought they were clever and added something to /etc/hosts, which I never grow tired of recommending against. - But as the DNS sits underneath so much of what we do on the internet, you need to remember that if I can control your DNS, I can likely do just about anything that otherwise would require very privileged access. I think we mentioned this when we talked about domain name registrars in an earlier video. Now given how important the DNS is, it's a service that is frequently outsourced to a third party provider, but - you want to keep in mind that every time you do that, you are giving up some control, and if that provider gets compromised, you are compromised as well. On the other hand, however, - it's not necessarily the case that you are in a better position to run such a critical service with sufficient redundancy yourself, and perhaps it _is_ better to let the experts manage it for you. This is a pattern we see over and over when it comes to running services in-house versus outsourcing them, and there isn't a clear, simple, always correct answer: what makes sense for one organization may not make sense for another. Ok, I think this brings to an end our discussion of the DNS. As you can tell, it's a topic that lends itself for many a deep dive and can go into many different directions, but is critical to understand as without the DNS, nothing much works on the internet. So make sure you follow along with all the examples and get busy dissecting pcaps in Wireshark and by hand; explore the different domains and see what other interesting things you can find out just by observation. Until the next time - thanks for watching. Cheers!