Hello, and welcome back to CS615 System Administration! This is week 6, segment 1, and we're continuing our discussion of the larger topic of networking. In last week's videos, we looked in some detail at the global view of the internet, the governance of the IP space and the administration of autonomous systems as well as some of the implications of the physical nature of the internet. But while that is all nice and well -- actually, to me at least, quite fascinating -- we also have to better understand networking on a more local level. That is, we want to understand how systems communicate _specifically_, and how we even get our systems to talk to one another, how they know where to send packets, and what those packets look like for the different protocols beyond the IP and IPv6 protocols we already looked at. So in this week's videos, we'll be tracing a very simple request and use that as an example to illustrate these concepts. Let's get started... --- What might a simple request look like? The Hypertext Transfer Protocol or HTTP is an obvious choice here, as it's ubiquitous and easy to understand. - So let's simulate a most basic HTTP request from our standard AWS EC2 dualstack NetBSD instance: We use the telnet(1) command to establish a TCP connection to www.yahoo.com on port 80 and issue an HTTP "HEAD" request. The server dutifully replies with some information and the connection is terminated. How's that for a simple request? Doesn't get much simpler than that, does it? --- So let's take this example and see what exactly happens under the hood. Superficially, it looks like, well, - the local host makes a connection to the remote host, - the local host sends some data - the remote host replies with some data - and we're done. Class dismissed. Oh, you're still here? Alright, let's try to dig a bit deeper. Let's focus - on this bit here. Just how exactly does the local host make a connection to the remote host? --- For that, let's repeat this simple request, but this time we're going to capture some information so we can then reconstruct in detail what happens on our system. We start with a tcpdump running in the background. - We'll exclude all traffic to port 22 -- SSH traffic -- since we are connected via SSH, and thus everything we type here and every output generated will lead to additional packets in the tcpdump output, so we'll exclude those. Then we clear out our ARP cache, to ensure we start with a clean slate all the way from the bottom layers. Next, we re-run the same telnet(1) command from before, but this time we're wrapping it in a call to ktrace(1). We again send the same simple HEAD request. When we're done, we stop our tcpdump collection. Let's just take a look at the packets we captured. Ok, looks like we've got a bunch here. Good. --- But so wait, we ran the telnet(1) command wrapped in ktrace(8) -- what's ktrace(1)? ktrace(1) is a tool that you can use to trace other commands to let you see what type of system calls it makes or what I/O it performs, for example. This allows you to really understand what happens when you run a command. On other Unix versions you will find similar tools, such as strace(1) on Linux, or dtrace(1) on, say, OmniOS. The output from ktrace(1) is in a binary format, so we use the kdump(1) tool to read it. Here's what that looks like. As you can see, there are all the different system calls the application makes: you see it opening some shared libraries and mapping them into memory, open some files, etc. Let's see what files it opens up under /etc, since that is where we know we have many system configuration files. Alright, so that looks interesting: the first two are not surprising, since telnet(1) is a dynamically linked executable. The files we care about here are the next files. Let's start with nsswitch.conf(5). What does that file do? Well, we're on a Unix system, so we don't need to guess nor Google, but instead can simply look up what this file does in the manual pages. Ok, so the manual page tells us that this file controls how internal library functions look up certain pieces of information. nsswitch.conf can specify certain sources where information can be found, such as other static files or an external service, like the DNS. The things nsswitch.conf can look up include groups, hosts, user information and so on. So what does our nsswitch.conf file look like? Here we see the default configuration for the standard C library resolver, telling us that if an application wants to turn a hostname into an IP address, it should first look in the local files and then, if that doesn't yield a result, fail over to the Domain Name System, or DNS. Ok, so what's the next file our applications looks up? Here we see it opening /etc/hosts -- what does _that_ file do? Again, the manual pages provide the answer. /etc/hosts is the host name data base, and was indeed, before the Domain Name System was introduced, _the_ central method to look up a hostname. You're probably familiar with it and know that you can add entries here to point a name to an address, and because this file takes precedence over the DNS, your application will use whatever information it finds in this file. Our file looks like this: By default, this file only contains entries for 'localhost'. Now /etc/nsswitch.conf told us that if we don't find an entry in /etc/hosts, we should ask the DNS. How do we figure out which DNS server to use? Looks like our application looks in /etc/resolv.conf next, so once again let's bring up the manual page: As you may know, /etc/resolv.conf now specifies how DNS lookups are to be done when an application calls one of the resolver functions. Most importantly, this file contains the IP address or addresses of the DNS server or servers that the system should use. Now our /etc/resolv.conf looks like this: Looks like our nameserver is 10.10.0.2, so if we are given a hostname that we couldn't map to an IP address via /etc/hosts, then we should go and ask this server. Let's see what we find in our kdump(1) trace output. If we look for what sockets our application opens up, we see that, over here, it opens a socket in the IP domain of type _datagram_ and then connects to the IPv4 address 10.10.0.2 -- our DNS server. The data it sends below is then our first DNS query for the name "www.yahoo.com", and the response we get back is shown a few lines below. We then see a second connection to the DNS server, and after that, a new socket in the IPv6 domain of type TCP to this IPv6 address here. After that, telnet(1) prints some information to the terminal, reads input from the user and then sends that data over the socket to the remote host. A bit further down, then, we see how we are receiving the data from the remote host. --- So we saw that our simple step of connecting to a remote host --- actually broke down into a few steps, namely - one step where we take the given hostname and turn it into an IP address, and - one step where we are making the connection to the remote system and communicate with it, --- but of course we then _also_ saw that the step of turning the hostname into an IP address broke down as well into several steps, namely - at first, a step that determines just how exactly it's supposed to convert the hostname into an IP address, by asking /etc/nsswitch.conf; - then by looking up the hostname in /etc/hosts, and, upon failing, going to the DNS. But for that, it first needs to - look at /etc/resolv.conf to determine what DNS server to ask, then - open a UDP socket to that server, send our query and hopefully get a reply, before we can _then_ connect to the remote host. And so a simple request like the one we started out with suddenly doesn't look quite so simple any more, especially when considering that so far we've really only --- looked at what happens on our local system, and have not yet looked at the network packets we're sending. Remember, we had initially captured tcpdump output as well, but we never looked at that. But don't worry, we'll make up for that in our next video, where we'll then trace those packets. --- Before you move on to the next video, though, here are a few exercises for you: - Repeat this exercise on an Ubuntu Linux, a RedHat Linux, and an OmniOS instance. On those systems, you will need to use a different tool from ktrace(1), which is a BSD tool. On Linux systems, you should find the strace(1) command, on OmniOS the dtrace(1) command. They both behave a little bit different, but provide the same information, letting you trace an application and see what exactly it does. But each OS will also use perhaps different configuration files, so make sure you can track how exactly the hostname is resolved and the connection made. - In our example, we saw connections being made to IPv4 and IPv6 systems -- but it might be interesting to compare to an IPv6-only system. Configure an instance to be v6 only and see what's different. - We mentioned that hostname lookups start in /etc/hosts -- verify that you can circumvent a DNS lookup by adding an address in that file and tracing the application. - And finally, if you look at the ktrace output, you might have noticed that we made two separate connections to the DNS server. But we only resolved a single hostname, didn't we? Why is that? Alright, with all that, as promised, in our next video - we'll look at the network packets we captured using tcpdump(1). Until then - thanks for watching. Cheers!