Hello, and welcome back to CS615 System Administration! This is week 8, segment 2, and we'll continue our discussion of the mail system and how emails are delivered. In our last video, we had looked at the Simple Mail Transfer Protocol and observed how it delivers the mail from a client to a mail server. We had used tcpdump to observe the packets and saw how the protocol lets us send a mail using, for example, the telnet command by connecting directly to the given MTA. In this video, we'll now take a look at the receiving end to see what happens when a server accepts the mail as well as talk about ways to protect the email information in transit from being intercepted or observed by an adversary. --- So let's take a look. Here we have three windows. Up here in the top one, are running a tcpdump on our receiving mail server, looking for packets from our AWS instance as well as DNS lookups. Here in the middle, we tail the mail logs on the mail server. Let's just clear up any current messages so we can start fresh. Then down here, we will now send our email, just like we did before. We compose our content, knowing full well that everything we type here is being sent in the clear to the remote side. And there we go - as soon as we finish our mail do we see the connection made, the packets flow, and the mail server log activity. So let's go up there now and look at what the mail server sees. Let's scroll back up all the way to the beginning. Ok, here we go. We first see an IPv6 reverse lookup -- as you will recall from the last video, our mail server will first try to send the mail using IPv6, but since our IPv6 address does not have a valid PTR record, looked up here, the mail is rejected. Next, we see a additional DNS lookup being performed based on what our client attempts to deliver; we'll see those repeated further down, so let's skip over these here. Alright, so here we then see the IPv4 connection from our AWS instance, and then, not surprisingly, we see the reverse PTR lookup of that IP address. We then send our 220 helo message and receive the ehlo from the client. As soon as we get the "MAIL FROM" message, we are then going to look up the MX record for the domain that this message purports to be from. After that, we see another DNS lookup in a domain named "spamhouse.org" -- we'll get back to that later when we talk more about Spam and abuse detection. Over here, we see that our server accepted the mail and lets the client know, upon which the client disconnects. But our work here isn't done -- as you may recall from our last video, we know that this mail server hands the mail off to "spamassassin", which performs a number of DNS lookups that we see here. Again, we'll discuss those in our next video, but do take note of the frequent use of the DNS for all these. If we look at the mail logs now, we see the same messages as we'd expect and as we had seen the last time: the IPv6 connection that we rejected the IPv4 connection that we do accept and let deliver mail the hand-off to spamassassin and the final delivery into procmail. Alright, so far, there's nothing unusual here. The various lookups on the server may seem surprising, but I promise we'll get back to those in the next video. --- But now, let's run through this once more, this time manually: Note that here, when we send our EHLO message, the server provides us with a list of additional features it supports, and one of those is "STARTTLS", which - provides a form of opportunistic transport encryption by allowing the client to upgrade this plaintext connection to an encrypted connection by, well, starting to use TLS. Anyway, so - why don't we give this a try? Ok, cool, looks like the server is ready, so now... uhm, damn, I forgot how to speak TLS. I don't think I'll manage a TLS handshake manually over telnet, so good thing the server aborts here. But ok, how _do_ we begin a STARTTLS connection then? Let's use our trusty 'openssl s_client' command, which supports the "-starttls" option. Here we go. Nice, now we're connected with a TLS connection between the server and our client here. And now when we say "EHLO", note that the server _no longer_ offers STARTTLS, because, well, we already are speaking TLS. So now when we send our various SMTP commands here, you'll notice that in the upper window our packet capture no longer is able to display the contents of the packets, only letting us know that the traffic is of type SMTP. So now, knowing that our connection is encrypted in transit, we can send confidential messages, resting assured it cannot be read by anybody capturing the packets in between our systems. Let's scroll back up here to take a look at what the openssl s_client command tells us. As you can tell, this all looks very similar to a common TLS connection to, say, an HTTP server, with the usual certificate chain, server certificate, and the negotiated session key and all that jazz, eventually showing us that our client was even able to verify the certificate the server presented. --- So what we saw here, then, was the communications between the sending and receiving MTAs: - The client or sender begins the connection and sends an EHLO command, to which the server - replies with a list of options. The sender selects - "STARTTLS", the server - replies, - the two perform a normal TLS handshake, and then - the remaining communications between the two are encrypted in transit. Excellent! But so why did we call "STARTTLS" "opportunistic" encryption? And what are some problems with this approach? To understand that, --- let's consider what things look like when we have an adversary in the middle. In this scenario, we envision an attacker able to intercept our packages between the client and the server, so that - when the client sends the EHLO, it will - replay that to the server, but when the server - replies with the options it supports, the man-in-the-middle can then - take those options and relay them to the client, but in addition, the attacker is then - able to strip out the "STARTTLS" option, thereby giving the client the impression that this server cannot speak TLS, thus causing - it to speak plain SMTP, and thus allowing the attacker to continue to observe the traffic in the clear. So this is clearly not desirable! But SMTP does not provide for any means to start out immediately speaking TLS, and a mail server may have to talk to hundreds or thousands of remote systems -- how is it going to know when a receiving mail server supports TLS? --- Wouldn't it be useful if there was a way to look up whether a given domain wants to use TLS for all its MTAs? Well, what do we do when we want to look up something? Why, we ask the DNS! So here are performing a TXT lookup against the domain in question with a special prefix: "_mta-sts", which stands for "MTA Strict Transport Security". If your domain has such an entry, it is expected to specify the current version of your policy as well as an identifier for this policy version, as you see here. But... now what? If you've determined that a domain has an MTA-STS policy, then you can retrieve it by looking for the file "mta-sts.txt" under the well-known path of the service named "mta-sts" under that domain. Such a policy then might look like this: In this, you can specify your MX servers covered by this policy, for how long the client should cache the policy, as well as in what "mode" the client should operate. [pause] - RFC8461, which specifies MTA-STS, defined the different modes, allowing mail server operators to carefully roll out this policy via a "testing" mode that then allows the remote client to simply send a report to the server operator if it finds any violations. In "enforce" mode, as shown here, we are telling the world that this MTA expects to always use STARTTLS, so if you're a client and you connect to this service, and somehow it doesn't offer you "STARTTLS", then you should abort, because you know something's fishy. - Alright, so that's pretty sweet: we can know whether a should be talking TLS or not. But we still have another issue, which is kind of at the root of the x509 PKI we use for TLS: how do we know that we can trust the certificate we are presented? Normally, we verify against a root trust bundle, but let's suppose that for whatever reason we can't -- or don't want to -- do that. Or perhaps let's consider that the evil man in the middle was able to issue a certificate for the target domain that _would_ validate via our trust bundle, but isn't in fact the one we want. We can simulate the case of an invalid cert by removing our trust bundle altogether... So then, our connection using STARTTLS will... tool tells us that the cert is no good. Now note that here we are _still_ able to talk SMTP -- that is, the failure to validate does not prevent us from sending mail. To do _that_, we'd have to make a decision to abort. But anyway, so now, without a trust bundle, how would we know whether the certificate presented by the server is authentic? Well... what do we do when we want to look up some information? We ask the DNS. Again. If we make a DNS lookup for a "TLSA" record using this special name, encoding the port and protocol that we're interested in -- i.e., TCP port 25 for SMTP... ...we might get back a record like this. [pause] This type of record is part of the so-called "DANE" standard, which is defined in - RFC6698 and stands for "DNS-based Authentication of Named Entities". Using this standard, we can assert the authenticity of, say, a TLS certificate without the use of a PKI. That is, the TLSA record here includes the SHA256 checksum of the public key of the certificate, and because we used DNSSEC, as shown here by the "authenticated data" bit shown in the "dig" output, we know that this information is accurate. So now with this information - [continue] we can add these flags to the s_client(1) command here, pass it the TLSA record, and then our connection again becomes fully validated and authenticated. If we scroll back up here, we see that the openssl command tells us that the validation happened via DANE. --- Alright, so once again I hope I was able to spark your interest in a topic that is obviously much more complex than what we can cover here in this video, but let's quickly recap what we _did_ discuss: From our last video, we remember that SMTP is a clear-text protocol, so that in order to protect communications in transit, - we can use "STARTTLS", which allows the client to begin opportunistic encryption with the server if the server offers it. This is pretty neat, but as we know, TLS also allows us to get - authenticity guarantees, since we can validate the certificate and thus establish that the server we're talking to is indeed the one we _think_ we're talking to. But since STARTTLS is merely an option offered by an MTA, we also saw - that an adversary in the middle could simply strip it out, and we wouldn't be none the wiser. One mechanism to protect against that threat is - MTA Strict Transport Security, in which we use the DNS to look up information about our mail server, poll a web server using https, and then can determine whether we are supposed to talk TLS with the mail server in question. We also discussed another threat, however: - that of an attacker posing as our mail server and presenting an invalid certificate, and us looking for a mechanism to validate the certificate without relying on the x509 PKI. To address that, - we saw the use of the DANE protocol, which, once again, utilizes information from the DNS to validate the remote certificate. Now with all these mechanisms, we have a good level of assurance that the system we're talking to is the one we want to talk to, but this still poses one final dilemma: what to do if we are running into a problem? - Should any of these possible failures then prevent us from sending mail? If we do not fail closed here, then all these assurances are really not worth much, but on the other hand, experience has shown that it's easy to make mistakes and having a wrongly deployed certificate lead to mail not being delivered can be a high cost. For this reason, you'll find only few services on the internet configured for enforced MTA-STS, for example. Google recently -- within the last year or two, I think -- did enable this, but most of the other big email providers are still operating in "testing" mode, where they collect failure reports. Unfortunately, not many services on the internet utilize DANE, which of course is a result of DNSSEC not being widely deployed, as we had discussed in a previous video. As so often, I'm afraid I can't provide easy solutions, but I hope that with this video you learned a bit about how to protect mails in transit. We'll still need to discuss the remaining issues we had hinted at earlier: we've established ways for the email client to assert the authenticity of the server, but we have not at all addressed the problem that any information sent by the client can be completely made up. So in our next video, let's dig into how we can protect against having our poor mail server get abused to Spam our users. Until then, thanks for watching -- cheers!