Setting up SPF

Both my web and mail server VPSes from Evorack have native IPv6 connectivity, which is really nice, but I've been pretty lazy about getting around to setting up AAAA records for them. When I finally decided to do it, it turned out that my current DNS provider didn't actually offer AAAA support, so I started looking for cheap alternatives. I eventually decided to give Route 53 a go, which is part of Amazon Web Services (AWS). Route 53 is going to cost me 50 cents per month, plus a charge per DNS query answered. Queries are charged at 50 cents per million (pro rata) for your first billion. I'm definitely not going to need a million queries per month, so overall I'm looking at less than 1 dollar a month. This is pretty damn cheap, although you can actually go even cheaper (e.g. PowerDNS do hosting for $2 per year). For this low price you get 3 nameservers in geographically dispersed locations and you can administer your records from a nice AJAXy web interface. It's all worked very smoothly for me so far and I have no complaints - .maurits.id.au web and mail should now both be accessible over IPv6.

Route 53 offers you AAAA records, naturally (although the nameservers themselves aren't IPv6 accessible, which is a shame), but I noticed that they also let you create SPF records. Sender Policy Framework is a DNS-based anti-spam measure designed to make it hard for people to spoof email addresses. The SMTP protocol actually does exactly nothing to make sure email actually comes from who is says it comes from, making it trivially easy to make your email look like it was sent by someone with any email address you want, and this is the basis of a lot of phishing scams, etc. Briefly, SPF works by using DNS to distribute machine-readable information on which hosts are allowed to send mail for which domains. If your SMTP server gets an email claiming to be from someone@paypal.com, it can look up SPF records for the paypal.com domain, learn which servers the domain owners have explicitly authorised to send @paypal.com email, and compare the IP address of the server sending the email to decide whether it's legitimate or not.

After seeing that Route 53 offered SPF records, I decided to set this up for my domain. I mostly did this because it feels like it's the right thing to do these days if you're an informed admin, and also because I thought I'd learn a few things along the way. I certainly didn't do it because I've actually had problems with spammers sending spoofed email from my domain, or because my own emails have been being caught in other people's spam filters. Actually, I've never had any problems having email from my self-managed servers be accepted at its destination. Jeff Atwood at Coding Horror wrote an article not all that long ago that makes reliably sending email on the modern internet sound like the hardest thing in the world, suggesting that if you don't have a whole bunch of things set up just exactly right that most of your email will be caught by filters and never get where it's going. Obviously Jeff is a far more knowledgeable and experienced guy than I am, and I don't mean to accuse him of making stuff up, but for what it's worth this doesn't match my experience at all. For a long time, there was only one .maurits.id.au server, and it lived in my house. It had a dynamic IP address (I used DynDNS to update my records quickly when it changed), and presumably the addresses always belonged to a block of addressed that was used exclusively for residential ADSL connections. The reverse DNS PTR records for those addresses always would have resolved to a hostname at my ISP's domain, flatly contradicting the hostname that my SMTP client would have used in its HELO greeting to receiving SMTP servers. I didn't have SPF or SenderID or DomainKeys or any of that fancy stuff set up at all. It was basically the least legitimate-looking legitimate mail server one can imagine someone actually bothering to set up, and as far as I know none of my email ever failed to get where it was going, including when I was sending it to large universities or major email providers like Gmail, places which presumably are very careful about making sure they filter out spam.

Anyway, setting up SPF is relatively straightforward. The record syntax is quite simple, and openspf.org have enough documentation up to make it pretty obvious what you have to do. There's just one problem I ran into, and that's to do with there being two different kinds of record used by SPF. When SPF was first proposed, it was defined in terms of TXT records, which allow you to use DNS to associate arbitrary text with a host or domain name. However, once SPF became popular and proved its worth, a new kind of DNS record dedicated to it was introduced, the SPF record. Some DNS servers and clients support the relatively new SPF record, but others don't and can only handle TXT records. Therefore, if you want your SPF setup to work universally, you need to either use only TXT records or use both TXT records and SPF records (just duplicate the actual content for the two record types). Do not use SPF records only. If you do this, mail receiving hosts whose DNS resolvers do not support SPF records will fail to see that you have SPF implemented at all. This is exactly what I did at first. I sent a test email to Port 25's email verification service (which is a really great thing for them to offer for free), and got a reply saying that my SPF validation passed, so I assumed all was good and left it at that. If it weren't for the fact that I didn't receive a reply in the next couple of days to an email I really thought I'd get a quick reply to, I never would have looked into it any further. But that did happen, and I got nervous that I'd botched things and my mail had been flagged as spam because it appeared to be using a spoofed address. So I tried more testing services. Unlock the Inbox also told me I passed, but OpenSPF and AppMailDev both claimed that they couldn't find any SPF information for my domain, which really confused me. Of course, the situation is obvious now: Port 25 and Unlock the Inbox are both able to use SPF records, whereas OpenSPF and AppMailDev are not. I added TXT records and now all four services tell me things are set up right, and I guess I just really haven't got a reply to that email yet.

I hope this post prevents other people making the same mistake. For the record, I'm somewhat disappointed in all four of the SPF testing services mentioned above. I mean, okay, they're free tests, I shouldn't complain at all. But it seems really lame to me to run such a service on a machine that can't even send queries for SPF records, and I also feel like the services which can see SPF records should check for both SPF and TXT and include a warning in their reports if you don't have both set up. That's exactly the kind of subtle mis-configuration that people rely on automatic tests to pick up on.