SPF breaks email forwarding!

As described here, a little less than a month ago I set up SPF for the maurits.id.au domain. Tonight, I am removing it.

A little while ago I sent an email to a colleague from the University of California at San Diego, using their @ucsd.edu address. Over a week later I heard back with an apology for the late response - he'd only just found my email in his spam folder. This concerned me greatly, since I completely manage the mail services for my domain and if legitimate messages are getting flagged as spam due to a misconfiguration on my end, then I want to know about it and fix it ASAP.

I asked him to send me the mail headers for my message to see if I could figure out what went wrong. He did so, and explained that he had his UCSD mail account set up to forward mail to his personal Gmail account, in case that was relevant. It turned out to be crucial. From the perspective of Google's mail servers, my mail came from a UCSD machine and, obviously, neither that machine's hostname nor IP address were mentioned in my SPF configuration. Google dutifully checked my SPF records and came to the conclusion that my mail had a forged From address, consequently flagging it as spam.

I was a bit baffled when I realised what was going on. On the one hand, Google seemed to be using SPF exactly as intended. But on the other hand, email forwarding is so incredibly normal that I took it for granted that a technology like SPF must be compatible with it. Maybe the UCSD server was supposed to add a header indicating what had happened so Google could take the forwarding into account and not do an SPF check?

Well, no. I did some searching and it turns out that SPF breaks email forwarding. Just like that. The SPF advocates know that SPF breaks forwarding, and they openly admit as much. Their attitude is that everyone operating a mail server that might forward email should deploy something called SRS, which does "remailing" instead of forwarding, and plays nicely with SPF.

I am astonished and disappointed that a system with such an obviously boneheaded shortcoming - that it completely breaks a long and well established aspect of email in a way that requires everybody, including people who aren't even using SPF themselves, to reconfigure their servers - could receive as much mainstream attention and support as SPF has. This is just a completely untenable situation. I have no way of knowing in advance if I'm going to have this problem when I email someone for the first time, so unless I pre-emptively add entire domains to my SPF list, this is guaranteed to happen again some day. Doing that is totally impractical and also largely defeats the purpose of SPF because I'm then just explicitly authorising entire domains to forge mail from my domain. It's absolutely not my place to ask other people or the admins of their employer/school's mailservers to change anything about the way they handle mail unless they're violating a core RFC, which people who haven't deployed SRS are absolutely not doing.

Of course, this exact issue (people forwarding email to Gmail and SPF breaking) has been discussed extensively all over the web. Disappointingly, the most often recommended solution seems to be for either the sender or receiver or both to drastically change the way they handle their email so that Google is more intimately involved in the whole process. This is not a rational response to an obviously broken technology: the obvious response is simply to stop using SPF. I'll be doing so, and looking into alternatives like DomainKeys.

Lesson: don't jump on tech bandwagons without doing research first (if I'd done research, I may have found articles like Why you shouldn't jump on the SPF bandwagon, or SPF is broken and must Die). Of course, I already knew this and abide by it in general, but I figured that a technology that is only visible/relevant to people who administer mailservers, and had achieved enough acceptance that people were writing support for a whole new class of DNS request into nameservers, would be an exception. Apparently not.