Skip the bollocks and show me how it's done
Some people have started to restrict the IP addresses from which mail from certain domains may be sent. See the pobox SPF site for more details.
This breaks mail forwarding, as discussed elsewhere on the pobox site.
Basically, the problem is that when we forward mail which originally came from a domain publishing SPF records, the ultimate recipient might reject the mail on the grounds that we are not a permitted sender for the original domain in question.
This demonstrates a fundamental misunderstanding on their part about the way the world actually works, with respect to mail forwarding and virtual domains.
To avoid this problem, when we're forwarding a mail to a recipient domain which we know is broken in this way, we invent a new one-time address at one of our own domains and we forward the original mail so that it appears to come from our new address. If the mail subsequently bounces, we get a bounce destined for our invented address, and we forward it to the original sender of the message in question.
It can be noted that generating one-time addresses is an idea fairly much orthogonal to SPF. They're not really related except in that SRS is proposed as a workaround for the brokenness of SPF.
SRS on its own, however, does look like a potential solution to the original problem which SPF tries to solve; the problem of faked senders in MAIL FROM:.
I have many old email addresses which are no longer active, but which may occasionally receive a valid mail from an old friend, so I keep them alive. Since there is never any valid mail sent from those addresses, they are configured never to accept bounces. It cuts down on a lot of collateral spam from people with misconfigured virus checkers who send reports to the supposed 'senders' even of a virus which is known to fake its sender.
This also means that third parties implementing sender verification callouts, to check that a bounce would be accepted to the address from which the mail they're being offered claims to come, will find that a bounce would not be accepted, and will therefore refuse to accept the mail with those addresses faked as the sender.
By using SRS, it's possible to protect even a current, valid email address in this fashion. I can use SRS on all my outgoing mail such that I no longer send mail with, for example, 'dwmw2@infradead.org' as its SMTP reverse-path. Then when all machines are updated to do SMTP AUTH via the SRS-capable servers, I can start to reject bounces to that address.
(NB. If this concerns you in any way it's probably because you need to note the difference between MAIL FROM: and the From: header.)
Now, we have effectively solved the problem which SPF set out to solve. Anyone who actually bothers to do callouts will no longer accept mail from any address protected in this manner unless it's actually genuine.
No Brave New World is envisaged; no workarounds which need to be implemented by every uninterested third party on the Internet who's still running a 1980s version of sendmail behind a firewall. It just works between participating hosts, end-to-end.
It should be noted that you need to allow not only bounces but also mail from postmaster@ to your 'SRS' addresses, since some sites do callouts with 'postmaster@<theirdomain>' as the source, rather than '<>'.
It should also be noted that you may theoretically lose the occasional mail from a broken autoresponder which sends its bounce to the From: address rather than the reverse-path. Since the collateral damage from this kind of broken autoresponder is what we're partly trying to avoid, and since this kind of broken autoresponder is, well, broken, that's a policy decision which I personally am happy to accept. It's certainly less onerous than the false negatives which SPF has.
The way we generate the address is by using a dedicated domain for these bounces and including the following information in the local part of the address:
These addresses take the form: SRS0+<hash>+<datestamp>+<domain>+<user>@<hostname>-<time>.srs.infradead.org. There's a wildcard MX for *.srs.infradead.org.
When we get a bounce to one of these addresses, we check its hash is valid and the datestamp hasn't expired, then forward to the original address.
This is how it's done, in Exim 4. These routers go immediately before the 'lookuphost' router which sends messages by remote SMTP. Note that you should make sure you accept 'postmaster@', but in my case that's done by an earlier router.
You can set '-DSRS_DEBUG' on the command line, but don't ever set it in the configuration file because it'll freely give out the correct hash to anyone who gets it wrong. It's good for getting addresses to test, though.
The configuration given below requires two configuration files, placed in the directory named in the macro CONFDIR:
always-srs-sendersThis text file contains a lookup database of sender addresses for which we should always perform SRS. See §10.16 of the Exim manual for more details, but basically each line contains a domain followed by a colon-separated list of local parts at that domain, possibly negated and possibly including '*'. It might look something like this:
infradead.org: !postmaster : * slowglass.com: dwmw2spf-afflicted-domains
This is a lookup file for those domains which we have discovered are implementing SPF checking, which we absolutely must succeed in forwarding mail to, and which have refused a polite request to correct their misconfiguration. It's a simple lookup which returns text containing any or all of 'all' to cause us to rewrite all addresses when forwarding to the domain in question, 'spf' to cause us to rewrite only sender addresses where there is an SPF record published, 'cid' to rewrite addresses for which these is a Caller-ID record published, and/or 'self' to rewrite any sender addresses which match or are subdomains of the target address. It might look something like this:
gmx.de: all gmx.net: all aol.com: spf pt.lu: self enthusiaticbutdim.com: spf,cid,self
NOTE: The Caller-ID lookup needs to check for a TXT record with an underscore in it. Make sure your dns_check_names_pattern option is set accordingly. Mine looks like this:
dns_check_names_pattern = (?i)^(?>(?(1)\.|())[_a-z0-9\xc0-\xff](?>[-_a-z0-9\x80-\xff]*[_a-z0-9\x80-\xbf])?)+$
# Define this to handle SRS-bounces SRS_SECRET=somesecret # And this if you want to (probably temporarily) accept two keys. #SRS_OLD_SECRET= domainlist rpr_domains = *.srs.mydomain.com SRS_HASH_LENGTH=20 SRS_DSN_TIMEOUT=7 SRS_URL=http://www.infradead.org/rpr.html # Define this to enable SRS on forwarding. SRS_DOMAIN=srs.mydomain.com
In this case you want a wildcard MX record for your SRS_DOMAIN. If you want to dispense with the diagnostic stuff and use only a single domain that's not hard to do. You could even use a domain which is also used for other things, with a simple precondition on the router.
.ifdef SRS_SECRET
# Urgh. Isn't there a better way to detect that we're in sender verification?
# -- This will apparently be fixed in Exim 4.31
rpr_mark_sender_verify:
caseful_local_part
verify_only
verify_recipient = false
driver = redirect
data = ${quote_local_part:$local_part}@$domain
address_data = verifying sender
redirect_router = rpr_bounce
# Verify, and extract return address from, an SRS-address.
# Don't allow non-bounces, except from postmaster@* since some people use
# that for sender-verification callbacks.
rpr_bounce:
caseful_local_part
driver = redirect
domains = +rpr_domains
allow_fail
data = ${if !match {$local_part}{\N^[sS][rR][sS]0\+([^+]+)\+([0-9]+)\+([^+]+)\+(.*)\N} \
{:fail: Invalid SRS bounce \
.ifdef SRS_DEBUG
(malformed)\
.endif
} \
{${if and {{!eq {$1}{${length_SRS_HASH_LENGTH:${hmac{md5}{SRS_SECRET}{${lc:$2+$3+$4@$domain}}}}}} \
.ifdef SRS_OLD_SECRET
{!eq {$1}{${length_SRS_HASH_LENGTH:${hmac{md5}{SRS_OLD_SECRET}{${lc:$2+$3+$4@$domain}}}}}} \
.endif
} \
{:fail: Invalid SRS bounce \
.ifdef SRS_DEBUG
(HMAC should be ${length_SRS_HASH_LENGTH:${hmac{md5}{SRS_SECRET}{${lc:$2+$3+$4@$domain}}}} not $1)\
.endif
} \
{${if <{$2}{${eval:$tod_epoch/86400-12288-SRS_DSN_TIMEOUT}} \
{:fail: Invalid SRS bounce \
.ifdef SRS_DEBUG
(expired ${eval:$tod_epoch/86400-12288-SRS_DSN_TIMEOUT-$2} days ago)\
.endif
} \
{${if >{$2}{${eval:$tod_epoch/86400-12288}} \
{:fail: Invalid SRS bounce \
.ifdef SRS_DEBUG
(timestamp in future)\
.endif
} \
{${if and { {!eq {$sender_address}{}} \
{!eqi {$sender_address_local_part}{postmaster}}\
{!eq {$address_data}{verifying sender}}\
} \
{:fail: Invalid SRS bounce \
.ifdef SRS_DEBUG
(Not DSN: $sender_address instead)\
.endif
}\
# Wheee. At last the actual rewrite part...
{${quote_local_part:$4}@$3}\
}}}}}}}}}
headers_add = X-SRS-Return: DSN routed via $primary_hostname. See SRS_URL
# Rewrite reverse-path so that forwarding to known SPF-afflicted
# servers doesn't break. We generate a limited-lifetime hash cookie,
# from which we can later recreate the original sender address. We
# include the hostname and more precise timestamp in the domain of the
# generated address, so that we can track down the offending message
# in the log if it _does_ offend us.
.ifdef SRS_DOMAIN
rpr_outgoing_goto:
caseful_local_part
driver = redirect
# Don't rewrite unless the recipient is in a domain we _know_ to be broken
# but for local reasons have decided we need to work around.
# The text file listing broken recipient domains should look something like:
# gmx.net: all
# gmx.de: all
# aol.com: spf
# This lookup will leave the result of the lookup in $domain_data.
domains = lsearch;CONFDIR/spf-afflicted-domains
# Don't rewrite if it's a bounce, or from one of our own addresses.
senders = ! : ! *@+local_domains
# We expect any or all of 'all', 'spf', 'cid' or 'self' in $domain_data from the textfile lookup
# If the reason for the breakage is listed as 'spf', then don't rewrite
# unless the sender's domain actually advertises SPF records.
# Likewise for 'cid' and Caller-ID records.
# If it's listed as 'self' then rewrite only if we're sending it
# mail claiming to be from itself or one of its subdomains.
# If it's 'all' then just rewrite everything.
condition = ${if or { \
{eq{$domain_data}{all}} \
{and { {match{$domain_data}{spf}} \
{match {${lookup dnsdb{txt=$sender_address_domain}{$value}{}}}{v=spf1}} \
}} \
{and { {match{$domain_data}{cid}} \
{match {${lookup dnsdb{txt=_ep.$sender_address_domain}{$value}{}}}{^<ep}} \
}} \
{and { {match{$domain_data}{self}} \
{match {$sender_address_domain}{${rxquote:$domain}\$}} \
}} \
} {1}}
# We want to rewrite. We just jump to the rpr_rewrite router which is itself unconditional.
data = ${quote_local_part:$local_part}@$domain
redirect_router = rpr_rewrite
# Some addresses are joe-job protected by _always_ using SRS, and never actually
# sending mail from that address. That way, we can always reject bounces to these
# addresses, and prevent joe-jobs from being received by anyone who actually bothers
# to do sender verification callouts.
rpr_always_else:
driver = redirect
senders = !@@lsearch;CONFDIR/always-srs-senders
data = ${quote_local_part:$local_part}@$domain
redirect_router = lookuphost
# This is now unconditional. Either rpr_outgoing_goto jumped here
# because it's a mail we're forwarding to a known broken server, or
# rpr_always_else _didn't_ jump over us because it's from a sender
# listed in always-srs-senders.
rpr_rewrite:
caseful_local_part
headers_add = "X-SRS-Rewrite: SMTP reverse-path rewritten from <$sender_address> by $primary_hostname\n\tSee SRS_URL"
# Encode sender address, hash and timestamp according to http://www.anarres.org/projects/srs/
# We try to keep the generated localpart small. We add our own tracking info to the domain part.
address_data = ${eval:($tod_epoch/86400)-12288}+\
${sender_address_domain}+$sender_address_local_part\
@${sg {$primary_hostname}{^([^.]*)\..*}{\$1}}-\
${sg {$tod_log}{^.* ([0-9]+):([0-9]+):([0-9+])}{\$1\$2\$3}}.\
SRS_DOMAIN
errors_to = ${quote_local_part:SRS0+${length_SRS_HASH_LENGTH:${hmac{md5}{SRS_SECRET}{${lc:$address_data}}}}+\
${sg{$address_data}{(^.*)@[^@]*}{\$1}}}@\
${sg{$address_data}{^.*@([^@]*)}{\$1}}
driver = redirect
data = ${quote_local_part:$local_part}@$domain
# Straight to output; don't start routing again from the beginning.
redirect_router = lookuphost
no_verify
.endif // SRS_DOMAIN
.endif // SRS_SECRET
One proposed 'solution' to the problem introduced by SPF is for every forwarding mail host on the entire Internet to implement SRS. Unfortunately SRS in its initial form is fundamentally broken since it turns participating mailers into open relays.
A somewhat saner workaround is Daniel Roethlisberger's solution, implemented to work around the problems he had in forwarding to SPF-afflicted and similarly broken domains.
If advocated by SPF proponents as a solution to the problems in SPF, it would require participation by every forwarding mail host on the Internet in order to succeed in making SPF reliable. In fact all the proposed 'solutions' require participation from the whole Internet, so let's just suspend our disbelief for a while and continue...
There's also my original solution based on Daniel's first incarnation, before I modified it to have shorter localparts and conform with (part of) Shevek's suggestion below.
There is a newer alternative to the original SRS, documented by Shevek's paper. It's less flawed than the original SRS, but still the 'shortcut' scheme requires an essentially open relay for certain, albeit limited, local-parts, leaving the implementer open for abuse.
Hint: In §2.7, he asks the question: "What do we actually need in order to forward a mail back to B yet make sure A never receives a spam?". But what about all the crap we send to SRS0+*@B, where B is any unsuspecting host on the net?)
In particular, consider what happens if you're implementing that scheme at your domain victim.com, and someone malicious spends a week sending spam and virii with SMTP reverse-paths of the form SRS1+navy.mil+HHH+TT+dummy.com+user@victim.com. You helpfully attempt to relay all the bounces to the MX servers for 'navy.mil' and the other domains which the attacker put in there to make your life interesting.... right up until the moment they arrive on your doorstep and pull the plug.
If you really must perform rewriting, it helps to observe that you don't actually need to do any rewriting if the original sender's domain doesn't publish SPF records. And that there's no need for a dedicated RPR domain such as srs.infradead.org to publish SPF records, since nobody else is likely to be forging addresses at that domain which even pass sender verification callouts. So there's no real need for any mail to get its sender rewritten more than once.
In fact, some people do limit the addresses from which they'll accept mail claiming to be from certain domains, using their own ad-hoc mappings of domains to acceptable IP addresses. But if the officially recommended SRS method avoided multiple rewriting, then nobody operating their own ad-hoc scheme would be justified in adding one of these domains which are explicitly for SRS to their database.
In fact the real problem here is SPF. It's attempting to make a change to the mail system which is fundamentally incompatible with the way it's always worked. SPF can only work reliably if everyone implements some form of sender rewriting when forwarding mail. In a world where some people haven't even discovered ESMTP yet, that's really not a very realistic prospect.
Using SRS to protect your own outgoing addresses, and callouts to verify the sender address in received mail, achieves much the same effect without any of the breakage.
More useful would be a scheme where instead of listing permitted IP addresses for the MAIL FROM:, a domain owner instead lists public keys which are required in all mail containing addresses at that domain in its From:, Sender:, Resent-From: and/or Resent-Sender: headers.
Such a scheme could be entirely transparent, without requiring complicated and abusable action by forwarders.
To take a primitive example, I could simply state that all mail from dwmw2@infradead.org should be GPG-signed, and if there were a standardised way for me to publish that information (and the key), then all forging of mail from my address would be preventable.
In fact, you can do better than GPG, which requires mucking around with MIME and will confuse people who won't understand the difference between 'real' keys which really do guarantee it was you who sent the mail, and the mostly insecure keys which are used for automatic signing of all outgoing mail. Some form of hash in the headers, which can deal with the fact that the mail may have a few lines added at the beginning or the end, is perfectly feasible.
The most interesting part of this idea -- the only part which isn't mostly trivial -- is how we sign a hash of the body without it getting broken by intermediate hosts adding crap to the top and bottom. That's discussed briefly in a mail here,
Any further thoughts on such an idea would be welcome.