To recap, my domain hosts its own SMTP server running Postfix, and /etc/postfix/virtual contains a bunch of entries forwarding "email@example.com" to whatever their actual email address is, usually gmail.
This has been mostly working fine for a decade or so, but lately there have been more bounces due to "strict SPF". For example, jksound.com's SPF record includes "-all" (dash instead of tilde) which means that when firstname.lastname@example.org tries to mail email@example.com, we forward that along to firstname.lastname@example.org, and then Google rejects it with 550 "SPF hard fail".
So, I don't know whether it has recently become more common for people to use dash-all instead of tilde-all, or whether Google recently started actually enforcing dash-all in a way that they didn't before, and while I am curious about that answer, it doesn't really matter.
Another thing that doesn't matter is that SPF is bullshit that solves no problems and should not exist. Let's just take that as a given and move on.
What does matter is, what the fuck do I do about it?
Telling all of these people, "Hey dummy, use tilde-all instead of dash-all" is obviously not practical.
"Provide an IMAP server for all of my employees" is a terrible answer, in terms of both maintenance headache and disk space.
"Turn over your MX record to some third party service" is an even more terrible answer, because so many of our custom internal systems touch email. Order confirmations. Shipment notifications. Calendar mailings. Sales reports. Bounce and unsubscribe handlers. Address verification and password resets.
Is there a third option?
Finally, here's a concrete question: let's say I desired to have a filter plugged into my Postfix that looked at a message, identified it as one that Google is definitely going to reject because of strict SPF and then... did something else with it. Like, say, forward it as an attachment instead. (This would obviously be insane and terrible, and yet still better than bouncing.) Is that a doable thing, or should I just stick forks in my eyes right now?
My current approach is to provide a POP3 server for all of my employees. It turns out that POP3 is a thing that still exists in the Twenty-First Goddamned Century. Gmail provides an option to download mail from external POP3 servers, if you trust them with your password. As far as I can tell so far, Google doesn't penalize my server for spam that is relayed that way, they just process it normally.
But, every now and then, instead of downloading a message, they deliver a message to the recipient that says "The message [...] contained a virus or a suspicious attachment. It was therefore not fetched from your account and has been left on the server." And in that case they leave it on my server forever, which is annoying.
This tools suggests you're using a deprecated SPF DNS record type.
Well that "super" tool doesn't work for me in multiple browsers, but I think the quoted complaint is saying "you are using a record of type SPF, you should use TXT instead" but in fact I have both records and they are identical so that doesn't matter.
The magic phrase to search for is "SRS" — Sender Rewriting Scheme. Your email server needs to mangle the envelope sender such that the emails it forwards appear to be from an address @dnalounge.com (or some other domain that has SPF settings you control and for which Google considers your email server to be a valid source of mail). There are a number of schemes for doing this reversibly so that when you receive a bounce you can forward it on to the original envelope sender.
No, see, that's insane. It means that when email@example.com sends mail to firstname.lastname@example.org the email they eventually receive has some shit like email@example.com in the From field. What decade is this?
It shouldn't be visible to the user unless they go looking for it: it's the envelope sender that's being rewritten, not any of the headers.
(I know many MTAs stuff the envelope sender into a Sender header immediately prior to delivery, but most MUAs [including Google] don't display this anywhere other than the "view full headers" option.)
The From field is kept intact, only the envelope sender, i.e. the SMTP FROM, gets mangled.
Says the person who suggested sending the original email as an attachment :-)
Two different non-solutions can be insane.
Unfortunately, this is actually the case. When you forward mail from a domain with a restrictive SPF policy, Gmail can and will refuse or roundfile that message when it arrives from your server with an envelope sender in the restrictive domain.
You can use SRS (which is absolute bollocks, a bullshit crazy "solution" to a terrible problem), or you can just forward the mail using the employee's dnalounge.com address as the envelope sender - that way, the SPF policy Google checks will be yours, and that policy will allow sending with those envelope sender addresses from your server.
Another potential wrinkle - I don't use Postfix, but I presume it does opportunistic TLS on outgoing SMTP connections, i.e. using EHLO to probe for STARTTLS support, and then issuing the STARTTLS command when the server does advertise it?
If Postfix doesn't do that by default, or is configured not to do that, much of your mail to Gmail-hosted domains will simply disappear - accepted for delivery at SMTP time, but the message never shows up in the recipient's inbox, spam, junk, or whatever else Gmail uses. The messages are just gone.
This was a relatively recent change - the last few months. I hadn't bothered with STARTTLS on outgoing connections because any email I send with confidential info is encrypted, so STARTTLS is unnecessary. But all email from my server to Gmail-hosted domains just started heading for the bitbucket.
After looking into current Postfix a little more (I'm way out of date on it...), I'm not sure it actually includes an easy way to forward while changing the envelope sender address to the virtual domain address that is being forwarded outside your machine.
The virtual table docs explicitly say it only changes recipient addresses, so that's out.
The canonical table docs don't seem to provide a way to change the envelope sender when forwarding, unless I'm reading them wrong.
If Postfix can forward by using a helper program - maybe it includes one, other MTAs certainly do - that lets you do this envelope change, then that is how I would do it. With some divisive software I'll just call AnotherMTA, this would be done by having a single config file - probably maps to an entry in a Postfix table file - for each address you're forwarding. The program would be called like:
For the config for firstname.lastname@example.org
... and the mforward command would be a one-liner shellscript that rewrites the envelope sender with an env var helpfully set by the MTA:
-f sets the envelope sender (aka envelope return-path) address only, and doesn't touch the message content. This sets the envelope sender address on the outgoing email to the envelope recipient address of the message that arrived destined for email@example.com, and sets the destination of the message to Joe's Gmail address.
This has *got* to be possible with Postfix - Wietse reimplemented most other features from AnotherMTA (although... badly in most cases). But hell if I can find it in the Postfix docs.
This still doesn't work because now you're mangling the source FROM address making replies difficult.
You can rewrite the sender to be 'firstname.lastname@example.org' but now you've got to deal with replies and there's only one reply-to field.
So let's say you move the sender to reply-to, but they've already got a a reply-to set -- I suppose you could preserve it but now you need some logic in that forwarding code.
In any event, the solution is wrong because of SPF. If people decided to stop using -all, this would be easier to solve, but it isn't.
No, you've misread what I said.
I only said to change the envelope sender - aka the envelope return-path - not touching any of the message content (headers or body).
The envelope return-path is not used by mail clients when constructing replies. They universally use an address extracted from one of the header fields - Resent-from:, Reply-to:, or From:, usually.
I used PostSRSd in the past and that helped me with that problem.
My use case was forwarding lots of emails for some of my domains to Gmail and other email providers.
Installing it is not too bad, and I even found a package in Fedora, but keeping it up to date and maintenance was annoying.
At some point I decided life is too short, and I did switch to a third party provider to do the forwarding.
I also use PostSRSd since June 2017. It solves exactly Jamie's problem.
Huh. To make sure I understand how this works, can you show some examples of what it does to your headers?
Also does it handle bounces right? The readme says that one can do so but it isn't clear that this package does.
Meta to my domain's MX (Mythic Beasts) with a .forward to Google, where I read mail, with SRS:
With SRS, the "envelope" from (i.e. SMTP MAIL FROM) changes, the header From: does not. Although most MTAs embed the SMTP MAIL FROM address somewhere as they add Received headers and whatnot, it's not where clients look for a From address.
As another stubborn self-hosted SMTP holdout, I ran into this a few months ago as well. At some point gmail just decided it hated me and would randomly deny inbound email from my servers, no matter how perfectly I crafted my SPF and DKIM records. It wasn't a full block, sometimes they'd get through, but many times they wouldn't.
My workaround in the end was to set up an outbound SMTP relay instead of attempting SMTP delivery directly, and route all my outbound email through a commercial mail sending service - Amazon SES, in my case. The DKIM and SPF records for my domain were then updated to include the relay's official outbound SMTP servers, and now Google will accept mail from my domain reliably.
Yup, it's terrible, but at least this way I can still retain full MX control for inbound mail, and only my outgoing mail is passed to a third party for final delivery.
I also use Amazon SES for some things (basically, bulk mailings only) but I fail to see how routing through SES would solve the problem I've described.
Yeah after reading some of the other replies I realized your problem is probably a bit different. In my case it was general deliverability problems for email originating from my own domain, but your case is complicated by the fact that you're doing cross-domain forwarding, and you're right, I don't think the extra hop through an outgoing relay would change anything there.
I mean like... maybe if the MX for dnalounge.com was set to SES, and they knew to forward to my server, that would work? But I don't know if that's even an option or what else that would break. It sounds like it could loop.
Wait, no, that wouldn't help at all, it just moves the problem.
Did you fix your problem, then? I have someone with their own domain name who has tried multiple times to send me email at my [professional me]@gmail.com address, and it doesn't even make it as far as my spam filters. Just gets eaten by the Invisible Pink Unicorn of the internet.
Yeah, that sounds like the same symptoms I was seeing. I could get through to some people, but would be completely dropped on the floor while emailing others. Relaying through SES completely resolved my deliverability issues, but jwz's setup involves forwards, which introduces some additional complexity.
https://docs.aws.amazon.com/ses/latest/dg/postfix.html is the procedure I followed, if it's useful to anybody else out there.
I’ve not used Postfix in a long time, but I’d want to give each address a temporary mbox which automatically forwards email on to the desired address. The emails would come from your @dnalounge addresses delivered to the gmail addresses of staff.
Storage requirements shouldn’t be crazy since you should be able to bin the mail once you’ve forwarded it.
Everything is shit.
Whatever people write here who simply used the first Google hit, do NOT try to get SRS working. It will simply turn SPF fails to DMARC fails, and to beat THAT you will need to use ARC (RFC 8617), and so on. Kafka couldn't have come up with this.
Mail forwarding is broken. You can try to use procmail&formail to simply re-mail with a new FROM, but the other problem is that if your people receive spam mails (which I guess they do?), your mail server IP will eventually be blacklisted as a spam source.
So really, the only "solution" is to set up POP3 for your people, and they have to configure gmail to poll from that (please don't kill me, I'm just the messenger).
SRS does, in practice, work for me™.
This starts with misinformation. DMARC is a policy system. It has many problems, but this ain't one. DMARC has two signalling inputs which you can control: SPF and DKIM. Either one passing will be fine. And the sort of mail-forwarding being done here won't break DKIM, that's only broken when you mangle the body or the signed headers, as most people configure mailing-lists to do.
POP3 is more reliable for getting messages into Gmail, yes.
For the sort of mail-forwarding being done here, SRS is pretty much necessary now.
For more on the Best Current Practices for running an independent mail-server sending email with "improved" chances of getting through to services such as Gmail, a two-year-old blog post of mine might be helpful: https://bridge.grumpy-troll.org/2020/07/small-mailserver-bcp/
This is about where I reached, and then I have to educate people about how to have Gmail check their POP3 mailboxes, and then I find out half of them ignored the instructions because it was too much so they skipped it...
Apologies to our host, but email is gefurchtbar. Relevant bits up top here: https://emailisbad.com/
It took me way too long to realize on skim that you meant Kafka the person, not the platform...
During which time I was like "Don't add Kafka to anything, then you'll have 2 problems..."
Get outside more.
This might turn out horribly formatted and / or clueless.
I run a wiki, and this is what I had to do to allow users to email each other; the wiki email host is purelymail.com . Enabling wgUserEmailUseReplyTo meant that email would go out as:
> From: (the smtp account, email@example.com)
> Reply-To: (the email address of the person writing email, firstname.lastname@example.org)
A test to myself landed in my Gmail inbox.
Personally I've noticed that providers in general tend to go for dash-all a bit more often (anecdotal, of course). What worries me more is that I've even seen providers hard-fail mails sent from domains without any kind of SPF. So we're on the brink of actually needing to setup SPF everywhere. That's not even supposed to work this way...I really hate it...
You really don't want to use SRS when forwarding stuff to Google. Any spam your employee receives will then look, to Google, as if it came from you, and hurt your reputation in their anti spam system. If you just plain forward the mail, they'll blame the original sender, as they should.
SPF alone (i.e. without DKIM and DMARC) is a problem, this is true, and particularly with the "-all". Forwarding just doesn't work in that situation. If it's important to such mail sent on, special handling where SRS is used for mail from those particular domains might be an option.
See https://support.google.com/mail/answer/175365?hl=en for Google's own suggestions.
Ideally, the problem sender (e.g. jksound.com) should be encouraged to add DKIM signing and a DMARC configuration. That's how we do things today. It allows plain forwarding to work, because the DKIM signature will be intact, so SPF can fail hard without DMARC failing (it just requires either SPF or DKIM to be OK), and the mail is delivered.
Frankly, if you want to send email today, with a reasonable chance of it being delivered, you should have SPF, DKIM, and DMARC, all working properly. Companies like JK Sound ought to be grateful for being told this.
So "Google's own suggestions" basically says "use DKIM and don't use SRS". Which I'm doing, and yet here we are.
Whether companies like JK "should" be grateful that things are busted is irrelevant here, because I'm not trying to solve a problem for them, I'm trying to solve a problem for me. JK doesn't give a shit that their emails to me bounce. Whereas, I care very much.
Speaking from experience as I had pretty much the exact same problem as yours:
The behaviour depends on the original sender. If the message carries a valid DKIM signature and the sender has DMARC set up, GMail will typically accept the message.
Problem is: you can't rely on all your senders having valid DKIM - many of them don't, and there's nothing you can do about that. You can't add a DKIM signature while redirecting the email because you don't have control over the domain mentioned in the From header, which is what the DKIM signature is checked against. You could rewrite the From header to use your own domain, but now the recipient will be utterly confused by the From addresses of the emails they're receiving.
Someone else mentioned Authenticated Received Chain (ARC). ARC is not a practical solution for these kinds of cases because ARC does not provide a way to establish trust in the first place. It assumes the chain of trust already exists. So you'd have to convince someone at Google to adjust their ARC config specifically to trust you. Good luck with that.
If there's no DKIM, GMail will still typically accept the message if there is valid SPF and DMARC. It doesn't matter what the DMARC policy is set to - "none" works, but the DMARC record has to be there.
Now, passing SPF is doable, you just need to make sure your envelope from (SMTP MAIL FROM) is from your domain. Note this is not completely trivial because you then need to handle the return path, i.e. asynchronous bounces, properly. It's a solvable problem though (see VERF and SRS).
But here's the thing though: in my experience, just passing SPF is not enough for GMail to look kindly at your message. You need SPF and DMARC.
Problem is: an email redirection service cannot pass DMARC with SPF alone. This is because DMARC requires what's called "identifier alignment" (RFC 7489 3.1). Identifier alignment means that, in order for the DMARC+SPF combination to work, the envelope from has to match the From header. Which in this case it won't. And so the SPF DMARC check won't pass. And GMail won't let you in.
If you wanted to work around this problem, you could change your envelope from to match the From header, but obviously now you can't pass SPF anymore. The other approach would be to rewrite the From header to a domain that you control, but again this will likely result in a pretty bad user experience for the recipient.
Once I understood all this I came to the difficult conclusion that there is no way to make it work. "Perfect redirection" with GMail recipients simply can't be done. I ended up moving my domain to Google Domains to use their email redirection feature, which works because in that setup it's just Google redirecting email to itself, and in that case it seems to only filter at the initial entry point.
Oh, one last thing:
It's not just a terrible answer, it's also a wrong answer. That third party service will have the exact same problem you currently have, and I doubt any of them has a special deal with Google to let them in. There's no reason to believe any third party service could do anything to solve the fundamental problem. In fact, I had this problem with a reputable third party service (Gandi).
The only redirection service that doesn't have this problem is Google Domains because that only requires Google to trust itself.
And there we have it: if you look at it from the perspective that the “big three” (Google, MS and whoever company you think is the next biggest “email” provider) want to capture the whole market and have the power to enforce paradoxical rules and hide their absurdity behind incomprehensible “knowledge base” articles, everything seems clear, doesn’t it?
They whittle us down, self-hosted domain by self-hosted domain, until there is no one left who remembers what federation even means.
They have won this game years ago. Now its only a question of how much this-will-solve-spam-for-real-now! regulatory capture bullshit we can make them invent for our amusement before the end.
This part of Google's suggestion seems interesting:
"You can use third-party software to identify spam message and prevent them from being forwarded. If you never forward spam to Gmail, you can change the envelope sender to your domain."
... so I guess what they're suggesting is setting up SpamAssasin or whatever and reject messages from forwarding that way? This seems like it could go wrong in a lot of ways and somehow seems like a least terrible option...
Yeah, but "if I were a mail server I would simply not forward the spam" is not a very realistic solution.
This is a problem with mail forwarding in general and not with self-hosted servers like some suggest.
I have a domain hosted at Microsoft and it's impossible to have mail forwarding to gmail (or anywhere else really) work without massive amounts of rejected or straight-to-spam emails.
Since I have no control over MS's server's config to try out SRS or other ridiculous schemes, I ended up settling on pulling the emails out of MS using pop3.
Thank god I'm not in the business of setting up email distribution lists anymore, it must be hell nowadays.
Forwarding mail like this has not really been viable for a long time due to SPF. Google has indeed recently become more strict with regard to authenticated email and will now reject emails that have don't have either SPF or DKIM (not just fail, actually if missing, now).
As others have said, SRS will make it work again in gmail's case, but even so, the future is looking tenuous for mail forwarding.
I recently threw in the towel on this and told people to set up email fetch in their gmail to download mail coming to my server. Forwarding, especially to gmail, is just so dang brittle. I’m hoping someone here gives you a better idea :)
In retrospect, "Rough consensus and running code" was not such a hot design paradigm for stuff that was going to turn out to be critical infrastructure.
Part the millionth.
I know this doesn't solve your immediate problem, but:
This is why I've always kept domains like @example.com strictly reserved for actual people (modulo stuff like email@example.com, e.g. for TLS certs), and set up a dedicated domain for everything infrastructure (like @example.net or @example.link). You can further subdivide this for customer-facing stuff, like @exampleapp.com, if a .net domain feels less presentable. It feels like having actual routable IPv4 addresses, instead of dealing with NATs, SNIs, SRVs, etc - you can dodge so many problems.
Also helps for other internal stuff, you can look at the address wiki.example.net and immediately tell it's not supposed to be public.
Install Mailman. Create one mailing list per employee. Set: Privacy Options... -> Sender filters -> dmarc_moderation_action = Munge From. I'd add, cry about the state of the world that this is necessary but we're all already there.
We used a similar approach when dealing with forwarding in the other direction from a Google Workspace tenant - since Google doesn't do SRS (as far as I can tell) we created groups (mailing lists) for each forwarding address and added the target address to the group. This is handy since Google don't charge for groups.
Speaking as the author of one of the first SPF implementations let me just say that SPF is bullshit, solves no problems, and should not exist.
So what I'm hearing so far is:
This summary is correct. POP3 is the only solution that has a chance of working.
I would add two notes:
1. Even without SRS, if you forward spam to Google they will eventually put you on their shitlist. When that happens, you have for most practical purposes lost the ability to send email from your domain. You're lucky this hasn't already happened.
2. When you configure Gmail to fetch email from a POP3 mailbox, you cannot effectively control the polling rate. It just decides to fetch email when it feels like it. In practice, this means that emails can be delayed for more than an hour.
If the 2nd note is not acceptable to your users, you're shit out of luck.
In the light of all this - and I have bookmarked the thread, being a part-time postmaster - setting up an imap server is a child's play. The MTA is really the heavyweight component here.
Setting up an IMAP server is not the problem. As I said, the problem is the commitment of hours per week of maintenance for the rest of my life, and the infinitely-ballooning storage requirements.
I serve mail for a uni institute (engineering, 25 - 40 staff) with Dovecot - 19 years worth of mail amount to 500 GB. Ongoing Dovecot maintenance is less than a day per year, assuming that you are running a machine for the MTA anyway.
Beats fighting Google any day. ;)
The default is still that mails are deleted after POP3 retrieval. Tell people to leave that checkbox alone in the import dialog.
I agree of course that everything else is still a major pain.
I just went through this insanity a few months ago. Gmail is picky but not nearly as bad as the cellphone email-to-SMS gateways. The solution was a crazy collection of various amulets to ward off spam filters that were attached to the originating server, the edge router, and the emails themselves.
I’ll lookup what I did when I get back to my home office later today but recall it included:
- ensuring that the originating IP address is indeed associated with the sender domain
- SPF of course
- DKIM identifiers on outgoing email. Gmail didn’t need this but the cellphone gateways did
- a reverse DNS PTR record served at the edge. Again gmail did not need this.
As pointed out above, SRS has become less-reliable in the case of forwarding to GMail - ARC addresses some of these issues.
I use forwardemail.net to achieve this because it has a collection of heuristics which look at various scenarios and tries to figure out the best thing to do. It tries to preserve the original sender semantics, but will fall back to various rewrites if that doesn't work.
There's a fairly straightforward self-hosting process (it's all on GitHub) if you don't want to use a third-party MX. Also take a look at the various issues and pull request discussions on GitHub, especially those mentioning ARC, for some details on how it handles various cases.
I, for one, vote for the forks. It is the least painful option.
As noted above, if you run Postfix than postsrsd is the solution -- and I am using it in exactly the way you are wanting to use it. It rewrites the envelope sender so that SPF passes, and DKIM should continue to verify because the From header is not mangled.
However, the upstream is a bit of a mess because it integrates as a canonicalization filter and runs against all Postfix services. I've fixed this so that the forward path can be run as a milter here:
If you're running a Fedora/EL-like OS, you can get packages here:
From the docs:
For sender rewriting, include the following optional argument in any desired smtpd stanza in master.cf that processes forwarded messages:
For recipient rewriting (for bounces), include this fragment in main.cf:
Except for the part where everybody (including Google) says that you absolutely cannot use SRS with Google or they will blame you for every spam you receive.
Google's spam blame is ineffable, and I've found it to be independent of SRS -- at best, the envelope sender domain is one parameter of many fed into a spiteful ML model. Also, their advice is literally useless overall as they don't offer a functional solution to the forwarding problem and their postmaster tools docs are often out-of-date or simply wrong.
SRS 100% works here. Yes, you ought to be running a spam filter on your incoming MX.
ARC is opaque garbage that doesn't work unless there is a trust relationship between Google and you; it only exists for the oligopoly mail players to prevent people like Mailchimp from being permanently blacklisted because they generate so much spam but are somehow "legitimate".
Oof, I'm almost afraid to answer, given the feedback I'm sure I'll get. But....You need to rewrite headers. I'm not familiar with postsrsd, but it sounds like I'm doing the same thing as what it does, but I do it via a shell script. Also, if you really want to get hax0ry and bypass Gmail IP/domain reputation issues, you can in theory actually use Gmail to SEND the mail. It's very easy to script. It will overwrite your from address, though, so you'll need to decide what to do about that. (I move the from address to the reply-to.) But, it works great. I use it for forwarded email and system status notifications. See https://www.spamresource.com/2022/04/lets-send-emailwith-curl.html to forward using curl, and https://www.spamresource.com/2020/09/lets-send-mailthrough-gmail.html if you want to forward via Postfix (either all mail or just mail with a certain x-header. Both work well for me.