Amazon SES Postfix relay

Lazyweb:

I am attempting to use Amazon SES as my first-hop relayhost out of Postfix, since many sites are preemptively blocking connection from my EC2-hosted mail server. I have done the DNS dance, verified my domains and DKIM, and when my instance sends mail with any address ending in @dnalounge.com in the From: line, it is accepted by the SES relayhost and delivered. That's fine.

But none of my employees can receive mail, because this happens:

"From: foo@bar.com; To: employee@dnalounge.com" arrives at my mail server. The aliases file leads me to forward that message along to employee's real address, example@gmail.com. So then SES says:

"554 Message rejected: Email address is not verified. The following identities failed the check in region US-WEST-2: foo@bar.com"

How I fix.

Tags: , , , ,

30 Responses:

  1. Ben Rosengart says:

    You won’t like this, but: get out of the mail forwarding business if you possibly can. It makes you look like a spammer to others, and your real outbound traffic suffers. It’s not worth it.

  2. bizzynderscore says:

    Its been a while since ive used postfix, but most likely a map for employees, not relaying through SES?

    • jwz says:

      And when one of those employees has the bad taste to be using Yahoo for their email, who won't accept mail from my server?

  3. Turnbull Stephen J. says:

    Sounds like what would happen if bar.com publishes a DMARC p=reject policy. (I assume you’ve already ruled that out, but I’ve had to learn more about DMARC than a self-respecting human should and I can’t get past it without an explicit denial.)

  4. Doctor Memory says:

    Verify those addresses directly via SES. Go to the SES console -> Identity Management -> Email addresses, enter the employee's email address. Tell them to click on the confirm link that will arrive in their inbox. Now pour yourself a drink and do it for all of your other employees. You can at least script this using the AWS CLI tool.

    Welcome to life in the Clown. :(

    • jwz says:

      No, this level of fuckery with onboarding is not tractable.

      If this is really how it works, then I am soliciting suggestions of other email provider relayhosts that actually function.

    • jwz says:

      I'm not convinced that what you suggested would work anyway, since it's complaining about the 3rd party customer in the From line, not about either of the addresses of my employee.

      • Doctor Memory says:

        Ugh, actually you're right. Sorry, it was late and I hadn't had to deal with this end of SES for a while.

        SES really just isn't meant to be a general-purpose SMTP relay: as ChoHag said below it's really just meant to be a spam outflow so large and nominally accountable that the world won't auto-block it the way they do the EC2 networks.

        If you wanted to attack the problem with anger, guile and a certain amount of applied stupidity, you could maybe rewrite the messages en route to have an envelope sender and From address @dnalounge.com but have a reply-to header pointing at the original sender. (Is procmail even still a thing? You could probably do that with procmail.) No guarantees that amazon wouldn't notice that kind of trickery, but at the volumes you're talking about it's probably less than line noise to them so you might get away with it.

        I'm trying to think of some reasonably non-obnoxious solution to this that won't end up tacking on another $25-50/mo to your operational cost here and so far coming up empty. :(

  5. monks says:

    As Dr Memory says, you need to verify addresses before you can send from them.

    https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-domains.html

    Amazon SES requires that you verify your email address or domain, to confirm that you own it and to prevent others from using it.

    Also, you may have already done this, but you will need to have your account “removed from the SES sandbox” before you can send to unverified domains.

    https://docs.aws.amazon.com/ses/latest/DeveloperGuide/request-production-access.html

    • RJ says:

      the truly obnoxious thing is, you have to ask for the production access in each region, so if you initially did the request in us-east-2, you now get to ask for it again in us-west-1!

  6. RJ says:

    are you supplanting any of the From, Return-Path, Reply-To headers with a dnalounge.com address? without rewriting those, it looks like the sender is the vendor, not dnalounge.com :/

    • jwz says:

      No I'm not rewriting From because I'm not the originator and so that would be insane.

      And it doesn't "look like" it's from someone else because my smtp connection from my server to Amazon is authenticated with the uid and pass that Amazon generated for my account - from the IP Amazon assigned me on the virtual machine that they run.

      So no. Don't try and tell me Amazon is somehow "confused" about this mail or "can't know" that the person asking that it be delivered is me, their fucking customer.

  7. ChoHag says:

    You may be in for a hard time because you're doing it wrong. Specifically you're assuming that SES is an smtp relay but in fact it's a spam relay that's simply so large the world is forced to accept it (this is a bit like the way Russia's totally not at war even though it obvious to anyone smarter than a brick that they're at war because if we called it a spam relay war then their money our money might go somewhere else).

    Specifically, https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses.html seems to imply that all four headers must be verified: 'You must verify each identity that you use as a "From," "Source," "Sender," or "Return-Path" address' (this being English, or seems to mean and in that sentence) but it does hint that you can stop them considering an address you've already verified by abusing the spec horribly to add '+foo' 'labels' to them.

    If you can't get it to ignore the from field I think the only options are to use a real mail relay or go nuts with address rewriting. This is as dumb as it sounds. I'd like to hear that I'm wrong and SES can at least be used as an smtp relay but it literally has a thing called a "reputation dashboard".

  8. David K. says:

    I ran a self-hosted email server for a decade, and finally gave up. I don't have anything to offer here, except a tangential observation. Even if you run your own server, you have to relay through one of the "biggies" to get your mail accepted at any of the others, regardless of all the mechanisms that were supposed to fix this. Large corporations have, once again, seized something by fiat that should have been free and open, under the guise of "security."

  9. jwz says:

    I can't even figure out how to make Postfix deliver "From: *@dnalounge.com" through SES while (trying and often failing) to deliver everything else directly because the interaction between sender_dependent_default_transport_maps, sender_dependent_relayhost_maps, transport_maps and default_transport is completely incomprehensible.

    • Zygo says:

      I ran into this trying to get my own mail forwarded to my own Google account. The problem I kept running into was that I couldn't receive mail at Google because Google made a private agreement with example.com such that Google wouldn't accept any mail with a "From" header containing "example.com" that didn't come from example.com infrastructure. This is why so many phishers can't spell bank names correctly, and it's why you can't forward mail from example.com to anyone important enough that example.com might want to make private agreements with them.

      So...that was the end of that. I gave up, and just had Postfix feed that mail to a Perl script LMTP server (alias the recipient to a virtual domain, then map the transport for that virtual domain to LMTP), which mangled the headers and fed the mail back to the outgoing queue. I can't figure out how to make Postfix do two separate things to the same email even though I have written entire MTAs from scratch.

      The script mangles the addresses by appending ".$hash.fwd.mydomain.org". Any replies to that domain would be captured, authenticated, demangled, and forwarded back. The $hash was used as an authenticator, combining a secret with the original domain. Most spam filter UIs demand stable correspondent addresses. Google doesn't like seeing stuff that looks like a phisher misspelling "example.com", but at least that mail ends up in the spam folder instead of being rejected outright at MAIL FROM:.

    • roeme says:

      Glad to see I'm not the only one. IMHO postfix' configuration settings are an arcane mess, though so far I managed given enough time wading through their docs. At least they understand the idea behind Hypertext.

  10. Darren says:

    Don’t try to think of SES as a mail server; think of it as a “do not reply to this email” mailing list origin. Every sender must be whitelisted, plus you really have to be on top of your To: list management; it doesn’t take many bounces for them to put you on prohibition.

    The Internet has changed a lot in the past 20 years - it’s almost impossible to run your own mail server these days unless you’ve already been running it impeccably for 20 years on the same IP and have that history.

    AWS doesn’t really have a product that is both a well-trusted email source and allows a remote origin. You’d be better off either (a) using an ISP just for your email (I use bluehost, YMMV) or (b) biting the bullet and moving your email to a hosted email provider like gmail (which may require $$$ depending on how many users you need.)

  11. William says:

    You probably don't actually want to use SES for this.

    Since you're already running postfix you can just keep doing that. If you don't already have an elastic IP, get one, then cut AWS a ticket to get the reverse DNS set to something that's not AWS: https://portal.aws.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request

    You'll need to set up an A record pointing to the box before they'll create it, but presumably you have an easy way to do that. I've been doing this for some number of years without having to deal with SES.

    • William says:

      Oh, I'm shitty at reading. Maybe it's just because I've had the same elastic IP for some number of years or I got in early enough to where it didn't get blacklisted for abuse, but you can always do the annoying thing where you change the IP again if it's not deliverable.

  12. Teddy says:

    Are you applying SRS when forwarding your mail? If not, that could be the problem; I would suggest using PostSRSd. Yes, SRS is stupid and bad, but that is the world we live in.

  13. jwz says:

    Well, I tried to make this happen:

    • Emails sent to the Announce list, and transactional emails from the store, would go through SES;
    • Other emails go direct.

    I failed.

    I can't figure out any combination of /etc/postfix/transport and /etc/postfix/header_checks that will allow me to do that.

    I thought I could key off of ^Return-Path: .*-(request|bounce)@dnalounge\.com but that didn't work, and neither did a few other things I tried.

    So I guess I'm back to not using SES, sending email directly always, and playing whack-a-mole when someone decides that my IP is ritually unclean.

    • RedGrittyBrick says:

      "none of my employees can receive mail".

      Known jwz skills:

      - email client.
      - app that runs on phones.
      - doing convoluted things to work round corporate craziness

      I should probably apologise in advance: sorry.

  14. Wild-haired Norseman says:

    While I don't live under a bridge, feel free to assume I'm a troll. On the off-chance you read this and it doesn't just get automagically directed into a bit-bucket of rotten slops...

    If you want a sane email provider, I can recommend https://runbox.com Their service is not free-as-in-beer, but I have had no issues with their service for many years now.

    And thank-you for XScreenSaver.

    • Malcolm says:

      I'm a runbox customer too, and I use them as a smarthost with my own domain. Things that might put you off: not cheap, and they are in Norway.

Leave a Reply

Your email address will not be published. But if you provide a fake email address, I will likely assume that you are a troll, and not publish your comment.

You may use these HTML tags and attributes: <a href="" title=""> <b> <blockquote cite=""> <code> <em> <i> <s> <strike> <strong> <img src="" width="" height="" style=""> <iframe src="" class=""> <video src="" class="" controls="" loop="" muted="" autoplay="" playsinline=""> <div class=""> <blink> <tt> <u>, or *italics*.