Postfix address rewriting

Dear Lazyweb,

How do I get postfix to log the entirety of inbound and outbound SMTP sessions? (Including the STARTTLS payload.)

I've mostly solved my mail-delivery problems by routing most (but not all) of my outbound mail through Amazon SES. But when I send a message that happens to bounce, the bounce is bouncing. I'm having a hard time understanding the thing that "notify_classes = 2bounce" is sending me.

<jwz@jwz.org> (expanded from <root@dnalounge.com>): host email-smtp.us-west-2.amazonaws.com[52.88.130.249] said: 501 Invalid MAIL FROM address provided (in reply to MAIL FROM command)
...
Reporting-MTA: dns; cerebrum.dnalounge.com
X-Postfix-Queue-ID: 4FD309B318
X-Postfix-Sender: rfc822; MAILER-DAEMON@cerebrum.dnalounge.com

Final-Recipient: rfc822; jwz@jwz.org
Original-Recipient: rfc822;root@dnalounge.com
Action: failed
Status: 5.0.0
Remote-MTA: dns; email-smtp.us-west-2.amazonaws.com
Diagnostic-Code: smtp; 501 Invalid MAIL FROM address provided

It looks like it's saying that SES rejected my "MAIL FROM" address, but I definitely have dnalounge.com and jwz.org on the list of verified, allowed domains. So I don't know how to tell what address it is complaining about. I've tried debug_peer_list but it is confusing and does not appear to answer my question.

Previously.

Tags: , , , , ,

14 Responses:

  1. Tony Finch says:

    Dunno about fixing the logging (and that would vex me too) but is SES objecting to MAILER-DAEMON@cerebrum.dnalounge.com because cerebrum is not on its list of allowed domains?

    • tb says:

      Either that, or it doesn't like that cerebrum.dnalounge.com is a CNAME ...

      I would check if postfix has myhostname in main.cf set up correctly (it should probably be mail.dnalounge.com - something that has an A record in DNS).

      • tb says:

        I hate to reply to myself, but after some googling it looks like Amazon SES will not let you send bounces - it will reject mail with empty envelope senders.

        • jwz says:

          I'm really confused about what's going on. My machine sends an email to bouncy@example.com, relaying it through SES. SES tries to deliver it to example.com, gets a rejection at the SMTP layer, and then should generate a bounce message that it delivers to my Return-Path.

          But instead I'm seeing a bounce message getting created by my server, and then... like... trying to send it to myself via SES? Which is why I wish I could just see the full text of every incoming and outgoing SMTP connection.

          • McDanno says:

            You have notify_classes=2bounce set. What's happening is that you're receiving the bounce from the remote server, which reaches you, but then for some reason the bounce cannot be delivered to the sender. When that happens 2bounce says "send this bounce to postmaster@mydomain (or whoever you have configured in 2bounce_notice_recipient)". So you're initiating that connection back to SES, yes. I'd start by figuring that part out; why is the bounce not deliverable to the sender?

            [Bounces mirrored this way go with a blank MAIL FROM by default, ie, MAIL FROM: <>. Amazon ignores that part of the RFC, because Amazon, and rejects it.]

          • McDanno says:

            Oh, and I forgot to mention: debug_peer_list is the option that will log transactions at a number of levels - hosts, ips, domains, netmasks. If you can't figure out the first thing on your own.

          • tb says:

            Translated to human speech, the report says: "cerebrum.dnalounge.com tried to send a message from MAILER-DAEMON@cerebrum.dnalounge.com to jwz@jwz.org. It tried to send it to email-smtp.uswest-2.amazonaws.com, but it got rejected with error 501 Invalid MAIL FROM ..."

            So, this is what probably happened:

            1. your machine tried to send the original message, using root@dnalounge.com as sender

            2. SES couldn't deliver it, and bounced it back to root@dnalounge.com

            3. your machine received the bounce, expanded root@dnalounge.com to jwz@jwz.org, and tried to send the bounce back to SES

            4. SES rejected the SMTP session with error 501

            5. your machine had nowhere to send the bounce, so you got the error

            Why did it try to send to jwz.org through SES is hard to tell without seeing your postfix config. Is jwz.org defined as a local domain?

            • jwz says:

              Yeah, #3 should definitely not be happening and I can't see why it would. Which is why being able to see the full SMTP log would be incredibly helpful. When I turn on debug_peer_list it prints a bunch of noise that bears no resemblance to that.

              • tb says:

                I usually use:

                egrep 'postfix/smtp.*(: > |: < )' /var/log/mail.log

                to see the smtp commands from debug_peer_list in postfix logs, but I don't think it will help you very much. To find out why did postfix decide to send a message to a specific transport, it would probably be more useful to add -v to trivial-rewrite in master.cf and reload. You should then see in logs what does it look at and where does it route a message.

                • jwz says:

                  Ok, here's what's going on: the way I am routing my bulk mail through SES is by putting an "X-DNA: bulk" header in certain outgoing mail, then doing:
                  header_checks = pcre:/etc/postfix/header_checks
                  which contains:
                  /^X-DNA: bulk/ FILTER smtp:[email-smtp.us-west-2.amazonaws.com]:587

                  So Amazon sends me a bounce, and the original message is attached to it. But header_checks is matching that "X-DNA" header in the attachment. The manual says "These are applied to initial message headers (except for the headers that are processed with mime_header_checks)." Which sure sounds like it should not be doing that. Especially since nested_header_checks also exists.

                  I tried using mime_header_checks but that doesn't fire on my X- header.

                  Any ideas?

                  • tb says:

                    By default, nested_header_checks and mime_header_checks are set to header_checks, so they are probably on (you can check with postconf).

                    If you don't use them, you can just add something like "nested_header_checks =" to main.cf, and that should help (or you can make a separate table for these checks if you need them).

                  • jwz says:

                    AHAAAAA! That was it! Thank you!

          • Ralf Hildebrandt says:

            What you can do is to use:

            debug_peer_list = ipaddress, ipadress2, ...

            in main.cf. It enables (very) detailed logging. Problem: You need to know the IP beforehand.

      • jwz says:

        I tried setting myhostname to mail.dnalounge.com, and also verified "cerebrum" and "mail" as domains in AWS. No go. (I suspect that by "domain" it means "TLD" rather than "hostname" so that's probably redundant anyway.)