spamassassin

Dear Lazyweb,

How the hell do I get SpamAssassin to work properly with Postfix? I've been through this several times before, but as far as I can tell, it's never worked right. Here is what I want:

  • Postfix feeds all locally-delivered mail through SpamAssassin;
  • Each recipient user on the system can have a .spamassassin/user_prefs file that uses directives like "required_score" and "blacklist_from" and "score". (Note that I said that I want this to be per user, not global. His blacklist is not her blacklist.)

Sounds simple, right? Sounds like the setup everyone in the world must have, right?

I can't get it to ever read the user_prefs files. And my mail log is full of errors about not being able to write lock files in ~user/.spamassassin/.

<LJ-CUT text="What I've tried... --More--(31%) ">

What I've tried:

  • /etc/postfix/main.cf:
    mailbox_command = /opt/local/bin/procmail

    (I see a lot of pages saying to use master.cf for this instead, but as far as I can tell, that would preclude a per-user user_prefs file ever getting consulted. Also the various FAQs on that approach make it sound flaky as all hell.)

  • /etc/procmailrc:

      DROPPRIVS=yes
      :0fw
      | /opt/local/bin/spamc -x -s 1000000

    (I tried it with and without DROPPRIVS.)

  • spamd: I have tried launching it as either
    /opt/local/bin/spamd
    or
    /opt/local/bin/spamd --user-config --username=postfix --groupname=mail

  • /Users/XX/.spamassassin/*: Owned by XX; I have also tried putting them in group "mail" and making them group-writable. Didn't help.

Here's the kind of crap that is getting logged:

    spamd: connection from localhost [127.0.0.1] at port 61095\n
    spamd: processing message <20090304090252.D91F4301D6@neptune.pollstar.com> for caroline:27\n
    locker: safe_lock: cannot create tmp lockfile /Users/caroline/.spamassassin/auto-whitelist.lock.cerebellum.dnalounge.com.8462 for /Users/caroline/.spamassassin/auto-whitelist.lock: Permission denied\n
    auto-whitelist: open of auto-whitelist file failed: locker: safe_lock: cannot create tmp lockfile /Users/caroline/.spamassassin/auto-whitelist.lock.cerebellum.dnalounge.com.8462 for /Users/caroline/.spamassassin/auto-whitelist.lock: Permission denied\n
    bayes: locker: safe_lock: cannot create tmp lockfile /Users/caroline/.spamassassin/bayes.lock.cerebellum.dnalounge.com.8462 for /Users/caroline/.spamassassin/bayes.lock: Permission denied\n
    spamd: clean message (0.0/5.0) for caroline:27 in 4.2 seconds, 24916 bytes.\n
    spamd: result: . 0 - HTML_MESSAGE scantime=4.2, size=24916, user=caroline, uid=27, required_score=5.0, rhost=localhost, raddr=127.0.0.1, rport=61095, mid=<20090304090252.D91F4301D6@neptune.pollstar.com>, autolearn=failed\n

(I'm especially impressed at the spurious \n on each line. Kwality.)

I have no idea how to tell what's running as who at the various stages of this pipeline.


Update: I think there were at least two missing pieces: spamd needed to be launched with -H, and procmailrc needed to both DROPPRIVS and run spamc with -u $LOGNAME. So my final, working config is:

  • /etc/postfix/main.cf:
    mailbox_command = /opt/local/bin/procmail
  • /etc/procmailrc:
    DROPPRIVS=yes
    :0fw
    | /opt/local/bin/spamc -u $LOGNAME -x -s 100000000

  • /opt/local/bin/spamd -H -s local7

  • /etc/syslog.conf:
    local7.*     /var/log/spamd.log

  • /Users/XX/.spamassassin/*: Owned by XX

  • /opt/local/etc/mail/spamassassin/local.cf:
    bayes_path /var/spool/spamassassin/bayes
    bayes_file_mode 0666

  • chown -R postfix:mail /var/spool/spamassassin
    chmod -R a+rw /var/spool/spamassassin

    (not sure if this is really necessary)

Tags: , , ,

27 Responses:

  1. blarglefiend says:

    My recollection of this stuff from last time I bothered trying to make it work was that spamd and user configs didn't really play very nicely together.

    The options seemed to be "call spamassassin for every delivery" or "store userconfigs in a database".

    This may have changed but I wouldn't count on it.

    • jwz says:

      I think I am calling spamassassin for every delivery. Or rather, calling procmail, which is calling spamc, which talks to spamd on a pipe, then writes the file in /var/mail/. How is that different?

      • blarglefiend says:

        There are two ways to run SpamAssassin. The simple way is to run "spamassassin" and pass it the message, the second is to have a spamd running and call spamc to pass the message to the running spamd.

        The latter is the way to go if at all possible as the former involves starting a new perl process and bytecompiling everything for each message. But doing it with spamd used to not play nice with per-user configs unless they were stored in a database rather than on the filesystem.

  2. jwm says:

    It could be that spamc, run by procmail isn't running as root (or some suitably privileged user) either because procmail is not setuid root, or it drops privileges[1], or spamd itself wants access to the user directories, and isn't running as root (or drops privileges too quickly). I'd say the latter judging from the log (but not knowing for sure what produced it).

    [1] procmail can log extended diagnositics if a log file and the VERBOSE option is set. One of those diagnostic lines is Assuming identity of the recipient, VERBOSE=off. If it does that before running spamc, that might be the problem.

  3. jmason says:

    Which OS is this? MacOS? Could there be some security limit that's blocking certain writes, when executed from the mailbox_command? Current postfix code imposes stringent file size limits, at least, which in the past has screwed me over when an mbox file exceeded 100MB in length. I bet it's something related.

    Here's what I'm using (Ubuntu 8.04.1), fwiw. in /etc/postfix/main.cf --

    mailbox_command = /usr/bin/procmail

    in ~/.procmailrc --

    PATH=/bin:/usr/bin:/usr/local/bin
    MAILDIR=$HOME/mail
    LOGFILE=$MAILDIR/procmail.log

    :0fw
    * < 800000
    | /usr/local/bin/spamc_dev

    And spamd is running as --

    /usr/local/bin/spamd_dev --create-prefs --max-children 5 --helper-home-dir --socketpath=/var/run/spamassassin/spamd_dev.sock -p 784 --daemonize -d --pidfile=/var/run/spamd_dev.pid

    The "_dev" stuff is because I dogfood SpamAssassin 3.3.0, but prior to that I was using 3.2.x this way. I also use the spamc.conf file to specify default spamc command line switches; /usr/local/sa_dev/etc/mail/spamassassin/spamc.conf --

    -U /var/run/spamassassin/spamd_dev.sock

    in other words connect using a UNIX domain socket when possible.

    I wonder if procmail does something "clever" when run from /etc/procmailrc instead of ~/.procmailrc? At least it _does_ seem to be dropping privileges correctly, assuming user "caroline" really does have a uid of "27" as per this log message:

    spamd: result: . 0 - HTML_MESSAGE scantime=4.2, size=24916, user=caroline, uid=27, required_score=5.0, rhost=localhost, raddr=127.0.0.1, rport=61095, mid=<20090304090252.D91F4301D6@neptune.pollstar.com>, autolearn=failed\n

    Suggestions: ditch -x from the spamc command line. If SpamAssassin breaks, it's more useful to have a shitload of unfiltered mail rather than no mail at all.

    The \n thing appears to be a bug in MacOS' syslogd. Upgrading Sys::Syslog will work around it: http://rt.cpan.org/Public/Bug/Display.html?id=24431 .

    And finally: failures to open/write to the auto-whitelist are non-fatal anyway, fwiw. but it's worth noting that the line that's failing is just a simple open(LOCK,">filename");. Nothing special, no funky locking, just a simple perl open command that should be fine assuming permissions are ok.

    Bottom line: I suggest investigating the ulimits.

    • jwz says:

      Yes, MacOS. The user "postfix" has uid 27.

      • jmason says:

        ok. it's not limits then.

        spamd is started as root, right? You're not using the '--username=postfix' switch, still?

        • jwz says:

          I am currently using --username, yes.

          • jmason says:

            and the "/Users/caroline/.spamassassin" dir is writable by the postfix uid? note that the auto-whitelist lock code needs to create a file in that dir.

            aha. Actually I note you mentioned that "/Users/XX/.spamassassin/*" was writable, but no mention of the dir itself... is that it?

            • jwz says:

              All of those are writable by group "mail", which spamd was also running in. When I let spamd run as root, it fails trying to mkdir /.spamassassin, which seems like a remarkably stupid thing for it to be trying to do... Apparently now it's trying to put lockfiles there instead of in the individual users' directories?

              But, when it runs as root, the uid= line in the logs is the proper uid number, at least.

  4. nugget says:

    It's been years since I faced this issue, but I think the problem lies with procmail bugginess with EUID when running recipes from the global rc file.

    The only way I've found to resolve this issue is to invoke spamc from each user's individual ~/.procmailrc instead of from within the global [/usr/local]/etc/procmailrc. You can rationalize this as providing them with more control over whether or not they choose to use spamassassin for their email, but in reality it sort of sucks.

    • nugget says:

      Ah, checking another box it looks like you might have some success with placing this in /etc/procmailrc:

      :0fw
      | /usr/local/bin/spamc -u $LOGNAME

      $LOGNAME being a magic procmail variable containing username.

      • jwz says:

        I think the -u $LOGNAME might have been the trick... There's still some weirdness going on, but it's reading my prefs now!

        • edm says:

          Yes, if you want to have per-user, user-owned, user prefs, you need to both run spamd as root (so it can setuid() as required for each message it processes), and (b) have spamc tell it which user to change to. If spamc is run in the context of user delivery (eg, ~/.procmailrc) then passing the current user (the default spamc behaviour) is fine; but IIRC if you are using /etc/procmailrc then the filters are run as a uid other than the user receiving the message in at least some instances. (And if spamd is given "root" as the user to run as, it'll change to another one, "nobody" by default.)

          Last time I ran into this I ended up running spamd as a non-root user (eg "spamd"), making all the .spamassassin directories in advance owned by that user ("spamd"), and giving the person who owned the parent directory permissions to change the files through group/world permissions. Ugly, but it did avoid running spamd as root.

          FWIW, there is a spamass-milter which was designed to work with sendmail but can be used with most vaguely recent versions of postfix. It has its own limitations, but does have the advantage of running earlier in the mail reception process (ie, before mail delivery) and of attempting to auto-detect the right user from the delivery address and pass that to spamd.

          Ewen

    • alierak says:

      We do this at $large_university, rationalized further with a website where people can check the little checkboxes without having to understand their procmailrc.

  5. houdini_cs says:

    Well, here's what my install looks like:

    Important (?) main.cf parameters:
    mailbox_command = /usr/bin/procmail

    Excerpt of my .procmailrc:
    :0fw:spamass.lock
    | /usr/bin/spamc

    :0:
    * ^X-Spam-Status: Yes
    .maildir/.Spam/

    I'm starting spamd like this (as root):
    /usr/sbin/spamd -d -r /var/run/spamd.pid -m 5 -c -H -l

    That's on Linux, not OS X.

  6. lovingboth says:

    I tried and failed with this a while back (albeit on a Debian virtual box with less RAM than it could have had).

    The solution was to give up and install Postgrey to do greylisting - it turns out that this stops well over 95% of spam, so I never bothered to get SpamAssassin to work.

  7. giantlaser says:

    We have this working fine with Ubuntu 8.04.2. Hope this helps.

    First, in /etc/postfix/main.cf:

    mailbox_command = procmail -f- -a $EXTENSION

    Then in /etc/procmailrc:

    MAILDIR=$HOME/Maildir/
    DEFAULT=$MAILDIR
    DROPPRIVS=yes
    :0fw: spamassassin.lock
    * < 256000
    | spamassassin

    This seems to read my ~/.spamassassin preferences just fine. Procmail can also read my personal procmailrc and it overrides the global one. Perhaps you'd like a comprehensive tarball of configs?

    • giantlaser says:

      Note: we are not using spamd, just spamassassin. That may affect correctly dropping permissions.

    • giantlaser says:

      Note: Postfix runs as root, which calls procmail as root, which then drops priviledges to the user because of the "DROPPRIVS=yes" line. Spamassassin then runs as the user because procmail invokes it.

  8. pingback_bot says:

    User adameros referenced to your post from Guilt by association. saying: [...] here, as I am a mail admin and work with spamassassin daily, in addition to sendmail and postfix: http://jwz.livejournal.com/1015987.html But sadly, a couple years ago someone sent out a ban list of everyone listed as a friend and I ... [...]

  9. zenspider says:

    Assuming you got this working... any chance you can post or update with your final setup?