Configuring Netscape Mail On Unix.
Jamie Zawinski <jwz@netscape.com>
Comments and suggestions to x_cbug@netscape.com.

Before you attempt to use Netscape Mail in a Unix environment, it's important that you understand the assumptions it makes about mail delivery. If you customize the way your mail is delivered, or if you manipulate your mail spool file in the wrong way, you could lose mail, and nobody wants that.

If you are receiving mail from a Unix POP3 server, but are reading mail with a Windows or Macintosh version of Netscape Navigator, this information does not apply to you. This document is specifically for people who are reading mail and browsing the web with a Unix version of the Netscape Navigator.

This document attempts to explain a few interrelated things:

• First, it gives a basic overview of how mail delivery works, to explain the terminology and concepts used later.
• Next, it describes the two methods available for downloading mail: POP3 and movemail.
• Then, it lists some dangerous things you should avoid, and explains why they are dangerous.
• Finally, it goes into some technical details about configuring movemail on various systems; you probably only need to worry about this part if you are having problems.

• Don't open /var/spool/mail/$USER as a folder.
• Don't make ~/nsmail/Inbox be a symbolic link to /var/spool/mail/$USER.
• Don't configure the system to deliver mail directly to ~/nsmail/Inbox instead of /var/spool/mail/$USER.

For an explanation of why these (and similar things) are dangerous to do with Netscape Mail, read on...

---

HOW MAIL DELIVERY WORKS

The programs that move mail between computers are called Mail Transport Agents (MTAs); the program which lets a user read their mail, and compose new mail messages, are called Mail User Agents (MUAs). Netscape is a User Agent. Most Unix systems use a Transport Agent called sendmail (though there are many others.)

When mail arrives for you, the Transport Agent will place a message in a mail spool file. Typically this is a file called /usr/spool/mail/$USER or /var/mail/$USER.

When it comes time to read mail, the User Agent must read messages out of that file. Should the User Agent want to remove messages from that file, rather than just look at them, there is a possibility of a collision between the Transport Agent and the User Agent: if both of them are writing to the file at the same time, the file could become corrupted, and messages could be lost.

Consequently, both programs must be careful to ``lock'' the file before using it, so that they have exclusive access to the file while they manipulate it: if one program has the file locked, the other program will wait until it is unlocked before manipulating it.

If you do something that causes the mail spool file to be manipulated without locking, you will lose mail! It is only a matter of time before it happens.

Later in this document, we will highlight some of the pitfalls you should avoid to ensure that all shared files are properly locked before being altered.

---

RETRIEVAL OF MAIL:
POP3

One way that many User Agents retrieve mail for the user is to contact a POP3 server on the mail host, and download that mail to the local system. (POP stands for Post Office Protocol.) This is very common on non-Unix machines, which don't necessarily have the resources or continuous connectivity necessary to run a Transport Agent of their own. However, it is often used on Unix systems as well, since it allows all mail to be delivered to a single host, while individual users can run their mail readers on whichever workstation they like.

If your site uses a POP3 server, you're done: simply enter the name of your server in the Options / Mail and News / Servers panel of the preferences, and Netscape will be able to retrieve your mail from that server.

When retrieving mail from a POP3 server, Netscape will download a message from the server, and write it to your ``Inbox'' folder (~/nsmail/Inbox). After the message has been safely written to the local disk, it will instruct the POP3 server to delete its copy (unless the ``Keep Mail on Server'' or ``Maximum Message Size'' preferences instruct it not to.)

---

RETRIEVAL OF MAIL:
MOVEMAIL

If you are on a Unix machine and you don't have a POP3 server, then you can have Netscape retrieve your mail directly from the mail spool file. This means that it reads messages out of the spool file (/var/spool/mail/$USER), copies them to your inbox folder (~/nsmail/Inbox) and then removes the messages from the spool file.

We call this process ``movemail,'' as that is the traditional name for a utility program that performs this action.

Some (but not all) Unix mail readers use a model that effectively treats the spool file and the inbox file as the same thing: rather than moving messages from the file into which the system has delivered them, they simply leave the messages there. Netscape does not work this way. If you attempt to configure Netscape to treat your spool file as ``just another folder'', you will get into trouble: proper file locking will not occur. Netscape does careful file locking whenever it removes messages from the spool file, but does not do file locking when manipulating ``normal'' folder files!

Things Not to Do:

Do Not open /var/spool/mail/$USER as a folder.

If Netscape Mail attempts to update the file at the same time that the Mail Transport Agent attempts to add a new message to the file, the folder could become corrupted, and mail could be lost.

Do Not make ~/nsmail/Inbox be a symbolic link to /var/spool/mail/$USER.

This is exactly the same situation.

Do Not configure the system to deliver mail directly to ~/nsmail/Inbox instead of /var/spool/mail/$USER.

This effectively makes Inbox be a ``spool file'' instead of a ``folder,'' and Netscape Mail is not prepared to deal with that. ``Delivery'' includes any mechanism by which some other program causes the file to be written to, including procmail and other .forward file magic.

Note that the ``Keep Mail on Server'' preference is not consulted when movemail is used; it is a POP3-only option, intended to ease the use of a secondary computer, for example, a laptop or home machine, in a situation where it is desirable to store most mail on another, primary machine. Since movemail requires the mail to be on the local file system, this option is not relevant to it. This option is explicitly not intended to be a way of combining the spool file and the Inbox into one.

The movemail Program

There are two ways in which Netscape Mail can move mail from the system mail spool file to the Inbox file. The first way is that Netscape can simply do it using code which is included in the Netscape executable (the ``built-in'' movemail method.) The second way is to use an external movemail program which you provide.

If you select the Built in Movemail checkbox on the Options / Mail and News / Servers preferences panel, then the builtin movemail method will be used.

If you select the External Movemail checkbox, you may provide the name of an external program to use instead. Typically, you will simply type movemail into the text field, and arrange for there to be a program called movemail on your path (for example, in /usr/local/bin/.)

On most systems, the internal movemail method will work fine. But on some systems with more complex security or locking requirements, the Netscape executable may not have enough access to the mail spool directory to write the files that it needs to write in order to do locking and remove mail messages. In this case, you will need to configure Netscape to use an external movemail program that does have the proper permissions.

In order to do locking, Netscape needs to be able to create a lock file in the same directory as the mail spool file: it will have the same name as the file it is locking, but with .lock appended to the name. Netscape assumes that if such a file exists, some other process has the file locked, and assumes that other programs will obey this same rule.

If Netscape does not have permission to create this file, or if Netscape does not have permission to write into your mail spool file, then you will need to use an external movemail program and give it extra permissions.

If you must use an external program, we recommend GNU movemail. This is the movemail program used by (and distributed with) all derivatives of GNU Emacs. If you already have Emacs on your system, chances are you already have movemail as well. For convenience, we provide the source and executable of the GNU movemail program; it is included with the Netscape distribution.

Important: Please note that Netscape Navigator and GNU movemail are covered by very different licenses. Netscape is a commercial product, while movemail is ``free'' software, covered by the GNU General Public License.

Typically, if you need to run an external movemail program, you will need to make it be setgid to the mail group to give it permission to manipulate the mail spool directory:

# chgrp mail movemail
# chmod g+s movemail

This will need to be done as root; you should talk to your system administrator first.

It is not safe to give the Netscape executable these extra permissions (in fact, Netscape will reject any extra permissions you attempt to give to it.) The movemail program, by being an independent program which is self-contained, easy to analyze, and distributed in source form, can more easily be believed to be safe to run with privileges.

On most systems which use lock files in the spool directory, it will probably be sufficient to make your mail spool directory be ``sticky'', which means that everyone has write permission to the directory, but one can only delete one's own files, not files owned by others. A ``sticky'' directory is one with the permission bits 01777. If the directory is sticky, it's likely that movemail does not need to be a setgid program (and the internal movemail method may work.) Again, consult your system administrator.

A Summary of Default Configurations

Our goal is for Netscape Mail, in its out-of-the-box configuration, to be able to safely retrieve mail from the mail spool on all systems on which it runs. But as you may guess, given the great heterogeneity in the real world, this is a difficult thing to do. We're still working out the details; here is what we know so far. If you know something we don't, please let jwz know.

System:	Mail Spool:	Writability:	MDA:	Lock Style:
AIX	/var/spool/mail/	setgid mail (sticky works)	/usr/bin/bellmail	dot-locking + lockf(???)
BSDI	/var/mail/	root	/usr/libexec/mail.local	flock
	No dot-locking; uses open() with O_EXLOCK as well as flock() with LOCK_EX.
HPUX 9	/usr/mail/	setgid mail (sticky fails)	/bin/mail	dot-locking
HPUX 10	/var/mail/	setgid mail (sticky works)	/bin/mail	dot-locking
IRIX	/var/mail/	setgid mail (sticky works)	/bin/mail	dot-locking
Linux	/var/spool/mail/	sticky	/bin/mail	dot-locking
OSF-1	/var/spool/mail/	sticky	/bin/mail	dot-locking + lockf
SunOS 4	/usr/spool/mail/	sticky	/bin/mail	dot-locking
SunOS 5 (Solaris 2)	/var/mail/	sticky	/bin/mail	dot-locking

(Some systems allow subdirectories in the mail spool directory: /var/spool/mail/$USER is actually a directory, and the ``real'' spool file is /var/spool/mail/$USER/$USER. In this case, the directory and file are typically owned/writable by $USER. OSF-1 supports this; I'm not sure which of the others do. If you know one way or another, please let us know.)

dot-locking means the file.lock convention described above is used.

lockf and flock are library routines for doing locking which have historically been extremely buggy in the face of NFS files. (Buggy not as in ``fail to lock'' but buggy as in ``could cause your program or whole machine to hang''.)

root means that only root can write new files to the directory.

sticky means that the directory has the sticky bit set (permission bits of 01777). Everyone has write permission to the directory, but one can only delete one's own files, not files owned by others (unless one has write access to those files.)

setgid mail means that the only way for a program to write a new file in that directory is if it is running with an effective owner or group which is the same as that of the directory (in this case, group mail.)

Some systems which, by default, use setgid mail will work if the spool directory is changed to be sticky instead. This is known to be true for AIX, IRIX, and HPUX 10.01; it is known not to be true for HPUX 9.03.

So, on the platforms on which we ship, the usual configurations will probably be:

Linux, OSF-1, SunOS 4.1, SunOS 5.1:

Internal movemail will work.

AIX, IRIX, HPUX 10:

By default, internal movemail will fail.

However, if the mail spool directory is made to be sticky, internal movemail will work. If not, an privileged external movemail will work.

HPUX 9:

Internal movemail will always fail.

A privileged (setgid mail) external movemail must be used instead.

BSDI:

Internal movemail will always fail.

But, the external movemail binary we ship won't work either, because it must use something other than dot-locks. And if the mail spool is NFS mounted, the user will be in a world of hurt (but they would be in that same world of hurt with /bin/Mail, too.)

The most unfortunate thing here is that the architectures listed above are the architectures of the machine serving the mail spool directory, which could well be different than the architecture on which the Netscape client is running, if the mail spool is NFS mounted.

So, Netscape assumes that the internal movemail will work; and if we are unable to read and write the appropriate files, we pop up a dialog telling the user that they need to get their sysadmin to reconfigure their mail spool directory, or need to use an external movemail program.

---

THE SOLARIS PROBLEM

Netscape uses the standard BSD mailbox format for its mail folders, on all platforms. (This is the format where lines beginning with ``From '' (From-space) are used to separate messages, and where any occurences of ``From '' inside a message body are replaced with ``>From ''.)

When using the internal movemail method, Netscape assumes that the mail spool file (in /var/mail/) is also in this format.

Unfortunately, Sun is alone amongst the Unix vendors in shipping a default configuration in which the spool file is not in the standard BSD mbox format. Solaris instead uses a similar, but subtly-incompatible, format which does not quote ``From '' lines, but instead rely on a Content-Length header to indicate the exact byte-position of the end of the message. (This method is fraught with serious technical problems not worth going into here.) Netscape does not currently support the Content-Length variant of the mailbox file format.

A side-effect of this incompatibility is that Netscape will sometimes split messages when it encounters a line beginning with ``From '' in the middle of a message (since the system was expected to have quoted those lines.)

If you are using a Solaris system, your sysadmin can configure it to use the older format, which is compatible with the rest of the Unix world, by making a change to sendmail.cf: simply add E to the mailerflags of the local (Solaris) MDA. Add E to F= on the line that reads:

Mlocal, P=/bin/mail, F=flsSDFMmnP, S=10, R=20, A=mail -d $u

so that it becomes:

Mlocal, P=/bin/mail, F=EflsSDFMmnP, S=10, R=20, A=mail -d $u

This will cause the Solaris MDA to properly quote ``From '' lines, and should have no serious side-effects on other mail-reading software.

This is not a problem when using a POP3 server, but only when using movemail. In the POP3 case, dealing with the format of the /var/mail/ file is the responsibility of the POP3 server, not the client.