(Normally POP3 was good enough for me, since I only tended to do intermittent and/or emergency email work from my phone, saving the heavy lifting for when I was back at my desk. But my main computer has been in the shop for a week, leading me to rely more heavily on handheld toys, and the lack of folder access just go too annoying to tolerate any longer.)
I have questions, dear Lazyweb.
1) When I send a message from Mail.app on my desktop, it appears to be storing the sent message in the "Drafts" folder, and marking it as unread, as if it has not been sent, even though it has. I know it was sent, because I got a BCC and can see the transport headers. So, WTF? There appears to be a MUA-generated copy in both "Drafts" and "Sent" (no transport headers) plus the copy that came back to my inbox (with transport headers).
I do have "store drafts on the server" and "store sent messages on the server" checked, because that seems sensible. I do not have "pretend sent messages are unsent drafts" checked, because it doesn't exist. 2) I am using Sieve to filter my mail into different folders on the server. Mail.app (on desktop and iphone) doesn't seem to update the "unread" badges on these folders until I have first manually selected that folder. This is just-slightly-less-than-useful behavior, since I can't tell whether I have mail without clicking on each folder first.
Mail.app on the desktop seems better at this -- the unread badges update sometimes, possibly even most of the time (but not always). On the phone, they update almost never (but not never).
Would things go better if my "INBOX" folder had sub-folders into which Sieve was filing things? If so, how does one accomplish that, since Mail.app won't let me drag a folder into INBOX on the IMAP server the way it will let me drag folders into other folders in the "On My Mac" section. Do I "mkdir" something on the server? Will Mail.app even understand these nested folders? I'm using Dovecot 2, and it appears to be using mbox files. Update:
Turning off "store drafts on server" seems the only solution to #1.
Still no solution to #2. However, switching from mbox to maildir allows me to make sub-folders. The incantation for that appears to have been to add this to /etc/dovecot.conf:
mail_location =
namespace {
separator = /
prefix = "#mbox/"
location = mbox:~/mail:INBOX=/var/mail/%u
inbox = yes
hidden = yes
list = no
}
namespace {
separator = /
prefix =
location = maildir:~/Maildir
}
Then do:
dsync -D -v -f -u jwz mirror 'maildir:~/Maildir' to make a "Maildir" copy of the "mail" directory.
After doing this, the folder that the IMAP server refers to as "INBOX" appears to be /var/mail/jwz rather than something under ~/Maildir/, and I'm not sure how to change that (or if doing so is wise). I suppose this is why I still can't create sub-folders of my Inbox, and don't know whether that would solve the "message counts" problem.
With regards to (1): I see the same behaviour with Mail.app on the desktop and a Google Apps GMail account via IMAP. It drives me up the wall. The only conclusion I've been able to draw is that Mail.app is just broken with respect to "Store drafts on the server", at least as far as some IMAP servers are concerned (GMail's IMAP implementation is somewhat "special", so I don't know how widespread this problem is). The only solution I've been able to fathom is storing drafts locally, and hiding the server-side drafts folder from IMAP clients.
Yes, I see the same thing, though it's not consistent - only happens sometimes, and until I mucked about and confirmed the mail really was sent it had me worried it wasn't sending stuff.
I don't think dovecot implements folders which can contain both messages and subfolders; just one or the other. The reason is that it doesn't use any clever naming in the filesystem to avoid collisions between files (mboxes) and directories (folders).
The only free imap server I know of which implements both is Cyrus (which is what the netscape mail server was descended from, btw.) It most definitely does not use mbox, if that matters to you.
Edit: it looks like dovecot supports multiple mailbox formats; perhaps switching it to Maildir will let you have both.
courier-imapd does what you want... buuuuut, as far as I'm concerned, Maildir is the only sensible mailstore for modern usage (unless you're operating something like gmail, which you're not).
Switch to maildir and see how many of your problems go away.
I really hope this wouldn't apply to jwz's systems, but - Maildir will start to fail horribly if you have several tens of thousands of messages in your systems. mbox actually performs much better if you have a sane indexing format. (Of course, if you're dealing with a gajillion messages, you should probably be using some sort of database-backed system, sigh.)
meh. I had millions of messages in my Maildir and it was fine, both reading it directly with mutt and over IMAP with courier-imapd and Mail.app/Thunderbird.
I think it's more a function of how you're accessing the mail than how many messages you have. Some IMAP servers may choke on that volume of mail. Most mail clients definitely will.
It's usually not the server qua server, but the filesystem and the access therein. Filesystems make crappy databases, which is what maildir does with its magic filenames.
No, I understand completely. I'm rejecting your assertion that "several tens of thousands" is the threshold. At peak I had 3.5m messages in a Maildir on a shared freebsd machine in a jail. It was No Big Deal™.
But, yes. If you're going to deal with that kind of mail volume, there will be things you need to think about.
Maybe it's a problem if you have (more than) several tens of thousands of messages... and it's 1995.
The latest (last) build of UW-IMAP supports mixed mode (messages and subfolders at one level of the hierarchy) with the MIX folder store format.
MIX is a high performance indexed store, and it outperforms naive Maildir by a good ways thereby.
We run thousands of users on UW-IMAP/MIX with very good success, indeed.
Regarding 2):
This wasn't the case a few version ago, however Mail.app is now the biggest piece of shit that Apple has crapped out of Cupertino in a very long time.
I use IMAP in pretty much exactly the way you describe, and had to stop using Mail.app because of the problem you describe, because of random hangs and crashes, and because it wanted to transfer 1 GB of read state for every subscribed folder on every startup. Somewhere around OS 10.4 they clearly put the "B" team on this, and it shows.
Apple, of course, is entirely unresponsive to bug reports.
As actual help:
I can't recommend anything that will help with Mail.app, and suggest just switching to a different client (which sucks because it's not as well integrated). Thunderbird is acceptable, but not great.
My solution was that I moved my entire infrastructure over to Zimbra. It's not too hard to beat their free Open Source Edition into working order and includes all the features I was cobbling together with Cyrus + SpamAssassin + Sieve + SquirrelMail + etc. I use the Zimbra web client instead of a desktop client, as I find it much more usable than any desktop client I've used.
You've probably spent enough time on your Dovecot++ solution that your sunk costs will prevent you from switching, but I really can't praise Zimbra enough. It has server-side "smart folders", so I can see them from my iPhone! It has CalDAV that actually works (with my iPhone)! It has search across my 10 GB of saved mail that is almost instantaneous! It's really, to borrow a Jobsism, magical and revolutionary.
If email is convenient you won't SMS as much on your iPhone.
I used to work for a place that ran zimbra for its mail, and I remember it being a total resource hog. How beefy a machine would one need to run a handful of accounts?
I run in in a Xen VM on my colo server. The colo server was the beefiest I could buy 2 years ago, so it's 2 quad-core Xeon with 8 GB of RAM. The VM is 1 core and has 2 GB of RAM allocated to it. It looks like right now it's using about 1.2 GB of that, with the rest in buffers, cache and free. It's used 916640s of VM time in 58 days, or 18% of one core. That's for about a dozen active users.
From this, I conclude that you need about 2 GB of RAM, but any reasonably recent processor platform is fine. In fact, TFM says "Production: Intel/AMD CPU 32-bit 2.0+ GHz, 2+ GB RAM, 10 GB free disk space for software and logs, mail storage space."
I'm not a fan of Zimbra at all. Probably because I like using a desktop client, hate webmail, and their "desktop" client is the biggest piece of shit out there, all the bloat of a MS Office product, all the limitations of a webmail client.
Far as I can tell the server portion works just fine.
Well, you can use the IMAP client of your choice, but that doesn't solve the Mail.app fuckage which the original post is about.
Dovecot only does nested folders if you're using Maildirs. You need to name the folders .foldername, it ignores non-dotted directories.
No idea how Mail.app handles it though.
Been a little while since I dealt with Mail.app, but last I did you could select a folder and from the Mailbox menu go to "Use This Mailbox For..." and it would allow you to mark which folders different messages states should map to. I always had to set that up manually as the default assumptions never mapped to my folder structure.
I have since done like most folks, said fuck administering an MTA, and let the Google Hive Mind pitch eerily accurate advertising at me.
1) sounds like a problem with your IMAP server. Mail.app does the "right" thing and deletes messages out of Drafts when the message is sent.
Normally, as far as the server is concerned, Drafts should be just another folder with no special handling. God only knows what Dovecot is doing, and god only knows how fucked up their mbox system is. I switched to Courier because Dovecot was being braindead.
I said it elsewhere, but as stupid as this sounds, try switching to Maildir and see if things get better. I'm guessing they will.
Any idea how to get Dovecot to convert from mbox to Maildir? The documentation is, as usual, bullshit. I thought it might be
dsync -D -v -f -u jwz mirror 'maildir:~/Maildir'
But no:
dsync(root): Fatal: Mail locations must use the same virtual mailbox hierarchy separator (specify separator for the default namespace)
I looked at that wiki page, and particularly enjoyed the bullshit about "this script requires a patched..." and I stopped reading. I can't remember how I did it, it's been ~6 years... plus I was converting from god-knows-what to courier. I think maybe you can use formail?
Probably, though, you want to find a script to do it for you, and figure out where Dovecot keeps its mboxes, and hope they're in a standard format. This script appears to be popular. If you're not allergic to DJB, you could check out your options here.
I am fully aware that this comment is mostly useless.
in the interest of providing slightly more useful content, I googled around for that error message, and came to the conclusion that you need to define a default namespace (whatever that means) with a separator defined (whatever that means).
This wiki page has some examples... the "mixed mbox and maildir" may do the trick to enable you to use dsync.
Yeah, that mostly did it, thanks. (That link got me close enough to figure the rest, anyway.)
Cool. Having done so, I think probably the right thing to do is revert to your old config, but with your Maildir in place of whatever mbox stuff was there before. I suspect that will alleviate some of the weirdness in your update.
You mean remove the "namespace" stuff and just use a "mail_location"?
I don't know how to phrase the "mail_location" line so that it knows about my Maildir folders plus my mbox /var/mail/jwz. As far as I can tell, the two namespaces are needed for that. Unless there's some way to remove /var/mail/ from the picture, of which I am unaware.
Oh, I see what you're saying. My fault for forgetting about the MTA's part in this whole fiasco. Can you configure your MTA to deliver into your Maildir so you don't have to bother with /var/mail anymore? Then you can run your sync thingy again and be done with it.
I don't have any idea. Postfix's mailbox_command invokes /usr/libexec/dovecot/deliver, so you'd think so...
Yeah, sorry, I'm already pretty lost here. Not to be That Guy, but trying to figure out wtf dovecot was doing is a lot of the reason I switched to Courier.
Ouch.
If you get desperate, and don't mind doing this the "wrong" way, a Maildir is a trivial filesystem layout. When I migrated from Cyrus, I just renamed the files and fixed the line endings, and it worked fine. You'll just have to split the mbox into files.
Dovecot adds pretty good caching on top of the filesystem layout, but it still supports old-style dumb clients that might mess with the folder. Which is great for scripting the thing from the shell.
Maildir is just file-per-message, and Dovecot's cleverness is auto-healing, so it detects when strange new arrivals appear in a folder (but you'll do better if you put files into the "new" directory).
To create a folder, do this: ike mkdir ~/Maildir/.${folder_name}/{new,cur,tmp}
To drop a message into $folder_name, you put it in ~/Maildir/.${folder_name}/new/${guid_filename}:2,
There are specific rules for constructing ${guid_filename} in the Maildir spec, all of which are irrelevant so long as the filename is unique. The filename needs to end in :2, which means the flags are version 2 (the only relevant version), and there are none of them at the moment (the letters after the comma).
Dovecot's maildir support is excellent. (But I'm using 1.2, because I am just using the Ubuntu build.) It is better than Courier's in my experience. Both Courier and Dovecot will allow Maildir folders to contain Maildir folders, although this is not really how things get laid out in the file system.
If you're using mboxes, a folder can't contain subfolders, because there is no smart about mbox that would allow a file to contain a directory.
It sounds like Mail.app assuming mail only ever arrives into INBOX. (Netscape had that bug for years, but Thunderbird doesn't, if I recall correctly. I say this not to taunt you, but to say, Thunderbird works for me now.)
If Mail.app has IMAP subscription support, you can try subscribing to your other incoming folders, which might turn on a less stupid mode, but this is just a guess. I would also guess that filtering into folders under INBOX will not trick it into doing the right thing.
Both Courier and Dovecot will allow Maildir folders to contain Maildir folders, although this is not really how things get laid out in the file system.
Really? It is with Courier, makes me wonder what kind of crazy crap Dovecot is doing.
They're doing the same thing. Dovecot's web site documents it as supporting Maildir++, as does Courier. See this one on Courier's web site.
A folder called foo will be stored in ~/Maildir/.foo, but if it has a subfolder bar, it will be in ~/Maildir/.foo.bar.
So for applications, bar is inside of foo, but it's not a subdirectory in the filesystem.
Ah, okay. My fault; I wasn't thinking deep enough. Typically ~/Maildir is itself a maildir, and like you said, ~/Maildir/.foo will be a maildir too... it's just when you get deeper than that (which I didn't really do) that it changes.
Ah. If you decide to go back, it makes the answer to your previous question easy: use POPFile and let it sort your mail (as well as spam filter it better than spamassassin IME).
Here are some discussions over on the Apple forums.
They come to the conclusion that it's not possible by fiddling with Mail.app or your IMAP server. There is this plugin which makes it possible, but it was last tested on Mac OS X 10.5, it's "not supported" any more. However, the author is willing to give you the source code. So good luck with that.
Thunderbird has a "Check for new mail in this folder" option which you probably wrote.
Well, that's marginally useful to 10.4 retrogrades like me, at least.
All it needs now is another hack to stop hard-coding its sent-mail folder to "Sent Messages" and then displaying this as "Sent" in the UI, alongside the actual IMAP folder called "Sent" used by my other MUAs, and maybe it'll just manage to be slightly less broken than Thunderbird. Sigh.
I've got ~/Maildir/.INBOX and ~/Maildir/.INBOX.subdirs by default on dovecot.
My dovecot.conf has
mail_location = maildir:~/Maildir
and no inbox= anywhere, so the default has the maildir .INBOX.*
I would think that removing the INBOX=/var/mail/jwz and moving the inbox=yes to the Maildir namespace would fix it:
# There can be only one INBOX, and this setting defines which namespace
# has it.
#inbox = yes
Trying that out on a test account first would be highly recommended, but you probably know that already :)
I have the same problem #2 with Courier, and it seems worse when there are several IMAP clients hitting the same mailbox at the same time, so if I have Mail.app on my home machine and also move messages around with another Mail.app or Thunderbird at work. The home Mail.app may or may not know about the changes until I enter the directories.
I found a couple of partial solutions, running famd on the server seemed to pretty much fix things, but every so often would run out of memory and I'd have to reboot my host, which was bad, and I was too lazy to figure out why. Also, someone suggests this small script to make Mail sync your account via cron. I would hope that forcing a sync would sync all inboxes, not just Inbox, but I haven't tried it yet, and it bugs me that the answer is "make cron run this thing every minute to fix Mail.app's inherent brokenness".
Though I have had very limited experience with it (liked To-Dos and overall integration, but ultimately had to go for something with Exchange), I thought I'd mention PostBox is on sale.
wow mail.app works for you? it hangs and crashes every time I try to launch it. piece of shit.
See here.
Well, all of that is still true. But given the absence of basic commands like "mark all read" on the devices I use, POP3 has become even more annoying than IMAP's technical deficiencies.