Carlos Fenollosa flips a table:Many companies have been trying to disrupt email by making it proprietary. So far, they have failed. Email keeps being an open protocol. Hurray? No hurray. Email is not distributed anymore. You just cannot create another first-class node of this network.
Email is now an oligopoly, a service gatekept by a few big companies which does not follow the principles of net neutrality. [...]
I lost. We lost. One cannot reliably deploy independent email servers. This is unethical, discriminatory and uncompetitive.
*Stares in previously.*
I started running my own email host before OP did, and still do, due to the trend toward overzealous and haphazard filtering on messages I receive. "Other people's spam filter" has had nightmarish error rates since the invention of RBLs.
A surprising number of independent email operators can't send mail. I've had problems with self-hosted mail setups run by enterprises who can afford to know better, who were running behind outgoing spam filters. If you choose to block your own outgoing mail as spam, that's a you problem, not a me problem.
I did the exponential-banning thing OP proposed until last year, but it still allowed spammers who do use entire netblocks to send 5-digit numbers of spam between ban intervals. Immediately after the netblock was completely filled in with individual IP [1] bans, those IPs would go dark as the spammer moved to new netblocks, making the exponential ban ineffective (and hitting the next owner of the netblock with the permanent IP bans that the previous owner earned). Since the half-life of these netblocks is about 18 months, a 1-year ban per IP for each received spam seems to work well. In case of filter error, senders get unbanned if the recipient hits the "not spam" button on the message that triggered the ban (the ban defers messages, forcing spammers to time out sending them, and giving well-behaved email senders a second chance to deliver). There's a 5-digit number of IPs in banned state on average. There's also a whitelist, where the first non-spam email grants queuing privileges to the sender IP for a year, revoked when the user hits the "is spam" button. An IP on neither list can queue mail when there is capacity in the spam filter; otherwise, it gets to try again later.
Unfortunately, machine-learning filters don't work if the users aren't diligently hitting their spam/not-spam buttons. I watch in horror as GMail users use their "Spam" folder as their inbox, without telling Google that their good mail is mislabelled, or they use the "is spam" button to unsubscribe from a mailing list. The mountain of machine-learning data that Google builds their spam filters on has a lot of noise embedded in it.
IP blocking is not a problem that the big providers don't also have. Microsoft thinks spam delivery is so important that they'll mobilize their fleet of ~4000 hosts on every continent (except Antarctica) for the cause. All of those hosts earned their banned status, and we'll see if they're still spamming us when those bans expire in 2023.
[1] IP bans make some sense with IPv4; with IPv6, all of this works less well.
The point worth underscoring is that users interpret Spam to mean *unwanted* email regardless of why it was sent or whether they ought to read it, they don't want to. Why don't they want this email? Maybe it's a scam pretending to be confirmation of their Amazon Prime subscription, an "urgent" message from the IRS via or a request for them to send "samples" of their product to a Chinese company. Maybe it's a "personal" email from Donald Trump himself urging them to donate just $10 per month extra to stop godless communists.
But equally maybe it's eBay telling them that the $5 auction they clicked on last night while high is about to end, or their Netflix telling them that an "unknown" person signed into their account from their boyfriends new place when they were over there, or a dozen other routine things they just don't want to deal with. The mailing list they subscribed to during the two weeks they were really into knitting, and the recruiter who they led along for weeks because they hoped their employer would outbid any offer (they wouldn't). I don't think those are spam, you don't think those are spam, but the average user does.
And worst of all, Spam is the new pile of unopened envelopes left by the door of a person who knows they can't pay the bills. If I don't open the envelopes, if all these emails go in my spam folder, well, I didn't really default on the loan right? Facts aren't true so long as I avoid thinking about them. Even if you can pay your electricity bill, you may not want to read about what a great time Steve had on that date, so, Steve goes to spam, you can't deal with him right now. Mom has "important news" (maybe she's finally divorcing dad? Or maybe they're trying again? You don't want to know) but has discovered that you're never going to pick up her voice messages? Into the spam it goes.
When I say "worst of all" I don't mean for anti-spam technology, who cares about that, it's a lost cause. I mean it's worst of all for our society, we helped people unhealthily ignore their problems.
If you put me on your mailing list because I had to give you my email address to register, and then you send me junk email, and I try to unsubscribe, but you either ignore my unsubscribe, or you make me login with junk credentials that I have to do work to find, or you tell me after the unsubscribe that it will take 7 to 10 days to process then fuck the shitty website cause at that point I know you are lying to me and you are getting marked as spam. You get one chance to make unsub work properly. Don't put this back on the users.
I'm sorry, but it's hard to summarize all the wrong in this article.
Yeah, I get it, self-hosting email in 2022 is *hard*. It's has to be, because of the spambags. I do it, and I've had deliverability issues occasionally because I forgot to tick some box somewhere and accidentally forgot to deploy a DMARC record or some such and gmail got all pissy about it and dropped my mail in the spam folder. But the author's insistence on returning to 1999 is laboring under a pile of misconceptions that make his "solutions" part of the problem. To take them point by point:
1) "Keep spam filters and crowdsourcing/AI." Well, of course, but you need more. As Zygo says above, users aren't doing what you think they should be doing with the spam/not-spam button. When I worked at a company whose secret sauce was user feedback on spam, we discovered that it was simply insufficient. You need both a critical mass of users to overcome the malicious ones (and yes, there are lots of malicious unblockers out there actively trying to get spam to the inbox) and a robust system that blocks on other signals, including sender IPs. I'd argue from personal experience that in at least half the cases the latter was more important than the former.
2) "Change blacklisting protocols so they are not permanent." Yeah, doesn't work. Bullet-proof hosters exist, and this gives them what they want. Especially laughable is a "ten minute" first block. That's a blip in a big spam run. That's equivalent to "let it all through." If you do exponential backoff, the first block has to be more than a week (I'd recommend either two weeks or a month.)
3) "Blacklists should not include whole IP blocks." No, get out. Spammers often own or lease an entire /24 or larger, and rotate IPs. Not only that, collateral damage is a strategy that has been used since at least 2000 (and the author should know this) by unscrupulous providers who cater to spambags - mix legitimate customers in with spammers to keep recipients from blocking ranges quickly. If you're a customer of such a provider, scream at them for helping spammers. If they don't do something about it, take your business elsewhere. To quote a famous internet denizen, "You can do it. I *believe* in you."
4) "Stop blackholing." Well, I guess that's one way to get a spammer to be RFC-compliant. Set up legitimate sender domains and get nice reports on who's blocking your mails so you can measure the effectiveness of your spam runs. Otherwiwse, see 3) above.
5) "There should be a recourse for legitimate servers." I agree! Some big providers do provide this, too. Google doesn't, because they are terrible at UX, but Microsoft provides a great "postmaster tools" guide (postmaster.outlook.com) for how to make sure your email gets delivered to their various systems and what to do if it's not working, to name one example. Others go one step further and provide a remediation form that will get you reviewed and whitelisted (at least until your misconfigured system sends more spam.)
Finally, since the author has been aware of all internet traditions as long as you and has been. [points gun at astronaut] There is no law (American or otherwise) requiring email to be delivered. If someone doesn't want to hear what you're saying, there is no requirement for them to listen. That includes self-hosters, that includes microsoft and google. Sucks, but true.
Again, I understand that it's not easy to do this any more, but it is not impossible. The rules have changed considerably in the past 22 years out of self-preservation, and it's completely fair to say "I don't want to deal with this any more, so I'm hanging it up." But don't propose fucking everything up for everyone because you want it to be 1999 again.
And yes, I deliberately focused on the technical aspects of his argument without diving into the social ones. I have other shit to do. Maybe later.
Ok, I'm sorry, I have to pick this nit, because... NO. Just no. Back before I half-surrendered and began paying Amazon to allow me to relay (some of) my outbound mail through SES, I was constantly getting re-added to the MICROS~1 blacklist. Filling out their stupid form would sometimes get me off the list for a few days, and then I would almost immediately get blacklisted again. Getting delivery to any MICROS~1 properties to work was absolutely impossible.
So if that's what passes for a "great" system, uhhhhhhhhhhhh
It's okay, I understand. I remember fighting with that stupid form too. They appear to have heard your complaints and responded by...getting rid of the stupid form, near as I can tell.
But, whatever they've done in the last year or so seems to have improved deliverability a lot, at least for me. I still get a little leery when I send an email to a lawyer here (who *all* seem to use outlook 365) and a secret thrill whenever they reply. So whatever combination of magic incantations I've set up on postfix and dns are doing the trick with them for now anyway.
My self-hosted mail server got blocked by Microsoft (for the 14th time) just a few months ago. I managed to get “temporarily” removed yet again, with no response to how I can more permanently be removed or what magic thing I could do to better prove to them that I’m a legit little family server.
I signed up for all the free notification service things they have, but none of that seems to have changed the situation.
A good place to start is mxtoolbox.com/emailhealth . Put in your domain and see what's broken. Anything red should be fixed. Many things yellow should be fixed but some don't matter - for instance, mine complains that my DMARC policy is none and something about SOA TTL which, "who cares."
My wild-ass guess is that at least one of your SPF/DKIM/DMARC sigils is not correctly inscribed.
You can get out of the Office 365' blocklist quite easily.
But good luck getting out of the Hotmail one.
Yes, Hotmail now run on the Office 365 infrastructure but they have different antispam rules.
Is there a list of ISPs who lease IP blocks to spammers? I have business with a couple, and I'd like to know if they're doing that. I've always assumed that spammers use compromised computers to relay e-mail, I was fully unaware that ISPs were actually shamelessly lending their IPs to spammers.
All of them.
No, all of them, really.
The sales department at any major (or minor) network provider does not care. They're paid on commission. There's a complaint process that takes weeks or months to work through and the lawyers will get involved. You won't lose your commission if the customer gets TOSed eight months down the line and you've already forgotten their name.
The call is coming from inside the house. We've known this for decades now-- Sanford Wallace bragged about it and nothing stopped him more than temporarily until he finally went to jail in 2016.
Several providers publish lists that you can query with RBL lookups. Spamhaus SBL is "a list of spammer-owned netblocks" and a similar product is available from several providers. SBL is kind of redundant--hosts listed there attempt spam delivery up to 10 times per second, so I know who they are without doing any RBL lookups, thanks.
(I wonder who the hosts on SBL are sending mail to in 2022...filtering them is utterly trivial and the top retail email providers all do it, so 99% of the audience for these spammers is going to be spamtraps and people who maintain the SBL lists...who's still buying what they're selling?)
Unfortunately, most netblocks are "no data" or "a homogenous mix of good and bad mail and stale data." If you combine netblock or RBL lookup result with other factors in a machine-learning model, you get a robust statistical estimate of how spammy the RBL population is, but that's not useful for testing individual messages for spam. The machine learning models usually ignore the RBL result because they pseek factors that are accurate predictors of spam, and RBLs aren't.
Back in the day, RBL lists of dialup and retail broadband were useful, though it meant you would get less mail from people running SMTP hosts in their basements. Retail broadband providers block outgoing SMTP access by default now. Retail VPS hosting is going the same way: there are lists of VPS provider netblocks available for filtering, and new VPS customers don't get outgoing SMTP access by default, because some percentage of all retail customers are commercial spam operators.
Thanks for the rebuttal.
The vicseral reaction I had from this was: It's been 10 years since I left the world of spamfighting and commercial email hosting - and it doesn't seem like anything has changed.
If the EU’s Digital Markets Act passes, a lawsuit in the ECJ against Google, complaining about their anti-competitive spam blacklisting, might be enough to force some change. Maybe.
In other "under attack stand by" news, I spent almost an hour on the weekend trying to figure out how to make the RSS feed of this blog work again. Firefox works, wget and w3m don't. I got this far:
...and then I gave up. Did you know that wasting time guessing people's bullshit random anti-DoS rules is not fun? It's not fun.
Don't omit the trailing slash.
Thank you.
Speaking as among other things the former postmaster@mail.com back in the stone ages, let me just say:
Sigh.
Every "simple" suggestion that the author offers here is, as they say, not even wrong. We tried that. No, we also tried that. Nope, we tried that too. Trust me, we tried every possible variation on whatever half-assed idea you just pulled out of your ass (we, also, enjoyed pulling fun ideas out of our butts) and it didn't work, it didn't scale, and you can't even begin to imagine the weird failure cases.
I mean, he's right about the fact that self-hosting your own email server in 2022 is a mug's game. But that was already true in 2012 and was closer than anyone was willing to admit in 2002: they paid me to play that particular game of 4-D whack-a-mole and walking away from it was the single best professional decision I ever made.
The ugly reality here is that SMTP email by its very nature became (and remains) an ecosystem that forced unending evolutionary improvement on both sides of the spam game. And like all evolutionarily successful strategies, the collection of tools we have right now is ugly, ungainly, prone to failure under unexpected stresses and above all does not give a shit about the comfort of individuals that have to try to use them.
A strange game, the only winning move is not to play. And that's basically what happened: the "winners" of the spam wars were non-SMTP, proprietary messaging services, primarily whatsapp/messenger and imessage for now, although they're facing their own spam problems.
This is, I will 100% admit, zero comfort to any small business that built customer-facing workflows on top of SMTP like our gracious host has. It sucks that you have to tithe 5% of your revenue and/or a nontrivial amount of person-hours in perpetuity just to use what was built as an open protocol. But the email providers are just trying to keep their heads above the same shit-filled waterline here.
If only it were that simple, though. I am in a situation where, if I wanted to solve my current set of problems by throwing money at it, I'm not even sure how. I don't have any real sense of what all would break, because the services on offer do not appear to be drop-in replacements that provide "your mail server, but works".
Yeah, as far as I've ever been able to tell that's not a product category that exists, more's the pity. Specialization has set in -- the set of people who want to handle your outbound customer-facing mail sending (mailgun, mailchimp, amazon etc -- and there's some further specialization between broadcast marketing mail and point-to-point "here's your receipt" commercial mail) is disjoint from the set of people who want to run your internal employee-facing mail receiving (google, msft, protonmail, etc) which in turn is different from the diminishing set of people who will run "mailing lists" as we once understood them and yet again different from the set of people who want to handle helpdesk-style tracked conversations with individual customers.
I think in most cases you can with some elbow grease get a GSuite domain to kinda/sorta function like "my email server, but shit works" but it's a painful process if what you're used to doing is just having a postfix/dovecot server that does what you tell it and of course what google actually wants you to do is contract through some of the aforementioned companies via their "marketplace" to do those parts for you for a nominal fee on top of google's $12/user/month in perpetuity.
Can confirm, I did this for money from 2003 to 2010 and it already sucked by the end of that. Now I still run my own personal email box for whatever reason that rarely makes sense, but it no longer hosts mailing lists like it used to because that game ended up totally not worth the effort. Honestly I'm just waiting for the day that my IP block gets blacklisted by one of the big providers, because then I'm going to have to move my mail somewhere where I have less control and can do less fun experimental filtering.
Yet Another chapter in that long book, "Rough consensus and running code is a poor basis for critical infrastructure." The problem is SMTP, and the fact that it never had any kind of sender authentication.
Yes, I know. A lot of people want anonymity. You have a right to be anonymous. But you don't have a right to anonymously pester people who don't want anonymous pestering.
The solution to spam (if there is one) is going to involve reliable sender authentication. I'm not holding my breath.
Erm, no? This is (maybe) a solution to phishing, but not to spam. I do get tons of non-anonymous spam, with DKIM/SPF passing with flying colors from perfectly configured mail servers. Now you can look up the registrar of that domain and... sue them? You think the country they are located in will care? Spam is not just a technical problem.
I didn't say it was the solution, I said the solution would involve that.
Authenticated senders implies the existence of a whole lot of infrastructure for connecting senders with real-world entitles (something like what X.500 was supposed to be, i.e. administered by responsible humans who have resources to match applications for certificates with real-world entities), and if you can whitelist a set of real-world entities who are not going to spam you, you will not have a spam problem.
Connect those meatspace identities with some kind of reputation tracking (again, not something that attempts to be automated) (maybe this is something you have to pay for) (maybe this is something you want to pay for, to get the algorithms you want) and you can effectively blacklist real-world entities because once they've trashed you (or another subscriber to whatever algorithms you use) they can't hide.
As I said, I'm not holding my breath. And having typed all that out and looked it over, maybe what I meant to say was "We are so fucked."
This is just crap. I have had the same pair of mail addresses for more than two decades. For at least the first ten years of that period I made no attempt to obfuscate them in public fora, and I still make no really serious attempt to do so. My email is not hosted by google or whoever so I'm not benefiting from whatever dark magic they do. And of course I get spam – perhaps I get ten times as much spam as real mail, I don't know.
And I don't know because the spam filter (which is not some secret magic: it's whatever macOS mail has) by-and-large works. Some still leaks in: I might spend a few minutes a day (but, really, less) marking messages as spam and a few minutes a month adding uninteresting-but-not-spam messages to my 'just delete it' rule. Very occasionally I look at the junk folder (which gets deleted after ?24 hours?) for false positives: I have found them but not for a fairly long time. This is, to put it mildly, a small fraction of the time I waste on other stuff. Spam is a non-problem for me. Ten years ago it was, but today, no.
Now of course I am not a big famous internet personality: I am sure they get a lot more spam than I do. I am quite good at classifying stuff that actually is spam as spam, not just stuff I do not want to see, for which the just-delete-it rules work fine.
I am, in fact, average. And spam is a non-problem for average people. Email spam is solved.
What's going on is something else. And it's kind of obvious I think: googlebook really like it if all the email that gets sent goes via googlebook so they can harvest all sorts of information from it. And, oh look, all sorts of reasons duly appear, as if by magic, for that to happen.
There's a vast penumbra of email blacklists out there, all of them in my experience run by completely unaccountable assholes drunk on their own power and made vastly overzealous by their perceived righteousness. I wonder, has it ever occurred to any of the titans of the new internet to run a mail exchanger whitelist? I can't imagine it'd be too much work to run hourly scans on an IP to test if it's relaying mail other than for its listed domains and to check that spam isn't currently being sent with valid DKIM signatures from that IP.
I'd be happy to pay the vig to a Cloudflare, a Google or an Amazon as long as it isn't too much because I'm already paying the vig to have Google host my mail as it is.
He fought the fight for a long time.
I gave up long ago, might be a decade, maybe ~15 years ago.
I thought maybe I had finally lost this war today when Gmail bounced a message, saying there was a problem with my SPF and/or DKIM records, thanks so much for not specifying which. I managed to fix it though. I'm sure a day will come when I'll lose for real, but it is not this day.
BTW email blacklists are bullshit, solve no problems, and should not exist.
What's really fun is when one of the big three email gorillas blocks all or part of one of the other three email providers.
Don't forget that a secondary problem, even if SPF and other problems are out of the way is that AWS, SES, and EC2 are purposefully downgraded in reputation for sending mails. I saw an awful lot of "low reputation" errors when trying to diagnose these problems from a variety of clients. It is very, very hard to get higher reputation when forwarding into Google.
I've been subscribed to the mailop list for a few years and it's been an eye opener. The big guys do all the bad things and worse to little guys even if every form is filled out and acronym configured. They just don't care. Join the list and read the archives and become depressed.
https://list.mailop.org/listinfo/mailop