Skip to main content

Posts about email

Caving and switching to Amazon SES

Electronic mail (email or e-mail) is a method of exchanging messages ("mail") between people using electronic devices. Email entered limited use in the 1960s, but users could only send to users of the same computer, and some early email systems required the author and the recipient to both be online simultaneously, similar to instant messaging. Ray Tomlinson is credited as the inventor of email; in 1971, he developed the first system able to send mail between users on different hosts across the ARPANET, using the @ sign to link the user name with a destination server. By the mid-1970s, this was the form recognized as email. For almost 7 years I have run my own email server sending and receiving emails on it's own IP address. A few years ago I switched to Digital Ocean as a VPS to reduce deliverability issues, recently the IP block I was in got blacklisted by zen.spamhaus. The age of my 100% independent email server is now over.

Big players make it hard for small servers

As outlined with this blog post Google is eating out email and the resulting discussion on Hacker News the big players appear to have no interest following the standard rules when it comes to emails. All the tools are designed for big users, Google and Microsofts's postmaster tools don't even register any statistics for a domain unless your sending over 200 emails a day.

Unfortunately we find ourselves in the same position when it comes to something as simple as an SMTP relay service. If you search online for what an SMTP relay is you will only see the marketing material that claims it is a tool for large marketers to deliver massive volumes of emails into the inbox of their customers. Not what it actually is, a forwarding service that allows multiple email servers to send from one IP address to outsource reputation management.

Clearly the main advantage of an SMTP relay should be small time servers can pool together to achieve the volume the big players want to see to be able to judge if your emails are "worthy" of being treated fairly.

Imagine my frustration when trying to find any service that directly caters to this and finding that they don't appear to exist. It's all pitched as a service for marketing email delivery, if you're not sending thousands of emails you are not the target market anymore.

Can I use Mailgun for my personal email address?

It’s not recommended.

There are plenty of hosted email services better suited for this than Mailgun; Rackspace Email, Gmail / Google Apps, Outlook, etc.

Mailgun is meant to be a tool for developers and their applications.

Fair enough if you don't have a personal email server but if you have a server used by a family? Not clear. Then when you do sign up for any SMTP service they ask for your Business name, website and business use.

In the end as I technically have donations listed on the about page of this blog I simply used that and the Transactional emails from my Nextcloud instance as a sample of what I would be sending.

Getting out of the sandbox was a thing

I initially tried signing up for SendGrid but they immediately lock new accounts and you have to contact support to even login, let alone send an email. So after they straight up ignored what I wrote in the support email I signed up with AWS then immediately discovered that even though I could log in and set things up I was isolated in a sandbox not allowed to play with the other children.

Even though when signing up I selected an Individual account for side projects they still wanted a business use. After I finished banging my head against the desk I submitted what info I could. After four emails where among other things I updated my Privacy policy page to include Opt-in/out language, I was denied.

We reviewed your request and determined that your use of Amazon SES could have a negative impact on our service. We are denying this request to prevent other Amazon SES customers from experiencing interruptions in service.

There was one chance. In the correspondence they asked very specifically

How do you handle bounces and complaints?

Given they appeared to be concerned with negative service impacts I discovered a method of Automating handling this based on this xenforo plugin. Using Node-red as an endpoint I can receive a notification of a bounce/complaint and shut down the server for manual review.

Informing Amazon of this change was enough to get through the bureaucratic hurdles .

Making the switch was easy at least

Thankfully this was the least of my issues. First I created an SMTP identity in AWS, added that to my postfix postfix-relaymap.cf and postfix-sasl-password.cf as outlined in docker-mailserver documentation. Add the Amazon domain key, SPF record and CNAME records for DKIM and I was off to the races.


Why I still host my emails myself

Electronic mail (email or e-mail) is a method of exchanging messages ("mail") between people using electronic devices. Email entered limited use in the 1960s, but users could only send to users of the same computer, and some early email systems required the author and the recipient to both be online simultaneously, similar to instant messaging. Ray Tomlinson is credited as the inventor of email; in 1971, he developed the first system able to send mail between users on different hosts across the ARPANET, using the @ sign to link the user name with a destination server. By the mid-1970s, this was the form recognized as email. Recently I have seen a fair amount of talk about email in general and self-hosting email ( Email is not broken, Email is broken and Why I no longer host my emails myself ). I started selfhosting email using kolab almost 7 years ago now and I would never go back to the bad old days of hosted email.

Using a well respected email stack/package like mailinabox, modoboa, mailcow or docker-mailserver is easy. They come with all the bells and whistles with very little low level stuff you need to configure yourself.

Some of the Advantages

Sieve Filters

Why so many providers refuse to provide something as simple and powerful as Sieve filters I'll never know. See wikipedia for an overview but basically it's the filters you have in Outlook and Thunderbird but on the server and with more options.

Backups

Many will tell you this is a disadvantage to selfhosting your own emails. I vehemently disagree, being able to backup your own emails without having to deal with the IMAP protocol saves you from countless headaches. You have to do backups for your install but you also get to backup for when you inevitably delete that one important email from 5 years ago.

Infinite email addresses

People pay for this service and praise google for allowing the . in email addresses allowing you to classify incoming emails based on the address used. Some nice examples are [email protected], [email protected]. If you have multiple accounts you can also do . addresses like [email protected] or [email protected]

No storage limits

1 gig email boxes? silly in this day and age. I just buy a new hard drive. Hosted email gets prohibitively expensive fast because the cloud is not meant for bulk storage.

Some of the Disadvantages

Spam

Honestly this is a solved problem for the most part, the big issue most people have is they don't train the filter. I more or less use: Dovecot: Anti spam With Sieve

  • With infinite email addresses and Sieve filters you can easily move untrusted emails into specific folders
  • Spamassassin when properly trained catches a large number of spam emails
  • Using Postscreen for greylisting along with the great Postwhite script I found I could drop drive-by spam emails by a massive amount whilst still receiving the majority of important emails quickly.

Deliverability

The main argument against selfhosting email is deliverability and set-up time. Deliverability is a pain thanks to Google and Microsoft ignoring standards and generally being bad digital neighbors.

The simplest solution is to go with a partial selfhosted solution. Receive all your emails on your own server but send your emails through a SMTP relay such as Mailgun or Amazon SES

Or You can follow best practices and after warming up your IP address with a good volume of emails send them yourself.

  • Don't send from a residential IP
  • rDNS (PTR)
  • DKIM - dkim.org
  • SPF - spfwizard.net
  • Valid SSL (i.e. Letsencrypt)
  • Add a MTA-STS record - Tutorial.
  • DMARC - dmarc.org
  • Send emails dual format, "Plain and Rich (html) text" when possible Thunderbird. - Google is picky about this one.
  • Avoid formatted links like google link instead use unformatted links https://google.com
  • Sign up for Google postmaster tools
  • Sign up for Microsoft's SNDS