Technology (old posts, page 3)

As I Tinker I Learn, Somtimes I Even Write It Down.

Why I still host my emails myself

Electronic mail (email or e-mail) is a method of exchanging messages ("mail") between people using electronic devices. Email entered limited use in the 1960s, but users could only send to users of the same computer, and some early email systems required the author and the recipient to both be online simultaneously, similar to instant messaging. Ray Tomlinson is credited as the inventor of email; in 1971, he developed the first system able to send mail between users on different hosts across the ARPANET, using the @ sign to link the user name with a destination server. By the mid-1970s, this was the form recognized as email. Recently I have seen a fair amount of talk about email in general and self-hosting email ( Email is not broken, Email is broken and Why I no longer host my emails myself ). I started selfhosting email using kolab almost 7 years ago now and I would never go back to the bad old days of hosted email.

Using a well respected email stack/package like mailinabox, modoboa, mailcow or docker-mailserver is easy. They come with all the bells and whistles with very little low level stuff you need to configure yourself.

Some of the Advantages

Sieve Filters

Why so many providers refuse to provide something as simple and powerful as Sieve filters I'll never know. See wikipedia for an overview but basically it's the filters you have in Outlook and Thunderbird but on the server and with more options.


Many will tell you this is a disadvantage to selfhosting your own emails. I vehemently disagree, being able to backup your own emails without having to deal with the IMAP protocol saves you from countless headaches. You have to do backups for your install but you also get to backup for when you inevitably delete that one important email from 5 years ago.

Infinite email addresses

People pay for this service and praise google for allowing the . in email addresses allowing you to classify incoming emails based on the address used. Some nice examples are [email protected], [email protected]. If you have multiple accounts you can also do . addresses like [email protected] or [email protected]

No storage limits

1 gig email boxes? silly in this day and age. I just buy a new hard drive. Hosted email gets prohibitively expensive fast because the cloud is not meant for bulk storage.

Some of the Disadvantages


Honestly this is a solved problem for the most part, the big issue most people have is they don't train the filter. I more or less use: Dovecot: Anti spam With Sieve

  • With infinite email addresses and Sieve filters you can easily move untrusted emails into specific folders
  • Spamassassin when properly trained catches a large number of spam emails
  • Using Postscreen for greylisting along with the great Postwhite script I found I could drop drive-by spam emails by a massive amount whilst still receiving the majority of important emails quickly.


The main argument against selfhosting email is deliverability and set-up time. Deliverability is a pain thanks to Google and Microsoft ignoring standards and generally being bad digital neighbors.

The simplest solution is to go with a partial selfhosted solution. Receive all your emails on your own server but send your emails through a SMTP relay such as Mailgun or Amazon SES

Or You can follow best practices and after warming up your IP address with a good volume of emails send them yourself.

  • Don't send from a residential IP
  • rDNS (PTR)
  • DKIM -
  • SPF -
  • Valid SSL (i.e. Letsencrypt)
  • Add a MTA-STS record - Tutorial.
  • DMARC -
  • Send emails dual format, "Plain and Rich (html) text" when possible Thunderbird. - Google is picky about this one.
  • Avoid formatted links like google link instead use unformatted links
  • Sign up for Google postmaster tools
  • Sign up for Microsoft's SNDS

TOTP with sudo (Google Auth)

I was reading the posts over on and saw this post: Is sudo almost useless?. Typically I see sudo as a safety belt to protect you from doing something stupid with administrator privileges rather than a security shield. But that doesn't mean it can't be both

As with ssh, outlined in my previous post TOTP with SSH (Google Auth), you can certainly boost your sudo usefulness security wise by throwing 2FA via google-authenticator-libpam on top of it.

Install google-authenticator-libpam

On debian/ubuntu:

    sudo apt update && sudo apt install google-authenticator-libpam

Set-up your secret keys

We now need to create the secret key, this should not be kept in the user folder, after all what is the point of 2FA if the user we are authenticating can just read the secret files. In my case I keep them in the root dir

Replace the variable ${USER} if/when you create a key for a user other than the active one.

sudo google-authenticator -s /root/.sudo_totp/${USER}/.google_authenticator
sudo chmod 600 -R /root/.sudo_totp/

You will see a QR code/secret key that you can scan with a TOTP app like andotp, authy, google authenticator or in my case I added it to my yubikey. There are also your emergency scratch codes that you should record somewhere safe.

Enable in PAM

You now need to let PAM know it should be checking the codes. There are two ways to do this, Mandatory and Only if secret key exists. I have it as Mandatory any user using sudo MUST have a secret key

In /etc/pam.d/sudo add the following configuration lines to the end of the file.

# Use Google Auth -- Mandatory
auth required secret=/root/.sudo_totp/${USER}/.google_authenticator user=root

# Use Google Auth -- Only if secret key exists
# auth required secret=/root/.sudo_totp/${USER}/.google_authenticator user=root nullok

Bonus do this for su as well

You can do the same thing for su as well however obviously the user variable will be root rather than the user attempting to elevate their privilege's.

Setup the key as before, just for the root user

sudo google-authenticator -s /root/.google_authenticator
sudo chmod 600 -R /root/.google_authenticator

In /etc/pam.d/su add the following configuration lines to the end of the file.

# Use Google Auth -- Mandatory
auth required secret=/root/.google_authenticator user=root

Node-red - Phonetrack HomeAssistant Bridge

Node-RED is a flow-based programming tool, originally developed by IBM’s Emerging Technology Services team and now a part of the JS Foundation. I threw together a quick way to bridge GPS data from the android app GPS Logger between HomeAssistant and the Nextcloud APP Phonetrack.

Note: Returns the highest http response code thrown by either service, this can result in the GPS logger submitting multiple times if there is any issues with either Nextcloud or HomeAssistant

Extra Nodes used



In HomeAssistant follow the instructions found here to obtain the GPS Logger web hook url and add this to the config node

In Nextcloud after creating a tacking session click the link icon and fetch the link labled GpsLogger GET and POST link : and add this to the config node

Edit the http auth Node with your desired credentials.

GPS Logger

Go to Logging details -> Log to custom URL -> URL and add your Node-Red url: https://example.tld/node-red/gps_logger?latitude=%LAT&longitude=%LON&device=[Your_Device_Name_Here]&accuracy=%ACC&battery=%BATT&speed=%SPD&direction=%DIR&altitude=%ALT&provider=%PROV&activity=%ACT&timestamp=%TIMESTAMP - Note: edit the url and the [Your_Device_Name_Here]

Go to Logging details -> Log to custom URL -> Basic Authentication add the username and password you set in the http auth node

[{"id":"f5b2b1b.1f8895","type":"http in","z":"4a1f60d7.aaf398","name":"GPS Logger endpoint","url":"/gps_logger","method":"post","upload":false,"swaggerDoc":"","x":180,"y":180,"wires":[["d18bbbaf.bd8138"]]},{"id":"c8badb69.d6dc78","type":"debug","z":"4a1f60d7.aaf398","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1210,"y":240,"wires":[]},{"id":"2b58f523.f2713a","type":"http request","z":"4a1f60d7.aaf398","name":"Home Assistant","method":"POST","ret":"txt","paytoqs":false,"url":"","tls":"","proxy":"","authType":"","x":660,"y":260,"wires":[["b894f1e0.b35c28"]]},{"id":"7b5c470a.a86d7","type":"change","z":"4a1f60d7.aaf398","name":"Build HomeAssistant Query","rules":[{"t":"set","p":"headers","pt":"msg","to":"{}","tot":"json"},{"t":"set","p":"headers.content-type","pt":"msg","to":"application/x-www-form-urlencoded","tot":"str"},{"t":"set","p":"url","pt":"msg","to":"homeassistant","tot":"flow"},{"t":"set","p":"payload","pt":"msg","to":"\"latitude=\" & req.query.latitude & \"&longitude=\" & req.query.longitude & \"&device=\" & req.query.device & \"&accuracy=\" & req.query.accuracy & \"&battery=\" & req.query.battery & \"&speed=\" & req.query.speed & \"&direction=\" & req.query.direction & \"&altitude=\" & req.query.altitude & \"&provider=\" & req.query.provider  & \"&activity=\" & req.query.activity","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":260,"wires":[["2b58f523.f2713a"]]},{"id":"ee593f24.f06ca8","type":"change","z":"4a1f60d7.aaf398","name":"Set Phonetrack URL","rules":[{"t":"set","p":"url","pt":"msg","to":"$flowContext(\"phonetrack\") & req.query.device & \"?lat=\" & req.query.latitude & \"&lon=\" & req.query.longitude & \"&acc=\" & req.query.accuracy & \"&speed=\" & req.query.speed & \"&bearing=\" & req.query.direction & \"&timestamp=\" & req.query.timestamp & \"&battery=\" & req.query.battery","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":420,"y":220,"wires":[["f9487911.1c6ed8"]]},{"id":"f9487911.1c6ed8","type":"http request","z":"4a1f60d7.aaf398","name":"PhoneTrack","method":"POST","ret":"txt","paytoqs":false,"url":"","tls":"","proxy":"","authType":"","x":650,"y":220,"wires":[["b894f1e0.b35c28"]]},{"id":"1b9f05f9.a0fb4a","type":"http response","z":"4a1f60d7.aaf398","name":"","statusCode":"","headers":{},"x":1210,"y":280,"wires":[]},{"id":"d9a79cbe.4e6668","type":"config","z":"4a1f60d7.aaf398","name":"URLS","properties":[{"p":"phonetrack","pt":"flow","to":"https://example.tld/apps/phonetrack/log/gpslogger/__phonetrackid__/","tot":"str"},{"p":"homeassistant","pt":"flow","to":"https://home.example.tld/api/webhook/__webhookkey__","tot":"str"}],"active":true,"x":150,"y":140,"wires":[]},{"id":"b894f1e0.b35c28","type":"join","z":"4a1f60d7.aaf398","name":"","mode":"custom","build":"array","property":"statusCode","propertyType":"msg","key":"url","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":850,"y":240,"wires":[["fc27dbec.975ca8"]]},{"id":"fc27dbec.975ca8","type":"change","z":"4a1f60d7.aaf398","name":"","rules":[{"t":"set","p":"statusCode","pt":"msg","to":"$max(statusCode.$number())","tot":"jsonata"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1000,"y":240,"wires":[["35d1687f.3e79d"]]},{"id":"51a8ee1e.cb9c38","type":"comment","z":"4a1f60d7.aaf398","name":"Set URLS for HomeAssistant and PhoneTrack","info":"","x":250,"y":100,"wires":[]},{"id":"35d1687f.3e79d","type":"switch","z":"4a1f60d7.aaf398","name":"","property":"statusCode","propertyType":"msg","rules":[{"t":"gt","v":"200","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":1030,"y":280,"wires":[["c8badb69.d6dc78","1b9f05f9.a0fb4a"],["1b9f05f9.a0fb4a"]]},{"id":"d18bbbaf.bd8138","type":"node-red-contrib-httpauth","z":"4a1f60d7.aaf398","name":"","file":"","cred":"","authType":"Basic","realm":"","username":"","password":"","hashed":false,"x":220,"y":220,"wires":[["ee593f24.f06ca8","7b5c470a.a86d7"]]}]

You should probably know about LetsEncrypt DNS challenge validation

Everyone knows the basic way to renew a LetsEncrypt cert. Open port 80 and let LetsEncrypt connect to your server. But what if you don't want to open your network or you limit access to a handful of IP addresses? Well you can just use the DNS challenge validation, no need for web servers and no need for port wrangling.

For example I use the certbot-dns-cloudflare for my work intranet allowing it to remain VPN only.

Another great option is to use as it supports a massive list of dns providers and the ever popular duckdns out of the box.

Given in the past I found the most fragile part of my LetsEncrypt setup was making sure port 80 was accessible to LetsEncrypt I personally use this method even if I have a network accessible from the wider internet.

Splitting a Facebook event calendar

Node-RED is a flow-based programming tool, originally developed by IBM’s Emerging Technology Services team and now a part of the JS Foundation. My friends make liberal use of Facebook events, unfortunately I find the events interface impossible to navigate. Luckily they do have a .ics available, unluckily events you haven't accepted are mixed with events you have.

So I made a simple flow that splits accepted events from tentative events. This way I can subscribe to this .ics in my Nextcloud instance and give tentative events a different color. I find a calendar is much more usable.

I used the node-red-contrib-httpauth node. The main bit is contained in a function node, mainly because I couldn't figure out how to sanely do this with the split node.

Use is simple:

  1. Update the http auth node with your preferred user:pasword
  2. Add your facebook event calendar url to the http request node
  3. subscribe to user:[email protected]/nod-red/facebook/accepted or user:[email protected]/nod-red/facebook/tentative

[{"id":"30abd373.bd5524","type":"http in","z":"ce798f74.64a9d8","name":"","url":"/facebook/:request","method":"get","upload":false,"swaggerDoc":"","x":160,"y":300,"wires":[["77e1feab.382658"]]},{"id":"705a7847.bc5d","type":"debug","z":"ce798f74.64a9d8","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":430,"y":400,"wires":[]},{"id":"77e1feab.382658","type":"node-red-contrib-httpauth","z":"ce798f74.64a9d8","name":"","file":"","cred":"","authType":"Basic","realm":"","username":"","password":"","hashed":false,"x":240,"y":340,"wires":[["116bd38e.4b7dcc","705a7847.bc5d"]]},{"id":"e1172439.6bf4b8","type":"comment","z":"ce798f74.64a9d8","name":"Secure with Basic Auth","info":"","x":200,"y":380,"wires":[]},{"id":"116bd38e.4b7dcc","type":"switch","z":"ce798f74.64a9d8","name":"","property":"req.params.request","propertyType":"msg","rules":[{"t":"eq","v":"accepted","vt":"str"},{"t":"eq","v":"tentative","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":3,"x":430,"y":300,"wires":[["eaff2195.a68b38"],["eaff2195.a68b38"],["83c3be58.0fc9a8"]]},{"id":"4d794bb0.d8ef74","type":"function","z":"ce798f74.64a9d8","name":"Split Calendar","func":"msg.payload = msg.payload.toString('utf8');\nmsg.payload = msg.payload.replace(\"END:VCALENDAR\", \"\");\nmsg.payload = msg.payload.split(/(?=BEGIN:VEVENT)/g);\nmsg.calendar = msg.payload[0];\n\nmsg.payload.forEach(function(part, index){\n   if (part.includes(\"PARTSTAT:ACCEPTED\") && (msg.req.params.request == \"accepted\")){\n       msg.calendar += part;\n   } else if (part.includes(\"PARTSTAT:TENTATIVE\") && (msg.req.params.request == \"tentative\") ){\n       msg.calendar += part;\n   }\n});\n\nmsg.calendar += \"END:VCALENDAR\";\nreturn msg;","outputs":2,"noerr":0,"x":1000,"y":300,"wires":[["6b67410a.701ab"],[]]},{"id":"13bdf419.e491ec","type":"http request","z":"ce798f74.64a9d8","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"","tls":"","proxy":"","authType":"","x":830,"y":300,"wires":[["4d794bb0.d8ef74"]]},{"id":"ab19b6e9.4b1ed8","type":"comment","z":"ce798f74.64a9d8","name":"Facebook event calendar URL","info":"","x":890,"y":340,"wires":[]},{"id":"83c3be58.0fc9a8","type":"http response","z":"ce798f74.64a9d8","name":"404","statusCode":"404","headers":{},"x":430,"y":360,"wires":[]},{"id":"6b67410a.701ab","type":"change","z":"ce798f74.64a9d8","name":"","rules":[{"t":"move","p":"calendar","pt":"msg","to":"payload","tot":"msg"},{"t":"set","p":"headers['content-type']","pt":"msg","to":"text/calendar","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1180,"y":300,"wires":[["d1e9086f.645b78"]]},{"id":"d1e9086f.645b78","type":"http response","z":"ce798f74.64a9d8","name":"Return","statusCode":"200","headers":{},"x":1330,"y":300,"wires":[]},{"id":"eaff2195.a68b38","type":"change","z":"ce798f74.64a9d8","name":"Set Browser User Agent","rules":[{"t":"set","p":"headers.User-Agent","pt":"msg","to":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":300,"wires":[["13bdf419.e491ec"]]}]

Docker Volume Backups


Backups are always an issue. I plan to switch to ZFS for the snapshotting and remote sync features, until then I have take the useful volume-backup and broken it until it works with rdiff-backup

Build the container

First you have to clone the repo and build the container

git clone
cd volume-backup
docker image build -t vbackup:1.0 .


Now you can run the container mounting the [volume-name] at /volume and your [backup-dir] at /backup

docker run -v [volume-name]:/volume -v [backup-dir]:/backup --rm vbackup:1.0 backup

Additional rdiff-backup options and be passed via the -o switch and a quoted option for example -o "--exclude"


To restore you must supply some form of options i.e. -o "-r 10D" or restore backup from 10 Days ago

docker run -v [volume-name]:/volume -v [backup-dir]:/backup --rm vbackup:1.0 restore -o "-r 10D"

Trimming old files

With incremental backups it's important to occasionally trim old files that just don't exist anymore. Like Restore you must provide some form of option.

Prior to doing a backup I will run this command to remove files older than 20 Backups

docker run -v [backup-dir]:/backup --rm vbackup:1.0 remote -v -o "--remove-older-than 20B"

I hope you found this helpful. It's not a very clean script, I had to hack apart the reference script quite a bit in order to get it all working. But it serves it's purpose quite well.